Context-based recognition system for interactive tv and telephony

Abstract: A method and apparatus is provided to enable a user watching and/or listening to a program to retrieve supplemental information or supplemental entertainment pertaining to an item just seen or words just heard. The apparatus includes a recognition system which recognizes the user's request and causes its satisfaction to be provided. A dynamically changing dictionary is created from items or keywords extracted from the most recently received program portion. The items or keywords may be contained in closed caption text in a TV signal. Particular items or keywords may be assigned a high priority so that they are maintained in the dictionary for a long duration. The user's request is matched against this continually changing dictionary. Upon recognition of the request, a search is initiated to access, import and deliver to the user the information required to satisfy the request. The information may be presented to the user separately or in coincidence with the normal program using picture in picture or split screen. Alternate embodiments enable preprocessing to obtain and store expanded information for such items and keywords deemed to be of high priority. This speeds up the satisfaction of the user's request for these items or 1 keywords.

Description: FIELD OF THE INVENTION The present invention is directed to the field of interactive multimedia. It is particularly directed to interactive database retrieval, storage, and display. BACKGROUND OF THE INVENTION It is a constant endeavor to find ways to provide to a user receiving a video and/or audio transmission more and more supporting information. A developing support process is to provide the user with interactive services. For instance, interactive TV proponents are putting forth a variety of concepts for interactive means for viewers to participate more in a communicating way with material displayed on a TV. This often takes the form of satisfying a user's inquiry based on information displayed and/or heard. Compact Disk - Interactive (CD-I) and other preprogrammed multimedia materials allow a user/viewer to navigate through a database's contents. It gives the viewer options of making inquiries based on preprogrammed menus. Inquiries may be made using electronic, electro-optic or infra-red direct or remote controllers for selection from a displayed menu. Some applications employ voice recognition techniques. These allow a user to make a voice inquiry or to make a voice selection from a simple menu, or from a more extensive set of options such as a table of contents, an index or even a complete book. Commercial examples exist for text and bibliographic database searches. Examples of this type of database include Dow Jones, Nexis/Lexis, and Dialog. Implementation of voice or other recognition selection techniques for such systems, generally requires that the recognition system be capable of matching the user input against a huge dictionary. The dictionary may have to include virtually all words in the language, even proper nouns, acronyms and jargon. Such recognition processing is hardware and software intensive and is expensive to implement in both dollars and processing time. Even then, the results are inherently subject to long delays, ambiguity and error because of the large target dictionary that has to be searched. In WO 94/14282 a viewer interface for a television program delivery system is described. The innovation relates to methods and devices for viewer pathways to television programs. Specifically, the interface involves hardware and software used in conjunction with a television at the view home to create a user friendly menu based approach to television program access. The device is particularly useful in a program delivery system with hundreds of programs and a data signal carrying program information. The disclosure describes menu generation and menu selection of television programs. SUMMARY OF THE INVENTION According to the present invention, there is provided a data processing system for processing an input signal stream associated with a program being presented to a user, said input signal having a plurality of user recognisable components, presented during the program, on which an application can provide additional information, said system comprising: a context extractor for extracting each of said plurality of user recognisable components from said input signal stream; a buffer for dynamically storing recently received user recognisable components extracted by said context extractor; a user interface for said user to make a request to receive said additional information regarding at least one of said plurality of user recognisable components; and a recognition processor responsive to said request to recognise if there is a match between said request and one of said dynamically stored components and, in response to a match, to cause said application to provide said additional information on said matched components to said user. The invention also provides a corresponding method of processing an input signal stream associated with a program being presented to a user. Thus, a user can watch and/or listen to a program to retrieve supplemental information pertaining to an item just seen or words just heard. A dynamically changing dictionary is preferably created from items or keywords extracted from the most recently received program portion and the user's request is matched against this continually changing dictionary. Upon recognition of the request, a search is initiated to access, import and deliver to the user the information required to satisfy the request. It is desirable for the dictionary to include an index of the selected information components, store them sorted in an ascending alpha numeric sequence, and to make them available to the user from a menu display means for displaying the index. In one embodiment, the system further comprises an application recorded on a recording medium to provide the expanded information for at least one of the information components. In one embodiment, the recognition processor performs preprocessing on at least one of the extracted information components. The buffer has a storage capacity which, upon becoming filled, receives additional information components by overwriting the oldest information components stored in the buffer. In one embodiment the information components are contained in closed caption text with the input signal, and the extractor includes a means to separate the closed caption text from the input signal. Preferably, both the program and the supplemental information are presented to the user by way of a visual display. However, the invention may also be applied to audio transmission. The application is preferably a database application capable of providing additional information for at least one of said plurality of topics. DETAILED DESCRIPTION OF THE INVENTION The present invention will now be described in more detail, by way of example, with reference to the accompanying figure which is a block diagram of an embodiment of the present invention. This embodiment of the invention provides a method and apparatus to enable a user watching and/or listening to a program to retrieve supplemental information or supplemental entertainment pertaining to an item just seen, or more generally pertaining to a word or group of words just heard. The program may be received via any transmission means including radio, TV and telephone. The apparatus includes a recognition system which can recognize the user's request and cause its satisfaction to be provided. This is accomplished by creating a dynamically changing dictionary of items or keywords extracted from the most recently received program portion. The user's request is matched against this continually changing dictionary. Upon recognition of the request, a search is initiated to access, import and deliver to the user the information required to satisfy the request. In this way, a relatively small, continuously updated dictionary is created in real time. Its entries need only include the items and/or keywords extracted from the most recently received portion of the program. The present invention recognizes that in terms of interactivity, a viewer is most likely to request information regarding items and/or words recently presented. Therefore, the invention creates a dictionary of items, key items, words or keywords recently presented to the user for the program in process. These form the totality of entries in the target dictionary of the recognition system. This reduces the recognition system's recognition requisite to that of recognizing only a limited vocabulary. This is much less complicated and more simple to implement in terms of storage capacity, software and hardware requirement, processing time, cost, power, accuracy and many other success criteria. It also makes possible the advantageous utilizations described herein. As used in this application, the reference to a program 'item' or 'word' refers to a possible target for request recognition. This includes, but is not limited to program 'items' and/or qqq'words' such as spoken words, text, visual or audio information. These are generally given or converted to a text type format. A reference to a 'dictionary' or stored 'vocabulary' refers to any array of coded information against which a recognition match can be sought. Thus, a program 'item' or 'word' is a candidate 'dictionary' entry and is part of the 'vocabulary' of that 'dictionary'. An 'item' or 'word' is said to be 'recently presented' if it is within the group of latest 'items' or 'words' received that fit within a storage buffer provided for that purpose. This may be dependent upon the storage buffer capacity and/or upon the performance capacity of the recognition system. Capacity may be a real physical limit, a user-defined limit, or an application-defined performance limit. As new items and/or words are received older non-repeated items are overwritten. The illustration shows a block diagram of an embodiment of the present invention. It shows a context extractor 102, buffer vocabulary 104, recognition processor 106, user interface 110, and a signal display 108. The context extractor 102 has a data processor to extract words and/or items from an audio, video or telephony signal. The extractor 102 samples the incoming signal and identifies 'words' and transmit those 'words' to the buffer for storage in the dictionary. The selection criteria may include matching all acceptable 'words' against an application-supplied listing of available topics. The signal is also passed undisturbed on to its normal display 108, say a TV set, radio or a telephone. The buffer vocabulary 104 is updatable in real time. It is a means to store the recently extracted items and/or 'words' that define the vocabulary for potential recognition requests. The buffer 104 receives 'words' from the context extractor 102 and operates with parameters that set how many 'words' to store or how long a period of time to store them. As the buffer fills, older or lower priority words are dropped and new entries replace them so that the buffer always contains the latest most relevant information for the user. The buffer is able to respond to a request from the user to display its contents, or any part thereof. The buffer also responds to a request from the recognition processor for the current vocabulary. The recognition processor 106 is a means to receive requests from the user and is capable of matching 'words' spoken or otherwise transmitted. The recognition processor can be similar to the 'IBM Voice Type Dictation Solution' system. It recognizes the user's request by matching the request 'words' against the vocabulary stored in the buffer 104. When a match is made, the recognition processor 106 interprets the meaning of the request and relays the request to the application 112. As an option, means are provided to anticipate requests for a match by relaying 'words' already in the buffer even before a request is made. When this is employed, the application can preprocess searches for those 'words' and make the search results ready for delivery to the user. In the event that the user actually makes a request for a preprocessed item the system satisfies the request more quickly. The user interface 110 is a means for the user receiving the radio, TV or telephone signals to make requests by voice or other input request means for more video or audio information about a topic recently seen or heard. The user interface gives a viewer or listener the ability to directly or remotely make requests to an application and to view the response. Options are provided for the user to view or hear the 'words' that are valid requests. Options may be provided for other forms of requests, say by button selection from a visual or audio menu of valid requests. The signal display 108 is a means to present audio, video and/or telephone signals to a viewer or listener. The signal display 108 may be a normal radio, TV or telephone that a user is watching or listening to. It presents whatever the user normally expects and in addition shows the response to context-stimulated requests. The display is also used to present the menu of valid requests when commanded by the user. For a video display situation, the request satisfaction can use selective video to replace or be shown together with the normal video. This can be accomplished using known techniques including picture in picture and split screen. A useful option is to provide a means for the user to display all or part of the buffer contents to enable the user to choose a valid request. This can be formed in the fashion of a menu of the dictionary entries. In one embodiment of this invention, the apparatus employs closed caption text as input to the storage buffer. In the United States, closed caption text is transmitted in line 21 of the vertical blanking interval in NTSC signals. It is desirable that the apparatus and method of this invention be able to capture and employ other video data services for searching, locating and importing information to satisfy a user's request. Examples of such services are teletext and the Electronics Industry Associations standard data services for NTSC, EIA-608 Standard: Line 21 Data Services for NTSC, 1994. It may be desirable to allow the setting of priorities for all 'items' and/or 'words'. Priorities may be set so that certain application relevant 'items' or 'words' are kept a longer than ordinary duration, or even for the whole length of the program being watched. This could depend upon the type of program being watched, news, sports, music, movie etc. A high priority may be assigned to words identifying the program itself. Some of this information may be captured directly from the visual or audio part of the signal. It may also be captured from imbedded parts of the signal, such as closed caption text transmitted in video vertical blanking intervals. Any stored item and/or word only occurs once in the dictionary. In some applications, the number of occurrences of an item and/or word in the program may be continuously updated and logged. This can serve as a basis for determining the priority of that item or word. It is also desirable to provide means to do preprocessing on items and words stored. The preprocessor may perform searches to seek, identify, prepare for retrieval or even retrieve information from one or more particular databases regarding items or words stored in the dictionary. This may be performed continuously as new items and/or words are received and added to the buffered storage. Preprocessing of key words representing potential user requests, decreases system response time when such a request is subsequently made. Preprocessing is desirably performed in conjunction with the priorities of particular items or words. It may only search for items or words assigned with at least a moderately high priority level, and actually retrieve information only for items or words assigned with a higher priority level than that of the moderately high priority level. There are many possible embodiments for the present invention. The following describes several of these for a program user seeing and/or hearing something that is of further interest. In one scenario, a user is watching a television news program, the viewer hears the newscaster mention some country, for example, 'France'. The user may even see a map of that country shown on the screen. The map shown is not informative enough for the viewer. Then the newscaster moves on to another topic. With this invention, the viewer calls up a menu of recently stored key words and notes that the word 'France' and/or the 'map of France' was indeed stored in the dictionary. Then the viewer inputs a request for 'France'. The input may be via voice, or via a direct or remote selection from a displayed dictionary menu. The recognition system matches that word against the words in its dictionary, and recognizes that the viewer wants some information about France. It issues a request to an application database for information about France. The application may be on a local or remote CD-ROM or database which includes an encyclopedia. The application retrieves the encyclopedia's data entry on France. Depending on the sophistication of the application, the retrieved data may be displayed in its entirety, the user may be offered topic entry segments for display, or the user may be provided with a listing of information headings for the selected topic, namely France. The retrieved information may be displayed as an overlay on the TV broadcast via a picture-in-picture, split screen, text at the bottom or by other means. The user may even be offered a display of the map of France. The viewer ordinarily has the ability to zoom in or out and to put the news story location in perspective. In another embodiment, the apparatus provides the needs for the following scenario. While watching a movie a viewer hears an actor refer to a character in the plot. A service provided by the movie program producer, broadcaster or a dial-up service provider, gives program relevant information upon a user's request. The entire index of provided information may be extremely large, and take up more memory capacity than is available for use as a menu display to a general user. It may include items such as program statistics, its characters and/or the cast playing the characters. The present invention reduces the amount of index information required in,the menu displayed to the user. The displayed menu is limited to index information regarding only items recently presented to the viewer. This limited amount of index information is indeed within the memory capacity of the general user's equipment . The index information is dynamic, and slowly changing to contain items and words from the most recently presented part of the movie. The viewer chooses the specific desired topic from the displayed menu. For example, if the user inputs the name of a character for whom additional information is desired, the name is compared to the menu dictionary. The recognition processor then compares the name against the valid names in the displayed menu. Upon making the match, the processor transmits the match to the database application. The desired information is retrieved and displayed as a subtitle type text, while the viewer is still watching the movie. A second portion of memory information is relatively static. It contains some menu entries that are normally maintained for the duration of the program. This includes the name of the movie and other information that may be used to pin point program relevant database segments. Another aspect of the present invention is a dynamic dictionary comprising an input information stream having a plurality of recognizable information components, a recognizer for receiving the information stream and recognizing and selecting the information components and a buffer for storing the selected information components. It is desirable for the dictionary to include an index of the selected information components. The selected information components may be stored in a sorted arrangement, and be made available to the user from a menu display means for displaying the index. The sort may be based upon the information component's priority, the sequence of being received, or by descending or ascending alpha numeric sequence.

Claims: A data processing system for processing an input signal stream associated with a program being presented to a user, said input signal having a plurality of user recognisable components, presented during the program, on which an application (112) can provide additional information, said system comprising: a context extractor (102) for extracting each of said plurality of user recognisable components from said input signal stream; a buffer (104) for dynamically storing recently received user recognisable components extracted by said context extractor; a user interface (110) for said user to make a request to receive said additional information regarding at least one of said plurality of user recognisable components; and a recognition processor (106) responsive to said request to recognise if there is a match between said request and one of said dynamically stored components and, in response to a match, to cause said application to provide said additional information on said matched component to said user. A system as claimed in claim 1 further comprising a visual display device (108) on which said program and said additional information may be displayed to the user. A system as claimed in claim 1 or claim 2 further comprising such an application program stored on a recording medium to provide said additional information for at least one of said plurality of user recognisable components. A system as claimed in any preceding claim wherein said recognition processor performs preprocessing on at least one of said plurality of extracted components, prior to a user request for additional information on said components. A system as claimed in any preceding claim wherein said request is a voice request made by said user. A system as in any one of the preceding claims wherein, upon said received components filling the storage capacity of the buffer, additionally received components are stored by overwriting the oldest stored components. A system as claimed in claim 2 or any one of claims 3 to 6 as dependent on claim 2 wherein said input signal includes closed caption text, and wherein said extractor includes a means to separate said closed caption text from said input signal, and wherein at least one of said plurality of user recognisable components is included in said closed caption text. A system as claimed in any one of the preceding claims further comprising a means for assigning a priority to each of said components extracted by said context extractor. A system as claimed in claim 8 wherein each of said information components extracted is stored in said buffer for at least a predetermined duration dependent upon said priority assigned to each of said extracted components. A method of processing an input signal stream associated with a program being presented to a user, said input signal stream having a plurality of user recognisable components, presented during the program, on which an application (112) can provide additional information, said method comprising the steps of: extracting each of said plurality of user recognisable components from said input signal stream; dynamically storing in a buffer (104) recently received user recognisable components extracted from said input signal stream; responding to a request from said user for said additional information regarding at least one of said plurality of user recognisable components by matching said request to said user recognisable components stored in said buffer and, if there is a match, causing said application to provide said additional information on said matched component to said user. A method as claimed in claim 10, comprising the steps of visually displaying said input signal stream and said additional information to said user. A method as claimed in claim 10 or claim 11 further comprising the step of preprocessing at least one of said extracted components, prior to a user request for additional information on said component. A method as claimed in any one of claims 10 to 12 wherein said request is recognised from a voice request by said user. A method as claimed in any one of claims 10 to 13 which, upon said extracted components filling the storage capacity of the buffer, comprises the further step of storing additionally received components by overwriting the oldest stored information components. A method as claimed in any one of claims 10 to 14 further comprising the step of assigning a priority to each of said extracted information components and wherein each of said extracted information components is stored for a particular duration and wherein said particular duration is dependent upon said assigned priority.

Inventors: BRODSKY MARC HERBERT, BRODSKY, MARC HERBERT
Applicants: IBM, INTERNATIONAL BUSINESS MACHINES CORPORATION
Application year: 1997
Main classification: H04N7
IPC codes: H04N5, G06F3, H04N7, H04M3, G09G5, H04H1, H04H60, H04H20