ADAPTIVE METHOD AND APPARATUS FOR AUTOMATICALLY CUSTOMIZING ENHANCED PROGRAM CONTENT TO USER PREFERENCES

Abstract: A method and apparatus are disclosed for providing personalized supplemental programming content that enhances primary programming content. Enhanced program content is automatically and selectively triggered in accordance with a viewer profile. The viewer profile provides a numerical representation indicating the viewer's level of interest a number of attributes. A disclosed viewer preference learning process dynamically adjusts the numerical representations set forth in the viewer profile based on the viewer's acceptance or rejection of enhanced program content. The viewer's profile is thus automatically adjusted by observing the viewer's reactions to presented enhancement content. The inferences contained in the viewer profile that automatically trigger the display of individual enhanced program content are strengthened or weakened based on the viewer's acceptance or rejection of enhanced program content, or based on an inferred determination as to whether the viewer enjoyed presented enhanced program content.

Description: Adaptive method and apparatus for automatically customizing enhanced program content to user preferences Field of the Invention The present invention relates to the enhancement of primary programming content with supplementary content, and more particularly, to a method and apparatus for tailoring such enhanced content to user preferences. Background of the Invention Supplemental programming content is often provided with primary programming material. The closed-captioning system, for example, provides a textual version of the audio information associated with primary programming content to permit people that are hearing impaired to read along with the programming content. A number of systems for enhancing programming content have also been proposed or suggested. WebTV™, for example, provides an enhanced interactive television service using Internet and digital technologies. For example, WebTV™ permits users to play along with game shows and participate in polls during television programming. While currently available enhanced programming services provide an improved viewing experience for viewers, they suffer from a number of limitations, which if overcome, could greatly expand the utility of such enhanced television systems and the relevance of the enhanced content they present to an individual viewer. Specifically, most available systems for enhancing programming content are not sufficiently customized to user preferences. One system for enhancing programming content is disclosed in United States Patent Application Serial Number 09/351,086, filed July 9, 1999, entitled "Method and Apparatus for Linking a Video Segment to Another Video Segment or Information Source," assigned to the assignee of the present invention and incorporated by reference herein. United States Patent Application Serial Number 09/372,959, filed August 12, 1999, entitled "Customizing Database Information For Presentation With Media Selections," assigned to the assignee of the present invention and incorporated by reference herein, embeds codes in the primary programming content to selectively trigger enhancement or supplemental content in accordance with a user profile. In this manner, the user profile filters the enhancement material that is presented to the user. In addition, United States Patent Application Serial Number 09/532,845, filed March 21, 2000, entitled "System and Method for Automatic Content Enhancement of Multimedia Output Device," hereinafter referred to as the "Automatic Content Enhancement System," assigned to the assignee of the present invention and incorporated by reference herein, utilizes video processing and feature extraction techniques to provide automatic content enhancement for primary programming content. The disclosed Automatic Content Enhancement System recognizes patterns or features in the primary programming material and thereafter automatically triggers the enhancement or supplemental content, optionally in accordance with a user profile. For example, a television program may include a logo for a given company that is recognized by the Automatic Content Enhancement System. The recognized logo is then correlated with enhanced content that is typically stored locally. Based on user preferences (for example, whether the user is interested in that particular company) and the correlated enhanced content, the Automatic Content Enhancement System modifies the primary programming content in an appropriate way. For example, the enhanced content presented by the Automatic Content Enhancement System might be a commercial video clip or the telephone number of a local agent of the company. In one embodiment, the modification to the primary programming content overlays the local agent's telephone number on the video signal or buffers the primary broadcast signal and plays a supplemental commercial video clip. Alternatively, the supplemental commercial video clip can be presented using picture-in-picture technology simultaneously with the continued presentation of the primary programming content. Although these profile-based systems significantly advance the quality and relevance of presented enhancement material, relative to conventional techniques, they do not adapt to whether or not the user actually liked presented enhanced program content. A need therefore exists for a method and apparatus for presenting enhanced program content in accordance with personalized preferences of individual viewers that are derived from viewing behavior. A further need exists for a method and apparatus for learning a viewer's preferences by observing the viewer's reactions to presented enhanced content. Yet another need exists for a method and apparatus that strengthens or weakens the individual inferences that trigger the display of enhanced program content based on viewing behavior. Summary of the Invention Generally, a method and apparatus are disclosed for providing personalized supplemental programming content that enhances primary programming content. The enhanced program content may include, for example, biographical information about an individual appearing in the primary programming content or current or historical events regarding a geographical area indicated in the primary programming content. The present invention automatically and selectively triggers presentation of the enhanced program content in accordance with a viewer profile. The viewer profile indicates the level of interest, in the form of a numerical representation, of the viewer in a number of attributes. For example, a viewer profile might have a numerical representation indicating that the user is a rock fan, with the sub-attribute that the user is a particularly heavy fan of a certain rock band. According to one aspect of the invention, a viewer preference learning process dynamically adjusts the numerical representations set forth in the viewer profile based on the viewer's acceptance or rejection of enhanced program content. In this manner, the present invention automatically adjusts the viewer's profile by observing the viewer's reactions to presented enhancement content. The inferences contained in the viewer profile that automatically trigger the display of individual enhanced program content are thus strengthened or weakened based on the viewer's acceptance or rejection of enhanced program content, or based on an inferred determination as to whether the viewer enjoyed presented enhanced program content. A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings. Brief Description of the Drawings FIG. 1 illustrates a personalized program enhancement system in accordance with the present invention; FIG. 2 illustrates a music broadcast where the content is enhanced with several layers of enhanced program content in accordance with the present invention; FIG. 3 is a table illustrating an exemplary enhanced program content database that classifies attributes of the enhanced program content; FIG. 4A is a table illustrating an exemplary viewer profile of FIG. 1 ; FIG. 4B illustrates a symbolic representation indicating the relative level of interest of a viewer in a particular program attribute; FIG. 5 is a flow chart describing an exemplary personalized program enhancement process embodying principles of the present invention; and FIG. 6 is a flow chart describing an exemplary viewer preference learning process embodying principles of the present invention. Detailed Description FIG. 1 illustrates a personalized program enhancement system 100 in accordance with the present invention. The present invention seeks to enhance the enjoyment of a user accessing primary programming content, by providing associated, customized enhanced program content. The primary programming content and enhanced program content may comprise audio, video, textual or other multimedia information, or a combination thereof. The enhanced program content may include, for example, biographical information about an individual appearing in the primary programming content, such as a composer for a musical recording or current or historical events regarding a geographical area indicated in the primary programming content. As shown in FIG. 1 , primary program content is received from a primary program source 110, for example, using a wireless broadcast network, such as a cellular telephone network, a terrestrial television broadcast network, or a digital satellite service (DSS) television network, or a wired network, such as the Internet, Public Switched Telephone Network (PSTN) or a cable television network, or a combination of the foregoing. Likewise, enhanced program content is received from an enhanced program source 120, which may be local or remote. For a remote enhanced program source 120, the programming content may be received by means of any wireless or wired network (or both), in the manner described above. It is noted that the enhanced program content may be received simultaneously with the primary programming content or in an off-line manner. The primary programming content and the enhanced program content are integrated by an enhancement manager 160 and presented to the viewer 195 on a display 180. According to another feature of the invention, the enhancement manager 160 can coordinate the presentation or distribution of the enhanced program content. For example, the enhanced program content can optionally be presented simultaneously with the primary programming content, or the enhanced program content may be provided, for example, by electronic mail, for subsequent access. In a further variation, the primary content may be delayed, so that enhanced program content may be viewed first. The present invention automatically and selectively triggers enhanced program content in accordance with a viewer profile 150, discussed further below in conjunction with FIG. 4A. It is noted that the viewer profile 150 may be associated with a specific user or a group of individuals, such as a household, as would be apparent to a person of ordinary skill in the art. According to one feature of the present invention, the personalized program enhancement system 100 employs a learner 170 to automatically learn the viewer's profile 150 to greater precision by observing the viewer's reactions to presented enhancement content, as determined by a reaction capture module 190. The inferences contained in the viewer profile 150 that automatically trigger the display of individual enhanced program content can be strengthened or weakened based on the viewer's acceptance or rejection of enhanced program content, or based on an inferred determination as to whether the viewer enjoyed presented enhanced program content. These inferences can be stored in the viewer profile 150 thus influencing all future enhancement behavior by the system. Furthermore, the degree of adjustment to the inferences can vary based on the time of day or year, or whether the viewer selected to experience the enhanced program content simultaneously with the primary programming content or at a later time. For example, a user may have a level of interest in certain things only at certain times of the day or year, or when in certain moods. Likewise, an election to receive enhanced program content subsequent to the presentation of the primary programming content may suggest a lower level of interest in the topic than if the user elected immediate and simultaneous presentation of the enhanced program content. In another variation, any adjustments to the inferences can be reduced or eliminated if the user has elected to block all enhanced program content under certain conditions, such as during certain categories of programs. In other words, the fact that enhanced program content is not presented to a user under certain predefined "blocking" conditions, should not prevent similar enhanced program content from being presented to the user in the future. The personalized program enhancement system 100 also includes a matcher 140 that compares characteristics of the received enhanced program content to characteristics specified in the viewer profile 150 to identify enhanced program content that is of interest to the particular viewer. Thus, while the learner 170 infers information about the viewer based on viewer behavior, the matcher 140 applies what has already been learned about the viewer. The matcher 140 provides the personalized enhanced program content, together with the corresponding trigger conditions, to a trigger detector 130. The trigger detector 130, discussed further below in conjunction with FIG. 2, monitors the primary programming content for embedded codes or specific features, or both. The features in the primary programming content that trigger the display of ithe enhanced program content can be (i) embedded codes (i.e., annotations) placed in the enhanced program content by or on behalf of the content provider, or (ii) features that are dynamically recognized in the primary programming content by the trigger detector 130. For example, the trigger detector 130 may employ face recognition, feature recognition, speech recognition or other well-known techniques to recognize predefined features in the primary programming content. It is noted that the embedded codes can actually be the enhanced program content, or a pointer to a local or remote location where the enhanced program content is stored. For example, the trigger detector 130 can identify a particular actor in the primary programming content, and then trigger enhanced program content that is related to the identified actor, assuming the viewer is interested in enhanced program content about this actor. In one implementation, the embedded codes or extracted features are provided to the trigger detector 130, for example, by an indexer (not shown),_ as a stream of tags matched to points or segments in the primary programming content. The trigger detector 130 then monitors the tags and filters out those tags that are not of interest to the user, or that do not correspond to available enhanced program content. For a more detailed discussion of a system that triggers enhanced program content using dynamically recognized features, see, United States Patent Application Serial Number 09/532,845, filed March 21, 2000, entitled "System and Method for Automatic Content Enhancement of Multimedia Output Device," assigned to the assignee of the present invention and incorporated by reference herein. For a more detailed discussion of a system that triggers enhanced program content using embedded codes, see, United States Patent Application Serial Number 09/372,959, filed August 12, 1999, entitled "Customizing Database Information For Presentation With Media Selections," assigned to the assignee of the present invention and incorporated by reference herein. Trigger Detector As previously indicated, the trigger detector 130 monitors the primary programming content for embedded codes or specific features, or both. In an embodiment where the primary programming content contains embedded enhanced program content, the primary programming content is manually indexed with the enhanced program content (or a pointer to a file where the enhanced program content is stored). The indexing can be performed during content production or before content distribution, for example, by the content provider, the broadcaster or a third party. FIG. 2 illustrates a music broadcast 200 where the content 210 is enhanced with several layers 220, 230, 240 of enhanced program content about the same content 210. For example, the music broadcast 200 has been enhanced with biographical information about the composer in layer 220 and an explanation of the movements and passages in layer 240. In an embodiment where the primary programming content is processed to dynamically detect predefined features that trigger the enhanced program content, the trigger detector 130 performs visual or audio characterization using low-level features extracted from the signal. For example, audio-visual characterization can be performed off-line. During audio-visual characterization of the content, the features are extracted and then classified. In this manner, the class label serves as a trigger generator for the enhanced program content. An audio signal can be characterized by features such as: short time energy, band energy ratio, pause rate, and pitch. Other features using Fourier transform and Mel Spectrum Frequency coefficients can also be used. A weighted combination of these features can be used to characterize the audio part (e.g., music program) of a video segment. The features are extracted using a sliding window of N milliseconds with an overlapping of M milliseconds (where N could be, for example, 45 and M could be 30.) The features are then provided to the nearest neighbor classifier to generate k classification labels for each moving window. Then, the k classification decisions from each window are combined to generate a single decision for each segment. This decision represents the identity of the music segment that hopefully reflects the mood, harmony and the melody itself. Using pitch-to-midi conversion, the input audio signal (music) can be converted into a series of midi symbols, as described in A. Ghias et al., "Query by Humming," ACM Multimedia 95, 231-236 (1995), incorporated by reference herein. These are basically part of a play list indicating which notes have been played. The melody then can be converted into an indexed string representing the audio segment. If the next note is higher than the previous one, then it is labeled U (short for Up), if the next note is the same then it is labeled as S (for same), and if the note is lower then it is labeled as D (for down). The opening of the Beethoven Vth symphony is then SSDUSSD. In this manner, the whole database is analyzed and indexed. The content provider can then search for similar strings, i.e., similar music segments. If there are similar segments, then the segment is associated with a label (which serves as a trigger later on) of the name and location of the similar audio in the other video streams. When the program is played to the user, the user can see the related information about the matching music segments. MATCHER The matcher 140 executes a personalized program enhancement process 500, discussed further below in conjunction with FIG. 5, to compare features or characteristics of the received enhanced program content, discussed further below in conjunction with FIG. 3, to viewer preferences (attributes) set forth in the viewer profile 150, discussed further below in conjunction with FIG. 4A, to identify enhanced program content that is of interest to the . particular viewer. The matcher 140 can thus provide personalized enhanced program content, together with the corresponding trigger conditions, to a trigger detector 130. FIG. 3 is a table illustrating an exemplary enhanced program content database 300 that classifies enhanced program content. As shown in FIG. 3, the enhanced program content database 300 contains a plurality of records 305-310 each associated with a different segment of enhanced program content. For each enhanced program content segment identified in column 340, the enhanced program content database 300 provides a description of the segment in column 345 as well as identifying attributes that indicate the target audience for the segment in column 350. Thus, as discussed further below, the matcher 140 attempts to match the segments of enhanced program content to the appropriate viewers, based on the attributes specified in the target audience column 350. It is noted that in the illustrative embodiment, the enhanced program content database 300 assumes that the enhanced program content is stored and indexed locally. In alternate embodiments, the enhanced program content can be received over a network in real-time, simultaneous with the primary programming content. In such a real-time embodiment, the attributes of the target audience for the enhanced program content may actually be received with the enhanced program content itself, for example, in a parallel data stream or in header information. FIG. 4A is a table illustrating an exemplary viewer profile 150. As shown in FIG. 4, the viewer profile 150 contains a plurality of records 405-412 each associated with a different user attribute or sub-attribute (hereinafter, collectively referred to as "attributes"). In addition, for each attribute set forth in column 440, the viewer profile 150 provides a numerical or symbolic representation in column 450, indicating the relative level of interest of the viewer in the corresponding attribute. For example, a viewer profile 150 might have a numerical representation indicating that the user is a rock fan, with the sub-attribute that the user is a particularly heavy fan of the band U2. FIG. 4B illustrates a symbolic representation that indicates the relative level of interest of a viewer in a particular program attribute, such as rock music shows. As shown in FIG. 4B, the number of rock music shows watched by a user is translated into a discrete level of interest in rock music. Thereafter, if the user's level of interest in rock music is "high," for example, then enhanced program content corresponding to rock music will be presented to this user. The viewer profile 150 may initially be established explicitly, for example, in response to a questionnaire administered to the user, or implicitly, for example, by observing the behavior of the user. The present invention dynamically adjusts the numerical representations set forth in column 450 based on the viewer's reaction to presented enhanced program content. In an exemplary embodiment, the numerical representation in the viewer profile 150 includes an intensity scale such as: As previously indicated, the matcher 140 executes a personalized program enhancement process 500, shown in FIG. 5, to compare features of the enhanced program content (from FIG. 3) to viewer preferences (attributes) set forth in the viewer profile 150 (FIG. 4) to identify enhanced program content that is of interest to the particular viewer. It is again noted that the illustrative embodiment of the personalized program enhancement process 400 assumes that the enhanced program content is stored and indexed locally for later retrieval. For a discussion of an alternate implementation that stores and indexes the enhanced program content locally for later retrieval, see United States Patent Application Serial Number 09/372,959, filed August 12, 1999, entitled "Customizing Database Information For Presentation With Media Selections," assigned to the assignee of the present invention and incorporated by reference herein. As shown in FIG. 5, the personalized program enhancement process 500 initially accesses the enhanced program content database 300 during step 510 to retrieve attributes describing the target audience for a segment of enhanced program content. In addition, the personalized program enhancement process 500 accesses the viewer profile 150 during step 520 to retrieve the numerical representations for the corresponding attributes. A test is performed during step 530 to determine if the retrieved numerical representation(s) for the corresponding attribute (s) satisfy predefined criteria. For example, the predefined criteria may be a minimum score. It is noted that the numerical or symbolic representation set forth in field 450 of the viewer profile 150 may be dynamically adjusted based on the viewer's reaction to enhanced program content in accordance with the present invention, as discussed hereinafter. If it is determined during step 530 that the retrieved numerical representation(s) for the corresponding attribute (s) satisfy the predefined criteria, then the enhanced program content is presented to the viewer together with the primary programming content during step 540. The presentation of the enhanced program content and the primary programming content is coordinated by the enhancement manager 160. If, however, it is determined during step 530 that the retrieved numerical representation(s) for the corresponding attribute (s) do not satisfy the predefined criteria, then the enhanced program content is not presented to the viewer and program control terminates during step 550. LEARNER As previously indicated, the personalized program enhancement system 100 employs a learner 170 to modify the viewer's profile 150 by observing the viewer's reactions to presented enhancement content, as determined by a reaction capture module 190. FIG. 6 is a flow chart describing an exemplary viewer preference learning process 600 embodying principles of the present invention. The viewer preference learning process 600 can strengthen or weaken the numerical representations corresponding to each inference contained in the viewer profile 150, based on the viewer's acceptance or rejection of enhanced program content. As shown in FIG. 6, the viewer preference learning process 600 initially monitors the viewer behavior for an acceptance or rejection of enhanced program content during step 610. Once it is determined during step 610 that the viewer has accepted or rejected some enhanced program content, program control proceeds to step 620. It is noted that the viewer behavior suggesting an acceptance or rejection of enhanced program content may be express behavior or inferred behavior. For example, the personalized program enhancement system 100 may require the viewer to affirmatively accept enhanced program content before it is presented, for example, by providing an indication of available enhanced program content in a small window and requiring the viewer to press a button on the remote control to initiate the presentation of the enhanced program content. In addition, behavior suggesting a positive reaction to the enhanced program content is considered an acceptance of the enhanced program content. For example, the following behavior suggests a positive reaction: expanding a smaller picture-in-picture window containing enhanced program content or otherwise changing the focus from the primary programming content to the enhanced program content. Likewise, the following behavior suggests a negative reaction: reducing a larger picture-in-picture window containing enhanced program content or otherwise changing returning the focus from the enhanced programming content to the primary program content. A test is performed during step 620 to determine if the viewer has accepted or rejected enhanced program content. If it is determined during step 620 that the viewer has accepted enhanced program content, then the numerical ratings contained in the viewer profile 150 that are associated with the attributes that contributed to the selection of this enhanced program content are incremented during step 630. If, on the other hand, it is determined during step 620 that the viewer has rejected enhanced program content, then the numerical ratings contained in the viewer profile 150 that are associated with the attributes that contributed to the selection of this enhanced program content are decremented during step 640. It is noted that if a number of attributes contributed to the selection of a given segment of enhanced program content, the numerical representations can be adjusted in accordance with a sliding scale, based on the weights assigned to each contributing feature. Program control terminates during step 650. It is to be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of this invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims: CLAIMS: 1. A method for updating a viewer profile (150) indicating preferences of a viewer concerning enhanced program content (120), comprising the steps of: presenting said enhanced program content (120) to said viewer based on information in said viewer profile (150); and updating said viewer profile (150) based on a reaction of said viewer to said enhanced program content (120). 2. The method of claim 1 , wherein said viewer profile (150) includes a numerical representation indicating a level of interest in at least one attribute. 3. The method of claim 1 , wherein said viewer profile (150) includes a symbolic representation indicating a level of interest in at least one attribute. 4. The method of claim 2, wherein said updating step further comprises the step of adjusting at least one numerical representation. 5. The method of claim 2, wherein said numerical representation is updated in accordance with a sliding scale. 6. The method of claim 3, wherein said updating step further comprises the step of adjusting at least one symbolic representation. 7. The method of claim 1, wherein said enhanced program content (120) is presented simultaneously with primary programming content (110). 8. The method of claim 1, wherein said enhanced program content (120) is presented to said viewer asynchronously relative to said corresponding primary programming content (110). 9. The method of claim 1 , wherein said step of updating said viewer profile (150) further comprises the step of updating said viewer profile (150) based on whether said enhanced program content (120) is presented simultaneously with primary programming content (110). 10. The method of claim 1 , wherein said step of updating said viewer profile (150) further comprises the step of updating said viewer profile (150) based on a time of day. 11. The method of claim 1 , wherein said step of updating said viewer profile (150) further comprises the step of updating said viewer profile (150) based on a time of year. 12. The method of claim 1 , wherein said step of updating said viewer profile (150) further comprises the step of updating said viewer profile (150) based on a mood of the user. 13. The method of claim 1 , wherein said step of updating said viewer profile (150) further comprises the step of updating said viewer profile (150) based on whether a presentation of said enhanced program content (120) is blocked under certain conditions. 14. The method of claim 1, wherein said viewer profile (150) is associated with a specific individual. 15. The method of claim 1 , wherein said viewer profile (150) is associated with a group of individuals. 16. A system ( 100) for updating a viewer profile (150) indicating preferences of a viewer concerning enhanced program content (120), comprising: a memory for storing computer readable code; and a processor operatively coupled to said memory, said processor configured to: present said enhanced program content (120) to said viewer based on information in said viewer profile (150); and update said viewer profile (150) based on a reaction of said viewer to said enhanced program content (120).

Inventors: DIMITROVA NEVENKA, SCHAFFER JAMES D, DIMITROVA, NEVENKA, SCHAFFER, JAMES, D.
Applicants: KONINKL PHILIPS ELECTRONICS NV, KONINKLIJKE PHILIPS ELECTRONICS N.V.
Application year: 2001
Main classification: H04N7
IPC codes: G10K15, H04N7, H04N17