Patent Publication Number: US-8126220-B2

Title: Annotating stimulus based on determined emotional response

Description:
BACKGROUND 
     Large increases in the quantity of audio-visual data have spawned a new class of problems related to searching, indexing and rating such data. For example, for a given audio-visual stimulus such as a movie, acquiring such data as emotive audience response (happy, sad, afraid, disgusted, bored, etc) as well as a general measure of the strength of audience response can assist in providing a search and index system for that movie, as well as a rating guide to compare with other movies. Currently, such data must be acquired by means of manual markup. This is both time consuming and biased towards the personal preferences of the markup expert. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a high-level flowchart in accordance with an embodiment of the present invention. 
         FIG. 2  is an illustration of a system in accordance with an embodiment. 
         FIG. 3  is an illustration of a sequence in accordance with an embodiment 
         FIG. 4  shows an illustration of the annotation generation module in accordance with an embodiment. 
     
    
    
     As shown in the drawings for purposes of illustration, a method and system for of annotating audio-visual data is disclosed. In an embodiment, the method and system are designed to create real-time audio-visual annotations while the data is being displayed to an audience. Accordingly, since the annotations are created in real-time, the annotations can be used at the point of collection for various purposes (crowd control, targeted marketing, etc.). Alternatively, the annotations can be indexed and rated to be easily searched and accessed at a later time. 
       FIG. 1  is a flowchart of a method in accordance with an embodiment. A first step  101  involves detecting a plurality of facial expressions in an audience based on a stimulus. A next step  102  involves determining an emotional response to the stimulus based on the facial expressions. A final step  103  includes generating at least one annotation of the stimulus based on the determined emotional response. Again, by implementing this automated feature, the annotations can be used at the point of collection for various purposes. 
     For the purposes of this patent application, an annotation is a digital flag associated with a particular point of the stimulus that expresses the audience response in some way. It could be as simple as “how many people are there” or as complex as a list of the locations of all people in the audience and their measured expression at some point in the stimulus. In an embodiment, an annotation indicates the audience&#39;s emotional response at a given stimulus point whereby a “stimulus point” may be either a particular time in the playing of the content, or a range of times, so that the annotation may be “the audience found this bit funny” or “the audience found the whole thing funny”. 
       FIG. 2  is an illustration of a system  200  in accordance with an embodiment. The system  200  includes display means  202  for displaying an audio-visual stimulus to an audience  204 , video detection means  206  for detecting the facial expressions of the audience and computer processing means  208  for determining an emotional response to the stimulus based on the facial expressions and creating at least one annotation of the stimulus based on the determined emotional response. 
     In an embodiment, display means  202  is capable of displaying an audio-video stimulus and could be a movie screen, a television screen, advertising billboard or the like. Video detection means  206  includes an image capturing device such as a digital video camera or the like. 
     In an embodiment, the stimulus could be an audio-visual stimulus such as a marketing ad, a movie, or the like. However, the proposed system could be employed in conjunction with a real-time stimulus such as a sporting event, a speech, or the like. Accordingly, one of ordinary skill in the art will readily recognize that a variety of stimuli could be employed while remaining within the spirit and scope of the present inventive concepts. 
     The computer processing means  208  may be any type of personal computer, portable computer, or workstation computer that includes a processing unit, a system memory, and a system bus that couples the processing unit to the various components of the computer. The processing unit may include one or more processors, each of which may be in the form of any one of various commercially available processors. Generally, each processor receives instructions and data from a read-only memory and/or a random access memory. The system memory typically includes a read only memory (ROM) that stores a basic input/output system (BIOS) that contains start-up routines for the computer, and a random access memory (RAM). The system bus may be a memory bus, a peripheral bus or a local bus, and may be compatible with any of a variety of bus protocols, including PCI, VESA, Microchannel, ISA, and EISA. 
     The computer  208  also may include a hard drive, a floppy drive, and CD ROM drive that are connected to the system bus by respective interfaces. The hard drive, floppy drive, and CD ROM drive contain respective computer-readable media disks that provide non-volatile or persistent storage for data, data structures and computer-executable instructions. Other computer-readable storage devices (e.g., magnetic tape drives, flash memory devices, and digital video disks) also may be used with the computer. A user may interact (e.g., enter commands or data) with the computer  208  using a keyboard and a pointing device. Information may be displayed to the user on a monitor or with other display technologies. In some embodiments, the computer  208  also may consist of one or more graphics cards, each of which is capable of driving one or more display outputs that are synchronized to an internal or external clock source. 
       FIG. 3  is an illustration of a sequence in accordance with an embodiment. Accordingly, a single frame  305  depicting the audience of interest is captured and processed. A face detection component  310  then determines the face locations  305   a ,  305   b ,  305   c ,  305   d  in the frame  305 . These face locations  305   a ,  305   b ,  305   c ,  305   d  are then sent to an expression analysis component  320  which determines the expression of each detected face. A mood analysis component  330  then collates this information using standard statistical techniques to answer questions like: 
     What is the overall mood of the group in the video frame? 
     Is the group mood adequately expressed by a single overall result?? 
     If a single overall result doest not describe the group well, are there localized subgroups with relatively homogeneous moods? 
     Which individuals differ dramatically from the overall mood, or their local mood? 
     This analysis is repeated on each frame in a video feed of the audience of interest thereby enabling tracking of the change of group mood over time. Additionally, by clustering the faces in a frame using the distance between them as a measure, subgroups within the crowd can be treated in an identical fashion to the overall group. Consequently, the differing moods of subgroups within the overall group can be tracked. 
     In order to accomplish the above-referenced methodology, an annotation generation module could be employed to interact with a central processing unit (CPU) within the computer  208 . Referring now to  FIG. 4 , an exemplary annotation module  400  is illustrated. The annotation generation module  300  is configured to interact with the video detection means  206  and the CPU of the computer  208 . The annotation generation module  400  includes central processing unit (CPU) interface electronics  410 , video detection means interface electronics  420 , face detection logic  430 , facial analysis logic  440 , mood analysis logic  450  and annotation generation logic  460 . The CPU interface electronics  410  and the video detection means interface electronics  420  are coupled to the face detection logic  430  wherein the face detection logic  430  is further coupled to the facial analysis logic  440 . The facial analysis logic  440  is further coupled to the mood analysis logic  450  whereby the mood analysis logic  450  is further coupled to the annotation generation logic  460 . 
     Although the components of the above-described annotation generation module  400  are shown in a specific configuration, one of ordinary skill in the art will readily recognize the components of the annotation module  400  could be configured in a variety of ways while remaining within the spirit and scope of the inventive concepts. 
     The CPU interface electronics  410  and the video detection means interface electronics  420  include the electronic circuitry employed by the annotation generation module  400  to respectively communicate with the CPU (not shown) and the video detection means  420 . The facial detection logic  430  and the facial analysis logic  440  respectively include logic for detecting faces and facial expressions within the audience  204 . This can be accomplished by a wide range of methods that are well known to those skilled in the art. Examples of such methods are contained in the article “Real Time Face Detection and Facial Expression Recognition Development and Applications to Human Computer Interaction” by Bartlett et al and appearing in the proceedings of the CVPR, 2004. Accordingly, it should be noted that one of ordinary skill in the art will readily recognize that a variety of techniques could be employed while remaining within the spirit and scope of the present inventive concepts. 
     The mood analysis logic  450  includes logic for determining an emotional mood of the audience based on the detected facial expressions. In an embodiment, this is accomplished with a simple expedient of agglomerating the instantaneous emotions reflected by individual facial expressions over the audience of interest, and over the time period of interest. There are widely used coding systems, such as the FACS system which associate emotions with particular arrangements of facial muscles. A common set of emotions in expression recognition are joy, sadness, surprise, anger, fear, disgust and neutral. By taking appropriate sample statistics of expressions over a period of time, the mood of that group of individuals over that time period can be appropriately characterized. 
     The annotation generation logic  460  includes logic for generating at least one annotation of the stimulus based on the determined emotional response. This can be accomplished by associating the determined emotional responses of the audience with a timestamp noting the beginning and end of the response period. A collection of one or more quadruples giving the audience grouping, the start time, the end time and the emotional response may then be used to annotate the original stimulus by synchronizing the timestamps with the time at which the stimulus began being viewed by the audience. 
     The annotation generation logic  460  also includes logic for indexing the annotation(s) based on the determined emotional response(s) of the audience. This is just the process of maintaining a two-way linkage between the original stimulus and the annotations, so that the annotation quadruples above are augmented with a link to the relevant stimulus. These may be stored according to any standard database methodology, preferably enabling queries such as “all stimuli portions that provoked a response of 5 seconds or more of joy”. 
     As shown in the drawings for purposes of illustration, a method and system for of annotating audio-visual data is disclosed. In an embodiment, the method and system are designed to create real-time audio-visual annotations while the data is being displayed to an audience. Accordingly, since the annotations are created in real-time, the annotations can be used at the point of collection for various purposes (crowd control, targeted marketing, etc.). Alternatively, the annotations can be indexed and rated to be easily searched and accessed at a later time. 
     The above-described embodiment may also be implemented, for example, by operating a computer system to execute a sequence of computer readable instructions. Accordingly, a computing device typically includes at least some form of computer readable media. Computer readable media can be any available media that can be accessed by computing device. By way of example, and not limitation, computer readable media may include computer storage media and communications media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium which can be used to store the desired information and which can be accessed by a computing device. 
     Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media. Accordingly, an alternate embodiment includes a computer readable medium having computer executable components for annotating audio-visual data. 
     Without further analysis, the foregoing so fully reveals the gist of the present inventive concepts that others can, by applying current knowledge, readily adapt it for various applications without omitting features that, from the standpoint of prior art, fairly constitute essential characteristics of the generic or specific aspects of this invention. Therefore, such applications should and are intended to be comprehended within the meaning and range of equivalents of the following claims. Although this invention has been described in terms of certain embodiments, other embodiments that are apparent to those of ordinary skill in the art are also within the scope of this invention, as defined in the claims that follow.