Abstract:
A method of processing audio signals recorded during display of image data from a media file on a display device to produce semantic understanding data and associating such data with the original media file, includes: separating a desired audio signal from the aggregate mixture of audio signals; analyzing the separated signal for purposes of gaining semantic understanding; and associating the semantic information obtained from the audio signals recorded during image display with the original media file.

Description:
FIELD OF THE INVENTION 
     The invention relates generally to the field of audio processing, and in particular to embedding audio metadata in a media file of associated still or video digitized imagery. 
     BACKGROUND OF THE INVENTION 
     The digital multimedia revolution has spawned a vast array of products and devices upon which media can be viewed, such as personal digital assistants (PDAs), digital picture frames, cellular phones, liquid crystal displays (LCD), cathode-ray tube (CRT), projection devices, plasma screens, and the capture devices themselves. The multimedia/imaging industry will continue to embrace ways other than hardcopy prints to view and share imagery. This fact, combined with the proliferation of digital media stored in memory devices and repositories as diverse as the displays themselves, presents a significant challenge in terms of organization, search and retrieval of images of interest. 
     As the number of these digital images continues to grow, there is much effort in industry and academia spent on technologies that analyze image data to understand the content, context, and meaning of the media without human intervention. This area of technologies is called semantic understanding, and algorithms are becoming more and more sophisticated in how they analyze audiovisual data and non-audiovisual data, called metadata, within a media file. For example, face detection/recognition software can identify faces present in a scene. Speech recognition software can transcribe what is said in a video or audio file, sometimes with excellent accuracy depending on the quality of the sound and attributes of the speech. Speaker recognition software is capable of measuring the characteristics of an individual&#39;s voice and applying heuristic algorithms to guess the speaker&#39;s identity from a database of characterized speakers. Natural language processing methods bring artificial intelligence to bear as an automated way for understanding speech and text without human intervention. These methods produce very useful additional metadata that often is re-associated with the media file and used for organization, search and retrieval of large media collections. 
     There have been many innovations in the consumer electronics industry that marry media files such as digital still photographs with sound. For example, U.S. Pat. No. 6,496,656 teaches how to embed an audio waveform in a hardcopy print. U.S. Pat. No. 6,993,196 teaches how to store audio data as non-standard metadata at the end of a digital image file. 
     U.S. Pat. No. 6,833,865 teaches about an automated system for real time embedded metadata extraction that can be scene or audio related so long as the audio already exists in the audio-visual data stream. The process can be done parallel to image capture or subsequently. U.S. Pat. No. 6,665,639 teaches a speech recognition method and apparatus that can recognize utterances of specific words, independent of who is speaking, in audio signals according to a pre-determined list of words. 
     That said, there often is no substitute for human intuition and reason, and a real person viewing media will almost always understand and recognize things that computers have a hard time with. There are those who maintain that computers will one day equal or surpass the processing and reasoning power of the human brain, but this level of artificial intelligence technology lies far into the future. As an example, consider a system that analyzes an image with several people in a scene. The system may use face detection algorithms to locate faces, and recognition algorithms to identify the people. Extending this example into the video space, additional algorithms to detect and identify speech can be employed to produce a transcript, or to augment metadata through recognition of specific words in a list. While the existing technology is promising, it is arguable that such algorithms will compare unfavorably with a human performing these tasks for the foreseeable future. 
     Suppose two people are viewing images as a slideshow on a digital picture frame or other display device. The people can, and often do, comment on who is in the image, the circumstances in which the image was captured. Typically this commentary is ephemeral and has no lasting value beyond the viewing moment. By the time the next image is displayed, the commentary has withered from the minds of the viewers. 
     There has been much activity related to annotating image data with descriptive text. Some use variations on a simple text entry interface, where the viewer enters textual information through a keyboard input device, the text subsequently associated with the image data. For example, Google has a web application called Google Image Labeler, developed by Carnegie Mellon University. It is a collaborative real-time application that turns the task of image keyword tagging into a game. The system takes a “distributed human processing” approach, where individuals spend their own time viewing and tagging randomly chosen images. The words are then saved as keywords in the image file, to aid in future search queries. 
     Other methods for annotating images with additional metadata take advantage of audio, specifically speech. U.S. Pat. No. 7,202,838, teaches a graphical user interface which allows a picture database user to annotate digital pictures to promote efficient picture database browsing, where annotation can take the form of comments spoken by the user. U.S. Pat. No. 7,202,838 describes a system for showing medical imagery on a display, through which additional data can be gathered in several forms, including written annotation and speech, and associated with the imagery for diagnostic and other purposes. In another medically related patent, U.S. Pat. No. 6,518,952 describes a system and device for displaying medical images and controlling a way of recording, synchronizing, and playing back dictation associated with the imagery. 
     Similarly, in U.S. Pat. No. 7,225,131 describes a system and method of capturing user input comprising speech, pen, and gesture, or any combination thereof describing a medical condition, and associating the user input with a bodily location via a multi-modal display that shows a schematic representation of the human body. 
     SUMMARY OF THE INVENTION 
     The present invention permits the automatic capture of viewer commentary on media shown on a display, and subsequent association of the viewer commentary with the displayed media on the various display devices mentioned above. 
     Briefly summarized, in accordance with the present invention, there is provided a method of processing audio signals recorded during display of image data from a media file on a display device to produce semantic understanding information and associating such data with the original media file, comprising: separating a desired audio signal from the aggregate mixture of audio signals; 
     analyzing the separated signal for purposes of gaining semantic understanding information; and associating the semantic understanding information obtained from the audio signals recorded during image display with the original media file. 
     An advantage of the present invention stems from the fact that a media file&#39;s metadata can be appended with additional information or metadata by a combination of automatic algorithms and user-interactive processes for an improved semantic understanding of the captured image(s), for aid in search and retrieval algorithms, and for an enhanced user viewing experience. 
     The present invention has the following additional advantages: Automatic capture and association of audio and audio-related metadata concurrent with multimedia image data display. From the initial moment of display until the next scene, buffered audio samples are automatically stored as metadata in memory for analysis. The semantic analysis of the audio can occur during image display/audio capture in real time, or can occur some time after image display/audio capture. 
     A further advantage of the present invention results from the fact that audio signals captured during image display provides context of the scene and useful metadata that can be analyzed for a semantic understanding of the displayed image. A process, in accordance with the present invention, associates a window of audio signal information for the duration an image is displayed, allowing the viewer the freedom of not having to actively initiate the audio capture through actuation of a button or switch. The only physical actions required by the viewer are to provide commentary germane to the imagery currently displayed, and to progress to the next image or video in the case where the user is not viewing an automated slideshow. The management of the audio capture, and of association of the audio signal and extracted metadata with the image(s) is automatically handled by the device&#39;s electronics and is completely transparent to the viewer. 
     The display device has the option of displaying digitized imagery in various ways. For example, a collection of still images can be displayed sequentially or randomly for a fixed period of time, or they can be displayed according to a pre-determined script as in a slideshow with accompanying music or narration. Another viewing modality is for the viewer to view the pictures in an ad-hoc fashion, browsing through the collection of images within the storage hierarchy in the display device&#39;s internal memory, removable memory or as stored on a remote computer. 
     These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is block diagram that depicts an embodiment of the invention; 
         FIG. 2  shows a media file containing image and audio data; 
         FIG. 3  is a cartoon depicting a typical usage scenario, containing a viewer, a remotely located computer and a display device; 
         FIG. 4  is a detailed diagram showing digitized audio signal waveforms as time-variant signals that overlap an image display scenario; 
         FIG. 5  is a flow diagram illustrating the high-level events that take place in a typical use case, using the preferred embodiment of the invention; 
         FIG. 6  is a block diagram of a Semantic Analysis Process for analyzing the recorded audio signals; 
         FIG. 7  shows a view of the display device showing a group of associated image data thumbnail views. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     In the following description, the present invention will be described in its preferred embodiment as a digital picture frame device. Those skilled in the art will readily recognize that the equivalent invention can also exist in other embodiments. 
     It is understood that the phrases “image”, “captured image”, “image data”, “imagery”, “scene”, and the like as used in this description of the present invention relate to still images as well as moving images, as in a video. The terms “media” and “multimedia” are used interchangeably, and refer to audiovisual data as well as separate audio data, image data and video data as stored in a digitized computer file. When called for, the terms “still image” and “video”, or variations thereof, will be used to describe still or motion imagery display scenarios that are distinct. 
     Referring to  FIG. 1 , a display device  10  capable of displaying still or video imagery is shown. The display device  10  includes a display screen  15  and at least one built-in microphone  20 . Upon activation of the display device  10 , the viewer  80  (see  FIG. 3 ) has the option of viewing imagery as a slideshow, or through ad-hoc browsing of a media file  70  stored in internal memory  30  or a removable memory  35  device. The display device  10  can also include a wireless radio  40  and therefore communicate via wireless signals  95  (see  FIG. 3 ) with a remote computer  90  (see  FIG. 3 ) over wireless network protocols such as Bluetooth, 802.11 “Wi-Fi” or “Wi-Max”. The display device  10  additionally contains an audio analog to digital converter  45  for digitization of the microphone&#39;s  20  output into a recorded audio signal  65 . A computer CPU  25  within the display device  10  controls all encoding and decoding functions, graphical user interface functions, media file  70  input and output, signal processing  135  (see  FIG. 5 ) and any audio analysis  50  tasks, and other typical functionality commensurate with the display device  10 . 
     The display device  10  has the ability to show image data  75  (see  FIG. 3 ) from multiple sources. In the preferred embodiment, the display device  10  contains at least one of a number of possible ways for storing media files  70 . FIG.  2  shows a removable memory  35  such as a Secure Digital (SD) card or Compact Flash Memory card containing at least one media file. Each media file  70   a  can contain image data  75   a , as well as a recorded audio signal  65   a  such as voice annotation, an audio clip, or an audio stream or layer such as those typically associated with video. 
     Referring to  FIG. 3 , the preferred embodiment of the display device  10  shows image data  75   a  on its display screen  15  to a viewer  80 . The viewer commentary  85  from the viewer  80  is sensed by the microphone  20  and digitized by the audio analog to digital converter  45  (see  FIG. 1 ) within the display device  10 . This viewer commentary  85  often contains information that is germane to the image data  75   a  contained in the media file  70   a  (see  FIG. 2 ) currently shown on the display device  10 . For example, the viewer  80  can make mention of recognized individuals  77  or objects in the image data  75   a , or comment on location using place names, or describe actions taking place in the scene. This viewer commentary  85  recorded by the microphone  20  during the viewing of image data  75   a  on the display device  10  is useful with regard to semantic understanding of the associated media file  70   a , and can be considered newly acquired audio metadata in the form of a recorded audio signal  65   a  (see  FIGS. 2 and 4 ). 
       FIG. 4  is a detailed diagram illustrating a timeline  155  and associated temporally progressing depictions of the display device  10 , image data  75   a ,  75   b ,  75   c , and recorded audio signals  65   a ,  65   b ,  65   c . Referring to  FIG. 4 , the display device  10  loads media file  70   a  (see  FIG. 2 ) from one of several possible sources. As described earlier, this source can be the display device&#39;s  10  internal memory  30  (see  FIG. 1 ), a removable memory  35  device (see  FIG. 1 ,  2 ), or a media file  70  located on a remote computer  90  (see  FIG. 3 ). At the t 1  time marker  160   a , the displayed image data  75   a  is shown on the display device  10 . Concurrently, the microphone  20  (see  FIG. 3 ) records viewer commentary  85  (see  FIG. 3 ) and a digitized audio signal  65  is buffered in the internal memory  30 . At the t 2  time marker  160   b , the display device  10  ceases showing the displayed image data  75   a . A segment of the recorded audio signal  65  between the t 1  time marker  160   a  and t 2  time marker  160   b  is designated as recorded audio signal  65   a  associated with the displayed image data  75   a . As the display device  10  begins showing displayed image data B  75   b  at the t 2  time marker  160   b , the display device  10  continues to record the viewer commentary  85  that is buffered in internal memory  30  as a recorded audio signal  65 . At the t 3  time marker  160   c , the display device  10  ceases showing displayed image data B  75   b . A segment of the recorded audio signal  65  between the t 2  time marker  160   b  and t 3  time marker  160   c  is designated as audio signal B  65   b  associated with displayed image data B  75   b . The process of recording audio and associating the sequentially recorded audio signals  65   a ,  65   b ,  65   c  with sequentially displayed image data  75   a ,  75   b ,  75   c  continues until the viewer  80  decides to stop or the display device  10  runs out of media files  70  to show the viewer  80 . 
       FIG. 5  is a flow diagram illustrating the high-level events that take place in a typical usage scenario of the preferred embodiment of the invention. It is helpful to consider this usage scenario by describing the steps in  FIG. 5  and using  FIG. 4  to provide temporal context of the usage scenario. Referring to  FIG. 5 , the process starts with a user  80  (see  FIG. 3 ) initiating a viewing session on the display device  10  (see  FIGS. 1 ,  3 ,  4 ). The display device  10  reads a first media file  70   a  (see  FIG. 4 ) in a Read Media File step  100 . The media file  70   a  is then shown on the display device&#39;s  10  display screen  15  (see  FIG. 1 ) in a present current media file step  105 . During the present current media file step  105 , a parallel capture of audio step  115  captures the viewer commentary  85  (see  FIG. 3 ) as a digitized audio signal  65  (see  FIG. 4 ) as the viewer  80  comments on the image data  75   a  (see  FIG. 4 ) currently shown on the display device  10 . The display device  10  continues to present current media file  105  and capture of audio  115  until such time as the viewer  80  is finished viewing  110  the present current media file step  105 . The viewer  80  decides whether he/she is finished viewing  110  the image data  75  either by the viewer  80  manually progressing to the next media file B  70   b  or the display device  10  automatically advancing to the next media file B  70   b , as would be the case if the viewer  80  were viewing an automated slideshow. When the viewer  80  is either voluntarily or involuntarily finished viewing  110  the displayed image data  75   a , the display device  10  concludes capture of audio  120  of the viewer commentary  85 . With the conclusion of the capture of audio  120  for displayed image data  75   a  (see  FIG. 4 ), audio signal  65   a  is written to media file  70   a  in an append audio step  125 . Should the viewer  80  wish to display another image  130 , or if there remains additional media files  70  constituting a pre-arranged or automated slideshow, the process returns to the read media file step  100 , and the viewing of media files  70  continues until there is a negative result to the display another image  130  decision. 
     The beginning and ending of viewer commentary  85  for a specific media file  70   a  made during a viewing of multiple media files  70   a ,  70   b ,  70   c  can also be determined by analysis of the recorded audio signal  65 . Such analyses can include detection of pauses or specific transitory words (words signaling that the viewer  80  is progressing to the next media file  70 ) in the recorded audio signal  65 , and can indicate the beginning and ending of recorded audio signal  65   a  to be associated with media file  70   a . The pauses or transitory words can also provide control of media file  70  transitions, anticipating when the viewer  80  is finished providing viewer commentary  85  for media file  70   a  and ready to move on to media file B  70   b.    
     When viewing of all the media files  70   a ,  70   b ,  70   c  is complete, there exists recorded audio signals  65   a ,  65   b ,  65   c  associated with the displayed image data  75   a ,  75   b ,  75   c  (see  FIG. 4 ). In the preferred embodiment, the display device&#39;s  10  computer CPU  25  (see  FIG. 1 ) begins a semantic analysis process  60  (see  FIG. 5 ) that analyzes the audio signals  65   a ,  65   b ,  65   c . The semantic analysis process  60  can include a signal processing step  135  to prepare the recorded audio signals  65   a ,  65   b ,  65   c  prior to the extraction of semantic data  140 . Referring to  FIG. 5 , the signal processing  135  and extraction of semantic data steps  140 , depending on their complexity and the computational power of the computer CPU  25  on which they are performed, can be done in real time; i.e. in parallel to the capture of audio step  115 , or in the background while the display device  10  is otherwise idle. A networked remote computer  90  (see  FIG. 3 ) can also perform signal processing  135  and extraction of semantic data steps  140  steps by virtue of the semantic analysis  60  process&#39;s ability to reside external to the display device  10 . In the preferred embodiment of the invention, the display device&#39;s  10  computer CPU  25  performs the signal processing  135 , extraction of semantic data  140 , and update metadata operation  145 , shown in detail in  FIG. 6 . 
     The recorded audio signals  65   a ,  65   b ,  65   c  (see  FIG. 4 ) previously associated with the displayed image data  75   a ,  75   b ,  75   c , (see  FIG. 4 ) can be exploited in a number of ways through signal processing  135  and extraction of semantic data steps  140  (see  FIG. 5 ). Referring to  FIG. 6 , and continuing with the description of the preferred embodiment of the invention, media file  70   a  stored in the internal memory  30  now has a recorded audio signal  65   a  containing viewer commentary  85  (see  FIG. 3 ) spoken by the viewer  80  during viewing of the displayed image data  75   a  (see  FIG. 4 ). The media file&#39;s  70   a  accompanying recorded audio signal  65   a  is simply a segment of the recorded audio signal  65  (see  FIG. 4 ) recorded during viewing of the media files  70   a ,  70   b ,  70   c , however additional operations can be necessary in order to provide useful semantic understanding metadata. The semantic analysis process  60  can use methods such as Independent Component Analysis (ICA) or Principle Component Analysis (PCA) processing  165  to the audio signals  65   a ,  65   b ,  65   c  to enhance or separate the viewer commentary  85  signal component contained therein. Briefly, ICA and PCA  165  are signal processing techniques for extracting source signals from an aggregate mixture of sound  89  (see  FIG. 3 ), which is defined as the summation of sounds from all sound producing sources within the range of the display device&#39;s  10  microphone  20 ; in this case the aggregate mixture of sound  89  is the combination of the viewer commentary  85  from the viewer  80  and the ambient audio  87  emitted by an ambient source  82  (shown in  FIG. 3  as an airplane), summed to produce the recorded audio signal  65 . For a recorded audio signal  65 , those skilled in the art recognize that ICA and PCA allow for extraction of a source signal—in this case a speech component signal such as the viewer commentary  85  discussed above, from the aggregate mixture of sound  89 . The result of ICA/PCA processing  165  is an enhanced speech signal component  170  that is substantially more intelligible. The preferred embodiment of the invention uses ICA/PCA processing methods  165  applicable to a single-mixture signal, which is a mixture of two or more signals obtained using a single microphone  20 . Those skilled in the art will recognize that more effective ICA methods can be applied to multiple signal mixtures, or mixtures of signals obtained from multiple microphones  20 . 
     The enhanced speech signal component  170  is subsequently input to a speech to text operation  175 , whose output constitutes a textual metadata  180 . The textual metadata  180  can be keywords recognized from a pre-determined vocabulary, or a transcription of the speech utterances  85  (see  FIG. 3 ) captured during viewing of the image data  75 . The textual metadata  180  constitutes additional metadata, which is re-associated with the media file  70   a  in an update metadata operation  145 . The semantic analysis process  60  continues for subsequent media files B and C  70   b, c  and their respective recorded audio signals B and C  65   b, c.    
     Other types of semantic analyses can be combined with the analysis of the recorded audio signals  65   a ,  65   b ,  65   c . For example, in addition to a speech to text operation  175 , a face detection/recognition engine applied to the image data  75  (see  FIG. 3 ) can identify one or more faces  77  in the image data  75 , and prompt the viewer  80  to state the name(s) of the recognized persons&#39; faces  77  in the displayed image data  75   a, b, c  and subsequently associate the stated names&#39; textual metadata  180  with the identified faces  77 . Additionally, a scene analysis algorithm can determine places, objects, and activities in the image data  75 , prompt the viewer  80  to state the names of recognized places, objects and activities recognized in the displayed image data  75   a ,  75   b ,  75   c  and subsequently associate the scene-related textual metadata  180  with the identified places, objects and activities. Natural Language Processing (NLP) algorithms can achieve the above-described associations in the case where the textual metadata  180  is a transcription of the viewer commentary  85 . People&#39;s names, place names, object names, and the like are recognized as the NLP algorithm parses the textual metadata  180 . The recognized names, terms, and phrases can then be added to the metadata of the media file  70 , as shown by the update metadata operation  145  (see  FIGS. 5 ,  6 ). 
     Textual metadata  180  obtained from one media file  70   a  (see  FIG. 6 ) can be associated with the other media files  70   b ,  70   c  in cases where media files  70   a ,  70   b ,  70   c  share common metadata, as would be the case if multiple media files  70   a ,  70   b ,  70   c  were captured during a specific event at a specific location. The media files&#39;  70   a ,  70   b ,  70   c  common metadata will include similar or identical date and location or Global Positioning System (GPS) information. 
     Image data  75  (see  FIG. 3 ) that attract more viewer commentary  85  are generally more valuable than images with less viewer commentary  85 . A commentary value metric that reflects the amount of viewer commentary  85  can serve as metadata that informs automated or manual methods for composing stories, presentations, or other products that combine images. For example, a slideshow presentation can be put together based on which media files  70  contain substantial viewer commentary  85  or a substantial amount of associated textual metadata  180 . 
     The update metadata operation  145  (see  FIG. 6 ) can store the additional textual metadata  180  by appending the original media files  70   a ,  70   b ,  70   c  in proprietary or standard metadata tags. Alternatively, the update metadata operation  145  can store the additional textual metadata  180  in a database located in internal memory  30 , removable memory  35 , or an a remote computer  90  (see  FIG. 3 ) where it is associated with, but not necessarily included in, the original media files  70   a ,  70   b ,  70   c.    
     Many display devices  10  (see  FIG. 1 ) have user interfaces that allow for browsing of media files  70   a ,  70   b ,  70   c  (see  FIG. 4 ) located in internal memory  30 , removable memory  35 , or on a remote computer  90  (see  FIG. 3 ). Referring to  FIG. 7 , often it is the case that collections of associated media files  70   a ,  70   b ,  70   c  (see  FIG. 4 ) are shown graphically on the display device&#39;s  10  display screen  15  as a single static or animated thumbnail icon  185   a  or group of thumbnail icons  185  representing one or more associated media files  70   a ,  70   b ,  70   c ; for example, in  FIG. 7  a group of thumbnail icons  185  arranged in a grid on the display screen  15  can indicate that the media files  70   a ,  70   b ,  70   c  have some common attributes reflected in their metadata or their audiovisual content. When the viewer  80  (see  FIG. 3 ) views this, or a subset of, this group of thumbnail icons  185  or other group representation of media files  70   a ,  70   b ,  70   c  any viewer commentary  85  captured while the viewer  80  is viewing the group of thumbnail icons  185  is automatically associated with all media files  70   a ,  70   b ,  70   c.    
     PARTS LIST 
     
         
           10  Display Device 
           15  Display Screen 
           20  Microphone 
           25  Computer CPU 
           30  Internal Memory 
           35  Removable Memory 
           40  Wireless Radio 
           45  Audio Analog to Digital Converter 
           50  Audio Analysis Operation 
           60  Semantic Analysis Process 
           65  Recorded Audio Signal 
           65   a  Recorded Audio Signal 
           65   b  Recorded Audio Signal 
           65   c  Recorded Audio Signal 
           70  Media File 
           70   a  Media File 
           70   b  Media File 
           70   c  Media File 
           75   a  Image Data 
           75   b  Image Data 
           75   c  Image Data 
           77  Recognized Individuals/Objects 
           80  Viewer 
           82  Ambient Audio Source 
           85  Viewer Commentary 
           87  Ambient Audio 
           89  Aggregate Mixture of Sound 
           90  Remote Computer 
           95  Wireless Signals 
           100  Read Media File Step 
           105  Present Current Media File Step 
           110  Finished Viewing Step 
           115  Capture of Audio Step 
           120  Conclude Capture of Audio Step 
           125  Append Audio Step 
           130  Display Another Image Step 
           135  Signal Processing Step 
           140  Extraction of Semantic Data Step 
           145  Update Metadata Operation Step 
           155  Timeline 
           160   a  t 1  Time Marker 
           160   b  t 2  Time Marker 
           160   c  t 3  Time Marker 
           165  Independent/Principle Component Analysis (ICA/PCA) 
           170  Enhanced Speech Signal Component 
           175  Speech to Text Operation 
           180  Textual Metadata 
           185  Group of Thumbnail Icons 
           185   a  Thumbnail Icon