Abstract:
Video on demand services rely on frequent viewing and downloading of content to enhance the return on investment on such services. Videos in general and movies in particular hosted by video portals need to have extensive annotations to help in greater monetization of content. Such deep annotations help in creating content packages based on bits and pieces extracted from specific videos suited to individuals&#39; queries thereby providing multiple opportunities for piece-wise monetization. Considering the complexity involved in extracting deep semantics for deep annotation based on video and audio analyses, a system and method for deep annotation uses video/movie scripts associated with content for supporting video-audio analysis in deep annotation.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to video processing in general and more particularly movie processing. Still more particularly, the present invention is related to a system and method for deep annotation leading to semantic indexing of videos based on comprehensive analyses of video, audio, and associated script. 
       BACKGROUND OF THE INVENTION 
       [0002]    Video portals delivering content based services need to draw more users onto their portals in order to enhance revenue: one of the most practical ways to achieve this is to provide user interfaces that make users see the “bits &amp; pieces” of content that are of their interest. Specifically, a movie as a whole is of interest in the initial stages of viewership; with time, different users need different portions of the movie wherein the portions could be based on scene details, actors involved, or dialogs. The users would want to query a video portal to get the relevant portions extracted from several of the videos and the extracted content packaged for to be delivered to the users. This requirement of users is a great boon for video on demand (VoD) service providers: there is an excellent commoditization and hence, monetization of small portions of the content. Such a micro-monetization is not uncommon on the web-based services: for example, in scientific publishing, there are several opportunities for micro-monetization such as (a) relevant tables and figures; and (b) experimental results associated with the various technical papers contained in a repository. In all such cases and all of the domains, deep annotation of the content helps in providing the most appropriate answers to the users&#39; queries. Consider DVDs containing the movies: deep annotation of a movie offers an opportunity to deeply semantically index a DVD so that users could construct a large number of “video shows” based on their interests. Here, a video show is a packaged content based on “bits &amp; pieces” of a movie contained in the DVD. An approach to achieve deep annotation of a video is to perform a combined analysis based on audio, video, and script associated with the video. 
       DESCRIPTION OF RELATED ART 
       [0003]    U.S. Pat. No. 7,467,164 to Marsh; David J. (Sammamish, Wash.) for “Media content descriptions” (issued on Dec. 16, 2008 and assigned to Microsoft Corporation (Redmond, Wash.)) describes a media content description system that receives media content descriptions from one or more metadata providers, associates each media content description with the metadata provider that provided the description, and may generate composite descriptions based on the received media content descriptions. 
         [0004]    U.S. Pat. No. 7,457,532 to Barde; Sumedh N. (Redmond, Wash.), Cain; Jonathan M. (Seattle, Wash.), Janecek; David (Woodinville, Wash.), Terrell; John W. (Bothell, Wash.), Serbus; Bradley S. (Seattle, Wash.), Storm; Christina (Seattle, Wash.) for “Systems and methods for retrieving, viewing and navigating DVD-based content” (issued on Nov. 25, 2008 and assigned to Microsoft Corporation (Redmond, Wash.)) describes a system for enhancing a user&#39;s DVD experience by building a playlist structure shell based on a hierarchical structure associated with the DVD, and metadata associated with the DVD. 
         [0005]    U.S. Pat. No. 7,448,021 to Lamkin; Allan (San Diego, Calif.), Collart; Todd (Los Altos, Calif.), Blair; Jeff (San Jose, Calif.) for “Software engine for combining video or audio content with programmatic content” (issued on Nov. 4, 2008 and assigned to Sonic Solutions, a California corporation (Novato, Calif.)) describes a system for combining video/audio content with programmatic content, generating programmatic content in response to the searching, and generating an image as a function of the programmatic content and the representation of the audio/video content. 
         [0006]    U.S. Pat. No. 7,366,979 to Spielberg; Steven (Los Angeles, Calif.), Gustman; Samuel (Universal City, Calif.) for “Method and apparatus for annotating a document” (issued on Apr. 29, 2008 and assigned to Copernicus Investments, LLC (Los Angeles, Calif.)) describes an apparatus for annotating a document that allows for the addition of verbal annotations to a digital document such as a movie script, book, or any other type of document, and further the system stores audio comments in data storage as an annotation linked to a location in the document being annotated. 
         [0007]    “Movie/Script: Alignment and Parsing of Video and Text Transcription” by Cour; Timothee, Jordan; Chris, Miltsakaki; Eleni, and Taskar; Ben (appeared in the Proceedings of the 10th European Conference on Computer Vision (ECCV 2008), Oct. 12-18, 2008, Marseille, France) describes an approach in which scene segmentation, alignment, and shot threading are formulated as a unified generative model, and further describes a hierarchical dynamic programming algorithm that handles alignment and jump-limited reordering in linear time. 
         [0008]    “A Video Movie Annotation System-Annotation Movie with its Scrip” by Zhang; Wenli, Yaginuma; Yoshitomo; and Sakauchi; Masao (appeared in the Proceedings of WCCC-ICSP 2000. 5th International Conference on Signal Processing, Volume 2, Page(s):1362-1366) describes a movie annotation method for synchronizing a movie with its script based on dynamic programming matching and a video movie annotation system based on this method. 
         [0009]    The known systems do not address about how to bootstrap deep annotation of videos based video and script analyses. A bootstrapping process allows for errors in the initial stages of the analyses and at the same time achieves enhanced accuracy towards the end. In the bootstrapping process, it is important to have as much of possible coarse-grained annotation of a video as possible so that the error in script alignment is minimized and the effectiveness of the deep annotation is enhanced. The present invention provides for a system and method to use the script associated with a video in the coarse-grained annotation of the video, and uses the coarse-grained annotation along with the script to generate the fine-grained annotation (that is, deep annotation) of the video. 
       SUMMARY OF THE INVENTION 
       [0010]    The primary objective of the invention is to associate deep annotation and semantic index with a video/movie. 
         [0011]    One aspect of the invention is to exploit the script associated with a video/movie. 
         [0012]    Another aspect of the invention is to analyze the script to identify a closed-world set of key-phrases. 
         [0013]    Yet another aspect of the invention is to perform coarse-grained annotation of a video based on the closed-world set of key-phrases. 
         [0014]    Another aspect of the invention is to perform coarse-grained annotation of a script. 
         [0015]    Yet another aspect of the invention is to map a key frame of a video scene to one or more script segments based on the coarse-grained annotation of the key frame and the coarse-grained annotation of script segments. 
         [0016]    Another aspect of the invention is to identify the best possible script segment to be mapped onto a video scene. 
         [0017]    Yet another aspect of the invention is to analyze the script segment associated with a video scene to achieve a fine-grained annotation of the video scene. 
         [0018]    Another aspect of the invention is to identify homogeneous video scenes based on the fine-grained annotation of the video scenes of a video. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0019]      FIG. 1  provides an overview of a video search and retrieval system. 
           [0020]      FIG. 2  depicts an illustrative script segment. 
           [0021]      FIG. 3  provides illustrative script and scene structures. 
           [0022]      FIG. 3   a  provides additional information about script and scene structures. 
           [0023]      FIG. 4  provides an approach for closed-world set identification. 
           [0024]      FIG. 5  provides an approach for enhancing script segments. 
           [0025]      FIG. 6  provides an approach for coarse-grained annotation of script segments. 
           [0026]      FIG. 7  depicts an approach for video scene identification and coarse-grained annotation. 
           [0027]      FIG. 7   a  provides illustrative annotations. 
           [0028]      FIG. 8  depicts an overview of deep indexing of a video. 
           [0029]      FIG. 8   a  depicts an approach for deep indexing of a video. 
           [0030]      FIG. 9  provides an approach for segment-scene mapping. 
           [0031]      FIG. 9   a  depicts an illustrative segment mapping. 
           [0032]      FIG. 10  provides an approach for video scene annotation. 
           [0033]      FIG. 11  depicts the identification of homogeneous video scenes. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0034]    Deep annotation and semantic indexing of videos/movies help in providing enhanced and enriched access to content available on web. While this is so, the deep annotation of videos to give access to “bits &amp; pieces” of content poses countless challenges. On the other hand, there has been tremendous work on the shallow annotation of videos although there has not been a great success even at this level. An approach is to exploit any and all of the additional information that is available along with a movie. One such information base is the movie script: The script provides detailed and necessary information about the movie under making; that is, the script is prepared much before the movie shooting. Because of this factor, the script and the made movie may not correspond with each other; that is, the script and the movie may not match one to one. Additionally, it should be noted that the script, from the point of view of the movie, could be outdated, incomplete, and inconsistent. This poses a big challenge in using the textual description of the movie contained in the script. This means that independent video processing is complex and at the same time, independent script processing is also complex. A way to address this two-dimensional complexity is to design a system that bootstraps through incremental analyses. 
         [0035]      FIG. 1  provides an overview of a video search and retrieval system. A video search and retrieval system ( 100 ) helps in searching of a vast collection of video ( 110 ) and provides access to the videos of interest to users. An extremely user friendly interface gets provided when there is a deep and semantic indexing ( 120 ) of the videos. The input user query is analyzed ( 130 ), the analyzed query is evaluated based on deep and semantic indexes ( 140 ). The result of the query evaluation is in the form of (a) videos; (b) video shows (portions of videos stitched together); and (c) video scenes ( 150 ). In order to be able to build deep annotation, the content is analyzed ( 160 ), the script associated with each of the videos is analyzed ( 170 ), and based on these two kinds of analyses, deep annotation of content is determined ( 180 ). Using the deep annotation, it is a natural next step to build semantic indexes for the videos contained in the content database. 
         [0036]      FIG. 2  depicts an illustrative script segment. Note that the provided illustrative script segment highlights the various key aspects such as the structural components and their implicit interpretation. 
         [0037]      FIG. 3  provides an illustrative script and scene structures. 
         [0038]    A video script provides information about a video and is normally meant for to be understood by human beings (for example, shooting crew members) in order to effectively shoot a video.
       Video script prepared prior to shooting of a video provides the information about Locations, Objects, Dialogs, and Actions that would eventually be part of the video;   A Script is organized in terms of segments with each segment providing the detailed information about the shots to be taken in and around a particular location;   A script is a semi-structured textual description;   One of the objectives is to analyze a script automatically to (a) identify script segments; and (b) create a structured representation of each of the script segments;   A video, on the other hand, comprises of a video segments, each video segment comprising multiple video scenes, each video scene comprising multiple video shots with each video shot comprising video frames. Some of the video frames are identified as video key frames;   Another objective is to analyze a video to identify the video segments, video scenes, video shots, and video key frames;   Yet another objective is to map a video scene with a corresponding script segment;   Typically, a script segment describes multiple video scenes;       
 
       Structure of Script: 
       [0047]      
         [0000]    
       
         
               
               
             
               
               
               
             
               
               
             
               
               
               
             
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 Object &lt;NAME&gt; &lt;Description&gt; 
               
               
                   
                 Person &lt;NAME&gt; &lt;Description&gt; 
               
               
                   
                 Location &lt;NAME&gt; &lt;Description&gt; 
               
             
          
           
               
                   
                 &lt;Num&gt; 
                 Int.| Ext. &lt;Location&gt; &lt;Time&gt; 
               
             
          
           
               
                   
                 &lt;SCENE&gt; 
               
               
                   
                 &lt;DIALOG&gt; 
               
               
                   
                 &lt;ACTION&gt; 
               
               
                   
                 &lt;DIRECTIVE&gt; 
               
             
          
           
               
                   
                 &lt;Num&gt; 
                 Int.|Ext. &lt;Location&gt; &lt;Time&gt; 
               
             
          
           
               
                   
                 &lt;SCENE&gt; 
               
               
                   
                 &lt;DIALOG&gt; 
               
               
                   
                 &lt;ACTION&gt; 
               
               
                   
                 &lt;DIRECTIVE&gt; 
               
               
                   
                 ... 
               
               
                   
                   
               
             
          
         
       
     
         [0048]      FIG. 3   a  provides additional information about script and scene structures. 
         [0049]    A video script is based on a set of Key Terms that provide a list of a sort of reserved words with a pre-defined semantics. Similarly, Key Identifiers provide certain placeholders in the script structure for to be filled in appropriately during the authoring of a script.
       Key Terms: OBJECT, PERSON, LOCATION, INTERIOR (NT.), EXTERIOR       
 
         [0051]    (EXT.), DAY, NIGHT, CLOSEUP, FADE IN, HOLD ON, PULL BACK TO REVEAL, INTO VIEW, KEEP HOLDING, . . . ;
       Key Identifiers: &lt;num&gt;, &lt;location&gt; (instances of LOCATION), &lt;time&gt; (one of Key Terms), . . . ;   Use of UPPERCASE to describe instances of OBJECT and PERSON;   Script description is typically in natural language; However, in a semi-structured representation, some additional markers are provided;   &lt;SCENE&gt; to indicate that the text following provides a description of a scene;   &lt;DIALOG&gt; to indicate that the text following provides a description of dialog by a PERSON;   &lt;ACTION&gt; to indicate that the text following provides a description of an action;   &lt;DIRECTIVE&gt; provides additional directions about a shot and is based on Key Terms;       
 
         [0059]      FIG. 4  provides an approach for closed-world set identification. In the process of bootstrapping of deep annotation and semantic indexing, it is essential to perform incremental analysis using a video and the corresponding script. As a first step, it is essential to identify a set of key-phrases based on the given script. This set of key-phrases forms the basis for the video analysis of the video using multimedia processing techniques. 
       Closed-World (CW) Set Identification 
       [0060]    Input: A Video, say a movie, Script
 
Output: A set of key-phrases that defines the closed world for the given video;
 
Step 1: Analyze all the instances of OBJECT and obtain a set of key-phrases, SA, based on, say, Frequency Analysis;
 
Step 2: Analyze all the instances of PERSON and obtain a set of key-phrases, SB, based on, say, Frequency Analysis;
 
Step 3: Analyze all the instances of LOCATION and obtain a set of key-phrases, SC, based on, say, Frequency Analysis;
 
Step 4: Analyze SCENE descriptions and obtain a set of key-phrases based, SD, on, say, Frequency Analysis;
 
Step 5: Analyze DIALOG descriptions and obtain a set of key-phrases, SE, based on, say, Frequency Analysis;
 
Step 6: Analyze ACTION descriptions and obtain a set of key phrases, SF, based on, say, Frequency Analysis;
 
Step 7: Perform consistency analysis on the above sets SA-SF and arrive at a consolidated set of key phrases CW-Set.
 
         [0061]      FIG. 5  provides an approach for enhancing script segments. In a parsimonious representation of a script, typically, the descriptions may not be duplicated. Alternatively, entities are named, described once, and are referred wherever appropriate. However, in order to process the script segments efficiently, as an intermediate step, it is useful to derive a self-contained script segments. Obtain a script segment based on segment header ( 500 ). A typical header includes &lt;NUM&gt; Int.|Ext. &lt;LOCATION&gt; and &lt;TIME&gt; based entities. Analyze the script segment and identify the information about instances of each of the following Key Terms: OBJECT, PERSON, LOCATION ( 510 ). Typically, such instances are indicated in the script in UPPER CASE letters. For each one of the instances, search through the script and obtain the instance description ( 520 ). For example, the description of LOCATION such as MANSION and PERSON such as JOHN. 
         [0062]      FIG. 6  provides an approach for coarse-grained annotation of script segments. In order to effectively bootstrap, it is equally essential to achieve coarse-grained annotation of the script segments. Obtain a script segment SS ( 600 ). Analyze the OBJECT descriptions, PERSON descriptions, LOCATION descriptions, SCENE descriptions, DIALOG descriptions, and ACTION descriptions associated with SS ( 610 ). In order to obtain a coarse-grained annotation, a textual analysis, say, involving term-frequency is sufficient. Determine the coarse-grained annotation of SS based on the analysis results ( 620 ). 
         [0063]      FIG. 7  depicts an approach for video scene identification and coarse-grained annotation. The next step in the bootstrapping processing is to analyze a video to arrive at a coarse-grained annotation of the video. Obtain a video, say, a movie ( 700 ). Analyze the video and obtain several scenes (video scenes) of the video ( 710 ). Analyze each video scene and extract several shots (video shots) ( 720 ). Analyze each video shot and obtain several video key frames ( 730 ). Note that the above analyses are based on the well known structure of a video: segments, scenes, shots, and frames. There are several well established techniques described in the literature to achieve each of these analyses. Analyze and annotate each video key frame based on CW-Set ( 740 ). The closed-world set, CW-Set, plays an important role in the analysis of a key frame leading to a more accurate identification of objects in the key frame and hence, better annotation of the key frame. USPTO application titled “System and Method for Hierarchical Image Processing” by Sridhar Varadarajan, Sridhar Gangadharpalli, and Adhipathi Reddy Aleti (under filing process) describes an approach for exploiting CW-Set to achieve better annotation of a key frame. Based on video key frame annotation of the video key frames of a video shot, determine video shot annotation ( 750 ). There are multiple well known techniques described in the literature to undertake a multitude of analyses of a multimedia content, say, based on audio, video, and textual portion of the multimedia content. U.S. patent application Ser. No. 12/194,787 titled “System and Method for Bounded Analysis of Multimedia using Multiple Correlations” by Sridhar Varadarajan, Amit Thawani, and Kamakhya Gupta (assigned to Satyam Computer Services Ltd. (Hydreabad, India)) describes an approach for annotating of the multimedia content in a maximally consistent manner based on such a multitude of analyses of the multimedia content. Based on video shot annotation of the video shots of a video scene, determine video scene annotation ( 760 ). U.S. patent application Ser. No. 12/199,495 titled “System and Method for Annotation Aggregation” by Sridhar Varadaraj an, Srividya Gopalan, Amit Thawani (assigned to Satyam Computer Services Ltd. (Hydreabad, India)) describes an approach that uses the annotation at a lower level to arrive at an annotation at the next higher level. 
         [0064]      FIG. 7   a  provides illustrative annotations. 
         [0065]    Video analysis to arrive annotations makes use of multiple techniques, some are based on image processing, some are based on text processing, and some are on audio processing.
       Based on Image Processing (Video key frame level)
           Indoor/Outdoor identification   Day/Night classification   Bright/Dark image categorization   Natural/Manmade object identification   Person identification (actors/actresses)   
           Based on Audio Processing (Video scene level)
           Speaker recognition (actors/actresses)   Keyword spotting (Dialogs)   Non-speech sound identification (objects)   
               
 
         [0076]      FIG. 8  depicts an overview of deep indexing of a video. 
         [0077]    Given:
       Script segments—SS 1 , SS 2 , . . .   Video scenes—VS 1 , VS 2 , . . .   Coarse-grained annotations associated with video scenes—CA 1 , CA 2 , . . .       
 
         [0081]    For each scene VSi, Generate a fine-grained annotation FAi 
         [0082]    The process of deep annotation receives a content described in terms of a set of video scenes as input and uses the script described in terms of a set of script segments which is associated with the content and a set of coarse-grained annotations associated the set of video scenes to arrive at a fine-grained annotation for each of the video scenes. 
         [0083]      FIG. 8   a  depicts an approach for deep indexing of a video. 
         [0084]    Deep Annotation and Semantic Indexing 
         [0000]    Given: Script segments and video scenes
 
Note: Multiple video scenes correspond with a script segment
 
Step 1: Based on script structure, identify script segments and make each segment complete by itself;
 
Step 2: Analyze input script and generate a closed world set (CW-Set) of key-phrases;
 
Step 3: Use CW-Set and annotate each video key frame (VKFi) of each video scene VSi;
 
Step 4: For each VKFi of VSi, based on VKFAi (video key frame annotation), identify K matching script segments (SSj&#39;s) based on coarse-grained annotation associated with each script segment. This step accounts for both inaccuracy in the coarse-grained annotation and outdatedness of the script.
 
Step 4a: Apply a warping technique to identify the best possible script segment that matches with most of the key frames of the video scene VSi.
 
Step 5: Analyze the script segment associated with VSi to generate VSAi (video scene annotation). Note that this step employs a multitude of semi-structured text processing to arrive at an annotation of the video scene.
 
Step 7: Identify homogeneous video scenes called video shows based on VSA&#39;s. A typical way to achieve this is to use a clustering technique based on the annotation of the video scenes. The identified clusters tend to identify video scenes that have similar annotations and hence, the corresponding scenes are also similar as well.
 
         [0085]      FIG. 9  provides an approach for segment-scene mapping. 
       Segment-Scene Mapping 
       [0086]    Given: Video scene VS
       X Key frames of VS: VKF 1 , VKF 2 , . . . , VKFi, . . .   Corresponding annotations: VKFA 1 , VKFA 2 , . . . , VKFAi, . . .   K segments associated with VKF 1 : SS 11 , SS 12 , . . . , SS 1   j, . . .      K segments associated with VKF 2 : SS 21 , SS 22 , . . . , SS 2 ij . . .   . . .   K segments associated with VKFi: SSi 1 , SSi 2 , . . . , SSij, . . .
 
Note that above multiple sets of K segments form a Segment-KeyFrame Matrix;
   Each of the K segments are arranged in the non-increasing order of their closeness with the corresponding VFKA; Note that the key frame annotations are matched with the script segments, wherein each of the script segments is also described in manner so as to be able to match with the key frame annotations. In one of the embodiments, text processing techniques are used on the semi-structured script segment to arrive at a suitable coarse-grained annotation.   Each segment SSij is associated with a positional weight:
           SS 11 , SS 21 , . . . , SSi 1 , . . . are associated with a positional weight of 1;   SS 12 , SS 22 , . . . , SSi 2 , . . . are associated with a positional weight of 2;   SS 1   j , SS 2   j , . . . , Ssij, . . . are associated with a positional weight of J;   
               
 
       Step 1: 
       [0098]    Start from SS 11  and generate a sequence of X positional weights as follows:
       With respect to VKF 1 : positional weight is 1;   With respect to VKF 2 : search through the K segments SS 21 , SS 22 , . . . and locate the position of the SS 11 ; If found (SS 2   i ), the positional weight is the corresponding positional weight of SS 2   i  and Mark SS 2   i ; If not found, the positional weight is K+1;   Similarly, obtain the matching positional weights with respect to the other key frames;       
 
       Step 2: 
       [0000]    
       
         
           
             Repeat Step 1 for each of the segments SS 12 , . . . , SS 1   j, . . . ;    
           
         
       
     
       Step 3: 
       [0000]    
       
         
           
             Scan the Segment-KeyFrame matrix in a column major order and locate unmarked SSab; 
             Repeat Step 1 with respect to SSab; 
           
         
       
     
       Step 4: 
       [0000]    
       
         
           
             Repeat Step 3 until there are no more unmarked segments; 
             Note that in these cases, the initial positional weights are K+1; 
             Note also there are totally Y sequences and each such sequence is called as an ISOSEGMENTAL line; 
           
         
       
     
       Step 5: 
       [0000]    
       
         
           
             Generate an error sequence for each of the above Y sequences by subtracting unity from the sequence values; 
           
         
       
     
       Step 6: 
       [0000]    
       
         
           
             Determine the IsoSegmental line with least error; If there are multiple such lines, for each line, determine the minimum number of key frames to be dropped to get an overall error value of close to 0; Choose the line with minimum number of drops; 
           
         
       
     
       Step 7: 
       [0000]    
       
         
           
             The determined IsoSegmental line SSj&#39; defines the mapping onto VS; 
           
         
       
     
         [0111]      FIG. 9   a  depicts an illustrative segment mapping. For illustrative purposes, consider the following ( 900 ): 
         [0000]    Video Scene: 1; VS 1  is the video scene.
 
Number of key frames: 6; VKF 11 , VKF 12 , VKF 13 , VKF 14 , VKF 15 , and VKF 16  are the illustrative key frames.
 
Number of segments per key frame: 5; That is, the top 5 of the matched segments are selected for further analysis.
 
Total number of segments: 20
 
         [0112]      910  depicts the best matched segment (SS  5 ) with respect to the key frame VKF 11  while  920  depicts the second best matched segment (SS  6 ) with respect to the key frame VKF 16 . There are totally 7 IsoSegmental lines and  930  depicts first of them.  940  depicts the various computations associated with the first IsoSegmental line.  950  indicates the script segment number (SS  5 ),  960  indicates the positional weight sequence,  970  depicts the associated error sequence, and  980  provides the error value. Based on the error value associated with the 7 IsoSegmental lines, the IsoSegmental line 2 with the least error is selected as the best matched segment (SS  6 ) for mapping onto VS 1 . 
         [0113]      FIG. 10  provides an approach for video scene annotation. The approach uses the description of the portions of the script segment that matches best with a video scene. 
       Video Scene Annotation 
       [0000]    
       
         
           
             Given:
           Video scene VS; Multiple video key frames VKF 1 , VKF 2 , . . . ;   Video key frame annotations VKFA 1 , VKFA 2 , . . . ;   Mapped Script segment SS;   
         
             Output: Video scene annotation VSA;
 
SS comprises
 
             instances of Object (O 1 , O 2 , . . . ), Person (P 1 , P 2 , . . . ), and Location (L 1 , L 2 , . . . ); 
             Multiple descriptions involving &lt;SCENE&gt; (S 1 , S 2 , . . . ), &lt;DIALOG&gt; (D 1 , D 2 , . . . ), and &lt;ACTION&gt; (A 1 , A 2 , . . . ); 
             Multiple &lt;DIRECTIVES&gt; 
             As a consequence, a script segment is described in terms of these multiple instances and multiple descriptions as follows: 
           
         
         SSP={O 1 , O 2 , . . . , P 1 , P 2 , . . . , L 1 , L 2 , . . . , S 1 , S 2 , . . . , D 1 , D 2 , . . . , A 1 , A 2 , . . . } describes SS; In other words, each element of SSP defines a portion of the script segment; 
         Note: SS can map onto one or more video scenes; This means that VS needs to be annotated based on one or more portions of SS;
 
Step 1: Based on video key frame annotations, determine the matching of the various portions of SSP with respect to the key frames of VS ( 1000 ): For example,  1010  depicts how a portion of SS based on the description of an instance of Object O 1  matches with the various key frames; here, “O” denotes a good match while “X” denotes not so much of a good match; similarly,  1020  depicts how a key frame matches with the various portions of SS.  1030  depicts the counts indicating how well each of the portions of SS matches with respect to all of the key frames. Note that the counts provide information about the portions of SS matching with VS;
 
Step 2: Analyze each of the counts: CO 1 , CO 2 , . . . , CP 1 , . . . ;
       If a count of Cxy exceeds a pre-defined threshold, make the corresponding portion of SS a part of SSPVS; Note that SSPVS is a subset of SSP;
 
Step 3: Analyze SSPVS and determine multiple SVO triplets for each of the elements of SSPVS; Note that the portions of a script segment are described typically in a natural language such as English. Here, an SVO triplet stands for &lt;Subject, Verb, Object&gt; that is part of any sentence of a portion of the script segment. The natural language analysis is greatly simplified by the fact the scripts provide positive descriptions.
 
Step 4: Make the determined SVO triplets a part of VSA;
   
     
       
     
         [0126]      FIG. 11  depicts the identification of homogeneous video scenes. 
       Identification of Homogeneous Scenes 
     Given: 
       [0000]    
       
         
           
             A set of video scenes: SVS={VS 1 , VS 2 , . . . }; 
             Each VSi is associated with an annotation VSAi; 
           
         
       
     
         [0129]    Note that VSAi is a set with each element providing information in the form of SVO triplets associated with an OBJECT, PERSON, LOCATION, SCENE, DIALOG, or ACTION; 
         [0130]    Primarily, there are six dimensions: OBJECT dimension, PERSON dimension, LOCATION dimension, SCENE dimension, DIALOG dimension, and ACTION dimension; 
       Step 1: 
       [0000]    
       
         
           
             To begin with homogeneity is defined with respect to each of the dimensions: OBJECT dimension homogeneity:
           Form OS based on SVO triplets associated with each element of SVS such that the SVO triplets are related to the instances of OBJECT;   Cluster OS based on the similarity along the properties (that is, SVO triplets) of the instances of OBJECT into OSC 1 , OSC 2 , . . . ;   Based on the associated SVO triplets in each of OSC 1 , OSC 2 , . . . , obtain the corresponding video scenes from SVS;
               That is, each OSCi identifies one or more video scenes;   
               
         
           
         
       
     
       Step 2: 
       [0000]    
       
         
           
             Repeat Step 1 for the other five dimensions; 
           
         
       
     
       Step 3: 
       [0000]    
       
         
           
             Identify combinations of the dimensions that have high query utility; 
             Repeat Step 1 for each such combination; 
           
         
       
     
         [0139]    In order to identify homogeneous scenes, two things are essential: one is a homogeneity factor and the second thing is a similarity measure. The homogeneity factor provides an abstract and computational description of a set of homogeneous scenes. For example, OBJECT dimension is an illustration of a homogeneity factor. The similarity measure, on the other hand, defines how two video scenes along the homogeneity factor correlate with each other. For example, term by term matching of two SVO triplets is an illustration of a similarity measure. 
         [0140]    Thus, a system and method for deep annotation and semantic indexing is disclosed. Although the present invention has been described particularly with reference to the figures, it will be apparent to one of the ordinary skill in the art that the present invention may appear in any number of systems that need to overcome the complexities associated with deep textual processing and deep multimedia analysis. It is further contemplated that many changes and modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the present invention.