Patent Publication Number: US-2013251340-A1

Title: Video concept classification using temporally-correlated grouplets

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     Reference is made to commonly assigned, co-pending U.S. patent application Ser. No. 13/269,742 (Docket K000505), entitled: “Video concept classification using audio-visual grouplets”, by Jiang et al.; and to commonly assigned, co-pending U.S. patent application Ser. No. 13/269,753 (Docket K000652), entitled: “Video concept classification using video similarity scores”, by Jiang et al., both of which are incorporated herein by reference. 
    
    
     FIELD OF THE INVENTION 
     This invention pertains to the field of video classification and more particularly to a method for using a method for using a temporally-correlated grouplet representation to classify semantic concepts in videos. 
     BACKGROUND OF THE INVENTION 
     The rapidly increasing amount of digital multimedia data (e.g., digital still images and digital videos) makes the automatic classification of multimedia content an important problem. A wide range of semantic concepts can be used to represent multimedia content, such as objects (e.g., dog), scenes (e.g., beach) and events (e.g., birthday). Semantic concept classification in generic, unconstrained videos (e.g., videos captured by consumers and posted on YouTube) is a difficult problem, because such videos are captured in an unrestricted manner. These videos have diverse video content, as well as challenging conditions such as uneven lighting, clutter, occlusions, and complicated motions of both objects and camera. 
     A lot of effort has been devoted to developing methods to classify general semantic concepts in generic videos. Examples of proposed approaches include the TRECVid high-level feature extraction method described by Smeaton et al. in the article “Evaluation campaigns and TRECVid” (Proc. 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321-330, 2006) and the Columbia Consumer Video (CCV) concept classification method described by Jiang et al. in the article “Consumer video understanding: A benchmark database and an evaluation of human and machine performance” (Proc. 1st ACM International Conference on Multimedia Retrieval, 2011). 
     Most prior art approaches classify videos in the same way they classify images, using mainly visual information. Specifically, visual features are extracted from either two-dimensional (2-D) keyframes or three-dimensional (3-D) local volumes, and these features are treated as individual static descriptors to train concept classifiers. Among these methods, the ones using the “Bag-of-Words” (BoW) representation over 2-D or 3-D local descriptors (e.g., SIFT) are considered state-of-the-art, due to the effectiveness of BoW features in classifying objects and human actions. 
     The importance of incorporating audio information to facilitate semantic concept classification has been discovered by several previous works. (For example, see the aforementioned article by Jiang et al. entitled “Consumer video understanding: A benchmark database and an evaluation of human and machine performance.”) Such approaches generally use a multi-modal fusion strategy (e.g., early fusion to train classifiers with concatenated audio and visual features, or late fusion to combine judgments from classifiers built over individual modalities). 
     Cristani et al. in the article “Audio-visual event recognition in surveillance video sequences” (IEEE Transactions Multimedia, Vol. 9, pp. 257-267, 2007) describe a video classification method that integrates audio and visual information for scene analysis in a typical surveillance scenario. Visual information is analyzed to detect visual background and foreground information, and audio information is analyzed to detect audio background and foreground information. The integration of audio and visual data is subsequently performed by exploiting the concept of synchrony between such events. 
     The importance of incorporating textual information to facilitate semantic concept classification has previously been investigated. Such methods generally use the multi-modal fusion strategy, especially late fusion, which combines judgments from classifiers that are individually trained over visual and textual modalities, respectively (for example, see the article by Lana-Serrano et al. entitled “DAEDALUS at ImageCLEF medical retrieval 2011: Textual, visual and multimodal experiments,” CLEF 2011 Conference on Multilingual and Multimodal Information Access Evaluation, 2011). Some other approaches use textual information as side information or prior knowledge to help with analyzing the data structure for enhanced classification (for example, see the article by Jiang et al. entitled “Visual saliency with side information,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1765-1768, 2009). 
     There remains a need for a video classification method that better leverages temporal textual-visual or temporal textual-audio correlation in order to provide more reliable and more efficient semantic classification. 
     SUMMARY OF THE INVENTION 
     The present invention represents a method for determining a semantic concept classification for a digital video clip including a temporal sequence of video frames, a corresponding audio soundtrack and a corresponding temporal sequence of textual information, comprising: 
     a) receiving a grouplet dictionary including a plurality of temporally-correlated grouplets, the temporally-correlated grouplets including dictionary textual codewords representing textual information content, and either dictionary visual codewords representing visual content or dictionary audio codewords representing audio content or both, wherein the dictionary textual codewords, dictionary visual codewords and dictionary audio codewords in a particular temporally-correlated grouplet were determined to be correlated with each other during a training process based on an analysis of a plurality of training videos; 
     b) determining reference video codeword similarity scores for each reference video clip in a set of reference video clips by:
         i) analyzing the temporal sequence of textual information for a particular reference video clip to determine a set of reference video textual features;   ii) performing at least one of analyzing the temporal sequence of video frames for a particular reference video clip to determine a set of reference video visual features and analyzing the audio soundtrack for the particular reference video clip to determine a set of reference video audio features; and   iii) determining the reference video codeword similarity scores for the particular reference video clip by comparing the set of determined reference video textual features to the dictionary textual codewords, comparing any determined reference video visual features to the dictionary visual codewords and comparing any determined reference video audio features to the dictionary audio codewords;       

     c) determining codeword similarity scores for the digital video clip by:
         i) analyzing the temporal sequence of textual information for the digital video clip to determine a set of textual features;   ii) performing at least one of analyzing the temporal sequence of video frames in the digital video clip to determine a set of visual features and analyzing the audio soundtrack in the digital video clip to determine a set of audio features;   iii) determining the codeword similarity scores for the digital video clip by comparing the set of determined textual features to the dictionary textual codewords, comparing any determined visual features to the dictionary visual codewords and comparing any determined audio features to the dictionary audio codewords;       

     d) determining a reference video similarity score for each reference video clip representing a similarity between the digital video clip and the reference video clip responsive to the temporally-correlated grouplets, the codeword similarity scores and the reference video codeword similarity scores; 
     e) determining one or more semantic concept classifications using trained semantic classifiers responsive to the determined reference video similarity scores; and 
     f) storing indications of the one or more semantic concept classifications in a processor-accessible memory; 
     wherein the method is performed at least in part using a data processor. 
     This invention has the advantage that significant classification performance improvements can be achieved from the temporally-correlated grouplet representation. Each temporally-correlated grouplet contains a set of dictionary textual codewords and dictionary audio codewords or dictionary visual codewords that have strong temporal correlations in videos. The temporally-correlated grouplets capture not only the individual textual, audio and visual features carried by the discrete textual, audio and visual codewords, but also the temporal relations between textual, audio and visual channels. By using the entire temporally-correlated grouplets as building elements to represent videos, various concepts can be more robustly classified than using discrete textual, audio and visual codewords. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a high-level diagram showing the components of a system for determining a semantic concept classification for a digital video clip according to an embodiment of the present invention; 
         FIG. 2  is a flow diagram illustrating a method for forming audio-visual dictionaries and training semantic concept classifiers in accordance with the present invention; 
         FIG. 3  shows a more detailed flow diagram of the visual processing module in  FIG. 1  in accordance with a preferred embodiment; 
         FIG. 4  shows a more detailed flow diagram of the audio processing module in  FIG. 1  in accordance with a preferred embodiment; 
         FIG. 5  shows a more detailed flow diagram of the audio-visual temporal causal analysis module in  FIG. 1  in accordance with a preferred embodiment; 
         FIG. 6  is a flow diagram for determining a semantic concept classification for an input digital video in accordance with a preferred embodiment; 
         FIG. 7A  is an example illustrating the structure of audio-visual grouplets from an audio-visual dictionary and the computation of dictionary-based similarity scores; 
         FIG. 7B  is illustrates the computation of dictionary-based similarity scores according to an alternate embodiment; 
         FIG. 8  is a graph comparing the performance of the semantic concept classification method of  FIG. 6  with some other state-of-the-art approaches; 
         FIG. 9  is a flow diagram illustrating a method for forming temporally-correlated grouplets and training semantic concept classifiers in accordance with an embodiment of the present invention; 
         FIG. 10  shows a more detailed flow diagram of the grouplet construction step in  FIG. 9  in accordance with the present embodiment; 
         FIG. 11  shows a more detailed flow diagram of the distance metric learning step in  FIG. 9  in accordance with the present embodiment; 
         FIG. 12  is a flow diagram for determining a semantic concept classification for an input digital video in accordance with an embodiment of the present invention; and 
         FIG. 13  is graphs comparing the performance of the semantic concept classification method of  FIG. 12  with some other state-of-the-art approaches. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     In the following description, some embodiments of the present invention will be described in terms that would ordinarily be implemented as software programs. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention. Other aspects of such algorithms and systems, together with hardware and software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein may be selected from such systems, algorithms, components, and elements known in the art. Given the system as described according to the invention in the following, software not specifically shown, suggested, or described herein that is useful for implementation of the invention is conventional and within the ordinary skill in such arts. 
     The invention is inclusive of combinations of the embodiments described herein. References to “a particular embodiment” and the like refer to features that are present in at least one embodiment of the invention. Separate references to “an embodiment” or “particular embodiments” or the like do not necessarily refer to the same embodiment or embodiments; however, such embodiments are not mutually exclusive, unless so indicated or as are readily apparent to one of skill in the art. The use of singular or plural in referring to the “method” or “methods” and the like is not limiting. It should be noted that, unless otherwise explicitly noted or required by context, the word “or” is used in this disclosure in a non-exclusive sense. 
       FIG. 1  is a high-level diagram showing the components of a system for determining a semantic concept classification according to an embodiment of the present invention. The system includes a data processing system  10 , a peripheral system  20 , a user interface system  30 , and a data storage system  40 . The peripheral system  20 , the user interface system  30  and the data storage system  40  are communicatively connected to the data processing system  10 . 
     The data processing system  10  includes one or more data processing devices that implement the processes of the various embodiments of the present invention, including the example processes described herein. The phrases “data processing device” or “data processor” are intended to include any data processing device, such as a central processing unit (“CPU”), a desktop computer, a laptop computer, a mainframe computer, a personal digital assistant, a Blackberry™, a digital camera, cellular phone, or any other device for processing data, managing data, or handling data, whether implemented with electrical, magnetic, optical, biological components, or otherwise. 
     The data storage system  40  includes one or more processor-accessible memories configured to store information, including the information needed to execute the processes of the various embodiments of the present invention, including the example processes described herein. The data storage system  40  may be a distributed processor-accessible memory system including multiple processor-accessible memories communicatively connected to the data processing system  10  via a plurality of computers or devices. On the other hand, the data storage system  40  need not be a distributed processor-accessible memory system and, consequently, may include one or more processor-accessible memories located within a single data processor or device. 
     The phrase “processor-accessible memory” is intended to include any processor-accessible data storage device, whether volatile or nonvolatile, electronic, magnetic, optical, or otherwise, including but not limited to, registers, floppy disks, hard disks, Compact Discs, DVDs, flash memories, ROMs, and RAMs. 
     The phrase “communicatively connected” is intended to include any type of connection, whether wired or wireless, between devices, data processors, or programs in which data may be communicated. The phrase “communicatively connected” is intended to include a connection between devices or programs within a single data processor, a connection between devices or programs located in different data processors, and a connection between devices not located in data processors at all. In this regard, although the data storage system  40  is shown separately from the data processing system  10 , one skilled in the art will appreciate that the data storage system  40  may be stored completely or partially within the data processing system  10 . Further in this regard, although the peripheral system  20  and the user interface system  30  are shown separately from the data processing system  10 , one skilled in the art will appreciate that one or both of such systems may be stored completely or partially within the data processing system  10 . 
     The peripheral system  20  may include one or more devices configured to provide digital content records to the data processing system  10 . For example, the peripheral system  20  may include digital still cameras, digital video cameras, cellular phones, or other data processors. The data processing system  10 , upon receipt of digital content records from a device in the peripheral system  20 , may store such digital content records in the data storage system  40 . 
     The user interface system  30  may include a mouse, a keyboard, another computer, or any device or combination of devices from which data is input to the data processing system  10 . In this regard, although the peripheral system  20  is shown separately from the user interface system  30 , the peripheral system  20  may be included as part of the user interface system  30 . 
     The user interface system  30  also may include a display device, a processor-accessible memory, or any device or combination of devices to which data is output by the data processing system  10 . In this regard, if the user interface system  30  includes a processor-accessible memory, such memory may be part of the data storage system  40  even though the user interface system  30  and the data storage system  40  are shown separately in  FIG. 1 . 
     As discussed earlier, most prior art video classification methods that incorporate both video and audio information generally use a multi-modal fusion strategy which do not leverage temporal audio-visual correlation. The present invention takes advantage of the fact that temporal audio-visual dependencies can reveal unique audio-visual patterns to assist with automatic semantic concept classification. The correlations between temporal patterns of visual and audio codewords provide important discriminative audio-visual cues, such as the unique encapsulation of visual basketball patches and audio basketball bouncing sounds for classifying the “basketball” concept. Similarly, the encapsulation of visual stadium features and a audio music sounds can be used to classify a “music performance” concept. Such audio-visual cues have not been studied before in previous literatures. 
     From another perspective, beyond the traditional BoW representation, structured visual features have been recently found to be effective in many computer vision tasks. In addition to the local feature appearance, spatial relations among local patches are incorporated to increase the robustness of the visual representation. The rationale behind this is that individual local visual patterns tend to be sensitive to variations such as changes of illumination, views, scales, and occlusions. In comparison, a set of co-occurrent local patterns can be less ambiguous. 
     In accordance with the present invention, four types of Audio-Visual Grouplets (AVGs) are used to represent four types of temporal audio-visual correlations: correlations between visual foreground and audio foreground; correlations between visual background and audio background; correlations between visual foreground and audio background; and correlations between visual background and audio foreground. All of these types of AVGs are useful for video concept classification. For example, to effectively classify a “birthday” semantic concept, all of the following factors are important: the visual foreground (e.g., child), the visual background setting (e.g., cake and table), the audio foreground sound (e.g., cheering, birthday song, and hand clapping), and the audio background sound (e.g., music). By evaluating the temporal audio-visual correlations among these factors, we can identify unique audio-visual patterns that are discriminative for “birthday” classification. 
     To enable the exploitation of the foreground and background audio-visual correlations, coarse-level separation of the foreground and background is needed in both visual and audio channels. It is worth mentioning that due to the diverse video content and the challenging conditions (e.g., uneven lighting, clutter, occlusions, complicated objects and camera motions, and the unstructured audio sounds with overlapping acoustic sources), precise visual or audio foreground/background separation is not feasible in generic videos. In addition, exact audio-visual synchronization can be unreliable much of the time. Multiple moving objects usually make sounds together, and often the object making sounds does not synchronically appear in video. 
     To accommodate these issues, different from most previous audio-visual analysis methods such as the aforementioned method described by Cristani, et al. in the article “Audio-visual event recognition in surveillance video sequences,” IEEE Transactions Multimedia, Vol. 9, pp. 257-267, 2007) that rely on precisely separated visual foreground objects and/or audio foreground sounds, the method of the present invention has the following characteristics:
         Statistical temporal audio-visual correlations are evaluated over a set of videos instead of relying on the exact audio-visual synchronization in individual videos. By representing temporal sequences of visual and audio codewords as multivariate point processes, the statistical pair-wise nonparametric Granger causality between audio and visual codewords is analyzed. Based on the audio-visual causal matrix, salient AVGs are identified, which encapsulate strongly correlated visual and audio codewords as building blocks to classify videos.   The method does not rely on precise visual foreground/background separation. The aim is to build foreground-oriented and background-oriented visual vocabularies. In accordance with the present invention, consistent local points are tracked throughout each video. Based on both local motion vectors and spatiotemporal analysis of whole images, the point tracks are separated into foreground tracks and background tracks. Due to the challenging conditions of generic videos, such a separation is not precise. The goal is to maintain a majority of foreground (or background) tracks so that the constructed visual foreground (or background) vocabulary can capture mainly visual foreground (or background) information.   Similar to the visual aspect, the aim of audio processing is to build foreground-oriented and background-oriented audio vocabularies, instead of pursuing precisely separated audio foreground or background acoustic sources. In generic videos, the foreground sound events are typically distributed unevenly and sparsely. Therefore, a local representation that focuses on short-term transient sound events is used to capture the foreground audio information. Also, Mel-Frequency Cepstral Coefficients (MFCCs) extracted from uniformly spaced audio windows are used to roughly capture the overall information of the environmental sound. Based on the local representation and MFCCs, audio foreground and background vocabularies are constructed, respectively.       

     The present invention will now be described with reference to  FIGS. 2-8 .  FIG. 2  is a high-level flow diagram illustrating a preferred embodiment of a training process for determining audio-visual dictionaries  180  and training semantic concept classifiers  190  in accordance with the present invention. 
     Given an input video  100 , some preprocessing steps are first conducted. A shot boundary detection step  105  is used to analyze the video  100  to detect shot boundaries between different scenes in the video  100 , and to segment the video  100  into correspond parts, with a single shot in each part. A variety of methods for performing the shot boundary detection step  105  are known in the video analysis art, and any such method can be used in accordance with the present invention. 
     A bad shot elimination step  110  is used to detect video shots with very large camera motion, and eliminate them from the set of video shots so that they are excluded from further analysis. In some embodiments, other types of bad shots can also be detected and eliminated, such as out-of-focus shots and poorly illuminated shots. A variety of methods for performing the bad shot elimination step  110  are known in the video analysis art, and any such method can be used in accordance with the present invention. 
     The output of the preprocessing steps is a set of video shots  115  having relatively smooth motion that is used for further analysis. Each video shot  115  includes an image sequence  120  comprising a temporal sequence of video frames, together with a corresponding audio soundtrack  125 . 
     A visual processing module  130  is used to automatically analyze the image sequence  120  to generate visual foreground temporal features  140  and visual background temporal features  145 . Similarly, an audio processing module  150  is used to automatically analyze the audio soundtrack  125  to generate audio foreground temporal features  160  and audio background temporal features  165 . Next, an audio-visual temporal causal analysis module  170  is applied to the audio and visual foreground and background features to generate audio-visual dictionaries  180 . 
     An extract features step  182  is used to generate dictionary-based features  185  from the audio-visual dictionaries  180 . A train classifiers step  187  is used to train semantic concept classifiers  190  based on the dictionary-based features  185 . The semantic concept classifiers  190  are adapted to classify semantic concepts in videos. In a preferred embodiment, the semantic concept classifiers  190  are well-known Support Vector Machine (SVM) classifiers. Methods for training SVM classifiers are well-known in the image analysis art. 
     Additional details for a preferred embodiment of the visual processing module  130  will now be discussed with reference to  FIG. 3 . Given the input image sequence  120 , a SIFT feature extraction step  200  is used to extract a set of SIFT features from a set of uniformly sampled image frames from the image sequence. The image frames are sampled with a sampling rate of r fps (frames per second). The SIFT feature extraction step  200  can extract the SIFT features using any method known in the art. For example, in some embodiments, the 128-dim SIFT descriptor with the DoG interest point detector described by Lowe in the article “Distinctive image features from scale-invariant keypoints” (Int. Journal of Computer Vision, Vol. 60, pp. 91-110, 2004) can be used in accordance with the present invention. 
     Next, a SIFT feature matching step  205  is used to find pairs of matching SIFT features for adjacent image frames based on the Euclidean distance of their feature vectors. Ambiguous matches are discarded using the method described in the aforementioned article by Lowe. After that, the matching pairs are connected along the temporal dimension into a set of SIFT point tracks  210 , where different SIFT point tracks  210  can start from different image frames and last variable lengths. The sampling rate r is empirically determined by considering both the computation cost and the ability of the SIFT feature matching operation. In general, increasing the sampling rate will decrease the chance of missing point tracks, with the price of increased computation. 
     Each SIFT point track  210  is represented by a feature vector  215 . In one embodiment, the feature vectors  215  comprise a 128-dim SIFT vector concatenated with an 8-dim motion vector. The SIFT vector is the average of the SIFT features for all SIFT points in the corresponding SIFT point track  210 . The motion vector is an averaged Histogram of Oriented Motion (HOM) determined along the track. That is, for each adjacent matching pair in the track, the speed and direction of the local motion vector are computed. By quantizing the 2-D motion space into 8 bins (corresponding to 8 directions), an 8-dim HOM feature is computed where the value over each bin is the averaged speed of the motion vectors from the track moving along this direction. 
     Once the set of SIFT point tracks  210  and the corresponding feature vectors  215  are obtained, the SIFT point tracks  210  can be separated as foreground or background using the following two steps. First, a hierarchical clustering step  220  is used to roughly separate the SIFT point tracks  210  into candidate foreground SIFT point tracks  225  and candidate background SIFT point tracks  230 . This is accomplished by sequentially evaluating pairs of adjacent frames I i  and I i+1 . Matching SIFT pairs corresponding to the SIFT point tracks  210  that are included in both of the adjacent frames are extracted and are clustered into candidate foreground SIFT pairs and background SIFT pairs based on the motion vectors. Specifically, the matching SIFT pairs are grouped using a hierarchical clustering process, where the grouping criterion is that SIFT pairs within a cluster have roughly the same moving direction and speed. Those SIFT pairs in the biggest cluster are treated as candidate background SIFT pairs, and all other SIFT pairs are treated as candidate foreground SIFT pairs. The rationale is that foreground moving objects usually occupy less than half of the entire screen, and points on the foreground objects do not have a very consistent moving pattern. In comparison, points on the static background generally have consistent motion and this motion is caused by camera motion. 
     After all the pairs of adjacent frames have been evaluated, some of the SIFT pairs corresponding to a particular SIFT point track  210  may have been labeled as candidate foreground SIFT pairs, while others may have been labeled as candidate background SIFT pairs. A “voting” process is then used to designate the particular SIFT point track  210  as either a candidate foreground SIFT point track  225  or a candidate background SIFT point track  230 . In some embodiments, if more than predefined fraction (e.g., 50%) of the SIFT pairs for a particular SIFT point track  210  are labeled as candidate foreground SIFT pairs, then the particular SIFT point track  210  is designated as a candidate foreground SIFT point track  225 , otherwise it is designated as a candidate background SIFT point track  230 . The hierarchical clustering step  220  can distinguish background tracks fairly well for videos with moderate planar camera motions that occur most commonly in generic videos. 
     Next, a refine SIFT point tracks step  235  is used to further refine the candidate foreground SIFT point tracks  225  and the candidate background SIFT point tracks  230  to determine foreground SIFT point tracks  245  and background SIFT point tracks  250 . In a preferred embodiment, the refine SIFT point tracks step  235  uses a spatiotemporal image representation  240  of the image sequence  120  to guide the refining process. 
     In some embodiments, the spatiotemporal image representation  240  is the so-called “X-ray” image representation described by Joly et al. in the article “Efficient automatic analysis of camera work and micro-segmentation of video using spatiotemporal images” (Signal Processing: Image Communication, Vol. 8, pp. 295-307, 1996). In the X-ray image representation, the average of each row and each column in successive image frames are computed. The distribution of the angles of edges in the X-ray image representations can be matched to camera work models, from which camera motion classification and temporal video segmentation can be obtained directly. When used alone, such methods cannot generate satisfactory segmentation results in many generic videos where large motions from multiple objects cannot be easily discriminated from the noisy background motion. The performance drops even more for small resolutions (e.g., 320×240 for many videos). Therefore, instead of using the spatiotemporal analysis method to pursue precise spatiotemporal object segmentation, it is used to refine the candidate foreground and background SIFT tracks. The spatiotemporal image representation  240  is able to capture camera zoom and tilt information, which the refine SIFT point tracks step  235  can use to rectify those candidate tracks that are mistakenly labeled as foreground due to camera zoom and tilt. 
     Based on the foreground SIFT point tracks  245  and the background SIFT point tracks  250 , a visual foreground vocabulary  260  and a visual background vocabulary  265  can be built, respectively. Also, BoW features can be computed using the vocabularies, which can be used directly for concept classification. Additionally, the temporal patterns of codeword occurrences can be computed to study the correlations between audio and visual signals. 
     From the previous description, each of the foreground SIFT point tracks  245  and the background SIFT point tracks  250  is represented by a 136-dim feature vector  215 . All foreground SIFT point tracks  245  from the training videos are collected together, based on which, a clustering step  255  is used to determine a Dcodeword visual foreground vocabulary  260  (V f−v ). The clustering step  255  can use any clustering technique know in the art. In a preferred embodiment, the clustering step  255  uses a hierarchical K-means clustering method. Similarly, all background SIFT point tracks  250  from the training videos can be collected together and a clustering step  256  is used to construct a D-codeword visual background vocabulary  265  (V b−v ). 
     For each video V j , all of its foreground SIFT point tracks  245  are matched to the visual foreground codewords. A soft weighting scheme can be used to alleviate the quantization effects, and D-dim visual foreground BoW features  270  (F j   f−v ) are generated. Similarly, all of the background SIFT point tracks  250  are matched to the visual background codewords to generate D-dim visual background BoW features  275  (F j   b−v ). In general, both F j   f−v  and F j   b−v  have their impacts in classifying concepts (e.g., both the foreground people with caps and gowns and the background stadium setting are useful to classify “graduation” videos). 
     To study the temporal audio-visual correlations, the following histogram feature is computed over time for each of the visual foreground vocabulary  260  and visual background vocabulary  265 . Given a video V j , a set of foreground SIFT point tracks  245  is obtained. Each foreground SIFT point track  245  is labeled with the visual foreground codeword in the visual foreground vocabulary  260  (V f−v ) that is closest to the track in the visual feature space. Next, for each frame I ji  in the video, one can count the occurring frequency of each visual foreground codeword labeled to the foreground SIFT point tracks  245  that have a SIFT point falling in this frame, and a D-dim histogram H ji   f−v  can be generated. Similarly, a D-dim histogram H ji   b−v  can be generated for each image frame I ji  based on the visual background vocabulary  265  (V b−v ). After this computation, for each video V j , the visual foreground temporal feature  140  is obtained as {H j1   f−v , H j2   f−v , . . . }. Similarly, the visual background temporal feature  145  is also obtained as {H j1   f−v , H j2   f−v , . . . }. 
     Additional details for a preferred embodiment of the audio processing module  150  in  FIG. 2  will now be discussed with reference to  FIG. 4 . At a high-level, instead of pursuing precisely separated audio sound sources, background-oriented and foreground-oriented audio features are generated. The temporal correlations of these features with their visual counterparts will be evaluated in later steps to generate useful audio-visual patterns for concept classification. 
     Various descriptors have been developed to represent audio signals in both temporal and spectral domains. Among these features, the well-known Mel-Frequency Cepstral Coefficients (MFCCs) feature is one of the most popular choices for many different audio recognition systems. MFCCs features represent the shape of the overall spectrum with a few coefficients, and have been shown to work well for both structured sounds (e.g., speech) and unstructured environmental sounds. In soundtracks of generic videos, the foreground sound events (e.g., an occasional dog barking or hand clapping) are distributed unevenly and sparsely. In such a case, the MFCCs feature extracted from uniformly spaced audio windows capture the overall characteristics of the background environmental sound, since the statistical impact of the sparse foreground sound events is quite small. Therefore, the MFCCs feature is used as the background audio feature. 
     For each given video V j , an MFCC feature extraction step  300  is used to extract MFCCs feature  310  from the corresponding audio soundtrack  125  using uniformly spaced short windows (e.g., 25 ms windows with a hop size of 10 ms). The MFCCs feature  310  is represented as a  13 -dim feature vector. Next, the MFCCs features  310  from all training videos are put together and a clustering step  320  is used to construct a D-word audio background vocabulary  330  (V b−a ). The clustering step  320  can use any clustering method known in the art. In a preferred embodiment, the clustering step  320  uses the well-known hierarchical K-means clustering method. Similar to visual-based processing, two different histogram-like features are computed based on V b−a . First, audio background BoW features  340  (F j   b−a ) are generated for each video V j  by matching the MFCCs features  310  in the video to codewords in the vocabulary and conducting soft weighting. The audio background BoW features  340  can be used directly for classifying concepts. Second, to study the audio-visual correlation, an audio background temporal feature  165  {H j1   b−a , H j2   b−a , . . . } is generated for each video V j  as follows. Each MFCCs feature  310  is labeled with the codeword in the audio background vocabulary V b−a  that is closest to the MFCCs feature  310 . Next, for each sampled image frame I ji  in the video, a 200 ms window centered on this frame is taken. The frequency of the codewords labeled to the MFCCs are then collected and used to generate a D-dim histogram H ji   b−a . This H ji   b−a  can be considered as temporally synchronized with the visual-based histograms H ji   f−v  and H ji   b−v . 
     A transient event feature extraction step  350  is used to extract a transient feature  360  from the audio soundtrack  125 . As mentioned above, the audio soundtrack  125  of a generic video usually has unevenly and sparsely distributed foreground sound events. To capture such foreground information, local representations that focus on short-term local sound events should be used. The transient event feature extraction step  350  can use any transient event feature extraction method known in the art. In a preferred embodiment, the transient event feature extraction step  350  uses the method described by Cotton et al., in the article “Soundtrack classification by transient events” (IEEE Int. Conference on Acoustics, Speech and Signal Processing, pp. 473-476, 2011), which is incorporated herein by reference. According to this method, a local event-based representation is determined by locating a set of salient points in the soundtrack based on time-frequency energy analysis and multi-scale spectrum analysis. These salient points contain distinct event onsets (i.e., transient events). By modeling the local temporal structure around each transient event, an audio feature reflecting the foreground of the audio soundtrack  125  can be computed for use as the transient feature  360 . 
     More specifically, an Automatic Gain Control (AGC) is first applied to equalize the audio energy in both time and frequency domains. Next, the spectrogram of the AGC-equalized signal is taken for a number of different time-frequency tradeoffs, corresponding to window length between 2 to 80 ms. Multiple scales enable the localization of events of different durations. High-magnitude bins in any spectrogram indicate a candidate transient event at the corresponding time. A limit is empirically set on the minimum distance between successive events to produce 4 events per second on average. A 250 ms window of the audio signal is extracted centered on each transient event time, which captures the temporal structure of the transient event. Within each 250 ms window, a 40-dim spectrogram-based feature is computed for short-term signals (e.g., over 25 ms windows with 10 ms hops). These features from multiple short-term signals are concatenated together to form a spectrogram-based representation for each transient event. After that, PCA is performed over all transient events from all training videos, and the top 20 bases are used to project the original spectrogram-based event representation to 20 dimensions. 
     All of the transient features  360  from all training videos are collected and a clustering step  365  is used to construct a D-word audio foreground vocabulary  370  (V f−a ). Again, two different histogram-like features are computed based on V f−a . First, audio foreground BoW features  380  (F j   f−a ) are generated for each video V j  by matching the transient features in the video to codewords in the vocabulary and conducting soft weighting. Second, an audio foreground temporal feature  160  {H j1   f−a , H j2   f−a , . . . } is generated for each video V j  as follows. Each transient event is labeled with the codeword in the audio foreground vocabulary V f−a  that is closest to the transient event feature. Next, for each sampled image frame I ji  in the video, a 200 ms window centered on this frame is taken. The frequency of the codewords labeled to the transient events whose centers fall into this window are collected and used to generate a D-dim histogram H ji   f−a . Similar to H ji   b−a , H ji   f−a  can be considered as synchronized with H ji   f−v  or H ji   b−v . 
     Additional details for a preferred embodiment of the audio-visual temporal causal analysis module  170  will now be discussed with reference to  FIG. 5 . The goal is to capture the co-occurrences of audio and visual patterns over time. The rough foreground/background separation of both temporal SIFT point tracks and audio sounds enables a meaningful study of such temporal relations. For the purpose of classifying general concepts in generic videos, all of the following factors have their contributions: foreground visual objects, foreground audio transient events, background visual scenes, and background environmental sounds. Therefore, their mixed-and-matched temporal relations are explored to find salient AVGs that can assist the final classification. 
     From the previous sections, for each video V j , there are four features: visual foreground temporal feature  140  {H j1   f−v , H j2   f−v , . . . }, audio foreground temporal feature  160  {H j1   f−a , H j2   f−a , . . . }, visual background temporal feature  145  {H j1   b−v , H j2   b−v , . . . }, and audio background temporal feature  165  {H j1   b−a , H j2   b−a , . . . }, with four corresponding vocabularies: visual foreground vocabulary  260  (V f−v ), audio foreground vocabulary  370  (V f−a ), visual background vocabulary  265  (V b−v ), and audio background vocabulary  330  (V b−a ). 
     For each vocabulary, (e.g., the visual foreground vocabulary  260 ) each codeword w k  in the vocabulary can be treated as a point process, N k   f−v (t), which counts the number of occurrences of w k  in the interval (0, t]. The number of occurrences of w k  in a small interval dt is dN k   f−v (t)=N k   f−v (t+dt)−N k   f−v (t), and E{N k   f−v (t)}=λ k   f−v  is the mean intensity. For theoretical and practical convenience, the zero-mean process is considered, and N k   f−v (t) is assumed as wide-sense stationary, mixing, and orderly. Point processes generated by all D codewords of vocabulary V f−v  form a D-dim multivariate point process N f−v (t)=(N 1   f−v (t), . . . , N D   f−v (t)) T . Each video V j  gives one realization (trial) of N f−v (t) with counting vector (h j1   f−v (t), h j2   f−v (t), . . . , h jD   f−v (t)) T , where h jk   f−v (t) is the value over the k-th bin of the histogram H jk   f−v . Similarly, D-dim multivariate point processes N f−a (t), N b−v (t), and N b−a (t) can be generated for vocabularies V f−a , V b−v , and V b−a , respectively. 
     Granger causality is a type of statistical temporal causality and is a statistical measure based on the concept of time series forecasting, where a time series Y 1  is considered to causally influence a time series Y 2  if predictions of future values of Y 2  based on the joint history of Y 1  and Y 2  are more accurate than predictions based on Y 2  alone (see: Granger “Investigating causal relations by econometric models and cross-spectral methods,” Econometrica, Vol. 37, pp. 424-438, 1969). The estimation of Granger causality usually relies on autoregressive models, and for continuous-valued data such model fitting is straightforward. In an article entitled “Analyzing multiple spike trains with nonparametric granger causality” (Journal of Computational Neuroscience, Vol. 27, pp. 55-64, 2009), which is incorporated herein by reference, Nedungadi et al. have described a nonparametric method that has been developed which bypasses the autoregressive model fitting to estimate Granger causality for point processes. The theoretical basis lies in the spectral representation of point processes, the factorization of spectral matrices, and the formulation of Granger causality in the spectral domain. In a preferred embodiment, a compute nonparametric Granger causality step  400  is used to compute causal matrices  410  representing the temporal causality between audio and visual codewords using the aforementioned method of Nedungadi et al. The following details this process. For simplicity, those indexes f−v, b−v, f−a, and b−a, are temporarily omitted, without loss of generality, since Granger causality can be computed for any two codewords from any vocabularies. 
     The pair-wise statistical relation between two point processes N k (t) and N 1 (t) can be captured by the cross-covariance density function R k1 (u) at lag u: 
     
       
         
           
             
               
                 
                   
                     
                       R 
                       kl 
                     
                      
                     
                       ( 
                       u 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         E 
                          
                         
                           { 
                           
                             
                               
                                 dN 
                                 k 
                               
                                
                               
                                 ( 
                                 
                                   t 
                                   + 
                                   u 
                                 
                                 ) 
                               
                             
                              
                             
                               
                                 dN 
                                 l 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                           } 
                         
                       
                       dudt 
                     
                     - 
                     
                       
                         I 
                          
                         
                           [ 
                           
                             
                               
                                 N 
                                 k 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                             = 
                             
                               
                                 N 
                                 l 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                           ] 
                         
                       
                        
                       
                         λ 
                         k 
                       
                        
                       
                         δ 
                          
                         
                           ( 
                           u 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     where δ(u) is the classical Kronecker delta function, and I[·] is the indicator function. By taking the Fourier transform of R k1 (u), we obtain the cross-spectrum S k1 (f). Specifically, the multitaper method can be used to compute the spectrum, where M data tapers {q m } M   m=1  are applied successively to point process N k (t) (with length T): 
     
       
         
           
             
               
                 
                   
                     
                       
                         S 
                         kl 
                       
                        
                       
                         ( 
                         f 
                         ) 
                       
                     
                     = 
                     
                       
                         1 
                         
                           2 
                            
                           π 
                            
                           
                               
                           
                            
                           MT 
                         
                       
                        
                       
                         
                           ∑ 
                           
                             m 
                             = 
                             1 
                           
                           M 
                         
                          
                         
                             
                         
                          
                         
                           
                             
                               
                                 N 
                                 ~ 
                               
                               k 
                             
                              
                             
                               ( 
                               
                                 f 
                                 , 
                                 m 
                               
                               ) 
                             
                           
                            
                           
                             
                               
                                 
                                   N 
                                   ~ 
                                 
                                 l 
                               
                                
                               
                                 ( 
                                 
                                   f 
                                   , 
                                   m 
                                 
                                 ) 
                               
                             
                             * 
                           
                         
                       
                     
                   
                   , 
                   
                     . 
                     
                       
 
                     
                      
                     where 
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         N 
                         ~ 
                       
                       k 
                     
                      
                     
                       ( 
                       
                         f 
                         , 
                         m 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         
                           t 
                           p 
                         
                         = 
                         1 
                       
                       T 
                     
                      
                     
                         
                     
                      
                     
                       
                         
                           q 
                           m 
                         
                          
                         
                           ( 
                           
                             t 
                             p 
                           
                           ) 
                         
                       
                        
                       
                         
                           N 
                           k 
                         
                          
                         
                           ( 
                           
                             t 
                             p 
                           
                           ) 
                         
                       
                        
                       
                         
                           exp 
                            
                           
                             ( 
                             
                               
                                 - 
                                 2 
                               
                                
                               π 
                                
                               
                                   
                               
                                
                               
                                 ift 
                                 p 
                               
                             
                             ) 
                           
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     The superscript “*” is the complex conjugate transpose. Eq. (2) gives an estimation of the cross-spectrum using one realization, and such estimations of multiple realizations are averaged to give the final estimation of the cross-spectrum. 
     For multivariate continuous-valued time series Y 1  and Y 2  with joint autoregressive representations: 
     
       
         
           
             
               
                 
                   
                     
                       Y 
                       1 
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             p 
                             = 
                             1 
                           
                           ∞ 
                         
                          
                         
                             
                         
                          
                         
                           
                             a 
                             p 
                           
                            
                           
                             
                               Y 
                               1 
                             
                              
                             
                               ( 
                               
                                 t 
                                 - 
                                 p 
                               
                               ) 
                             
                           
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             p 
                             = 
                             1 
                           
                           ∞ 
                         
                          
                         
                             
                         
                          
                         
                           
                             b 
                             p 
                           
                            
                           
                             
                               Y 
                               2 
                             
                              
                             
                               ( 
                               
                                 t 
                                 - 
                                 p 
                               
                               ) 
                             
                           
                         
                       
                       + 
                       
                         ɛ 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   
                     4 
                      
                     
                         
                     
                      
                     a 
                   
                   ) 
                 
               
             
             
               
                 
                   
                     
                       Y 
                       2 
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             p 
                             = 
                             1 
                           
                           ∞ 
                         
                          
                         
                             
                         
                          
                         
                           
                             c 
                             p 
                           
                            
                           
                             
                               Y 
                               2 
                             
                              
                             
                               ( 
                               
                                 t 
                                 - 
                                 p 
                               
                               ) 
                             
                           
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             p 
                             = 
                             1 
                           
                           ∞ 
                         
                          
                         
                             
                         
                          
                         
                           
                             d 
                             p 
                           
                            
                           
                             
                               Y 
                               1 
                             
                              
                             
                               ( 
                               
                                 t 
                                 - 
                                 p 
                               
                               ) 
                             
                           
                         
                       
                       + 
                       
                         η 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   
                     4 
                      
                     
                         
                     
                      
                     b 
                   
                   ) 
                 
               
             
           
         
       
     
     their noise terms are uncorrelated over time and their contemporaneous covariance matrix is: 
     
       
         
           
             
               
                 
                   
                     Σ 
                     = 
                     
                       [ 
                       
                         
                           
                             
                               Σ 
                               2 
                             
                           
                           
                             
                               Ψ 
                               2 
                             
                           
                         
                         
                           
                             
                               Ψ 
                               2 
                             
                           
                           
                             
                               Γ 
                               2 
                             
                           
                         
                       
                       ] 
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     where Σ 2 =var(ε(t)), Γ 2 =var(η(t)) and ψ 2 =cov(ε(t),η(t)). The spectral matrix can be computed as: 
     
       
         
           
             
               
                 
                   
                     
                       S 
                        
                       
                         ( 
                         f 
                         ) 
                       
                     
                     = 
                     
                       
                         [ 
                         
                           
                             
                               
                                 
                                   S 
                                   11 
                                 
                                  
                                 
                                   ( 
                                   f 
                                   ) 
                                 
                               
                             
                             
                               
                                 
                                   S 
                                   12 
                                 
                                  
                                 
                                   ( 
                                   f 
                                   ) 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   S 
                                   21 
                                 
                                  
                                 
                                   ( 
                                   f 
                                   ) 
                                 
                               
                             
                             
                               
                                 
                                   S 
                                   22 
                                 
                                  
                                 
                                   ( 
                                   f 
                                   ) 
                                 
                               
                             
                           
                         
                         ] 
                       
                       = 
                       
                         
                           H 
                            
                           
                             ( 
                             f 
                             ) 
                           
                         
                          
                         
                           ∑ 
                           
                             
                               H 
                                
                               
                                 ( 
                                 f 
                                 ) 
                               
                             
                             * 
                           
                         
                       
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                     where 
                      
                     
                       : 
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
             
               
                 
                   
                     H 
                      
                     
                       ( 
                       f 
                       ) 
                     
                   
                   = 
                   
                     [ 
                     
                       
                         
                           
                             
                               H 
                               11 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                         
                         
                           
                             
                               H 
                               12 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                         
                       
                       
                         
                           
                             
                               H 
                               21 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                         
                         
                           
                             
                               H 
                               22 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                         
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     is the transfer function depending on coefficients of the autoregressive model. The spectral matrix S(f) of two point processes N k (t) and N 1 (t) can be estimated using Eq. (2). By spectral matrix factorization we can decompose S(f) into a unique corresponding transfer function H′(f) and noise processes Σ′ 2  and Γ′ 2 . 
     Next, the Granger causality at frequency f can be estimated as: 
     
       
         
           
             
               
                 
                   
                     
                       
                         G 
                         
                           
                             N 
                             1 
                           
                           → 
                           
                             N 
                             k 
                           
                         
                       
                        
                       
                         ( 
                         f 
                         ) 
                       
                     
                     = 
                     
                       ln 
                        
                       
                         
                           
                             S 
                             kk 
                           
                            
                           
                             ( 
                             f 
                             ) 
                           
                         
                         
                           
                             
                               H 
                               kk 
                               ′ 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                            
                           
                             Σ 
                             2 
                             ′ 
                           
                            
                           
                             
                               
                                 H 
                                 kk 
                                 ′ 
                               
                                
                               
                                 ( 
                                 f 
                                 ) 
                               
                             
                             * 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       G 
                       
                         
                           N 
                           k 
                         
                         → 
                         
                           N 
                           1 
                         
                       
                     
                      
                     
                       ( 
                       f 
                       ) 
                     
                   
                   = 
                   
                     ln 
                      
                     
                       
                         
                           
                             S 
                             ll 
                           
                            
                           
                             ( 
                             f 
                             ) 
                           
                         
                         
                           
                             
                               H 
                               ll 
                               ′ 
                             
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                            
                           
                             Γ 
                             2 
                             ′ 
                           
                            
                           
                             
                               
                                 H 
                                 ll 
                                 ′ 
                               
                                
                               
                                 ( 
                                 f 
                                 ) 
                               
                             
                             * 
                           
                         
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     The Granger causality scores over all frequencies are then summed together to obtain a single time-domain causal influence, i.e., C N     1     →N     k   =Σ f G N     1     →N     k    and C N     k     →N     1   =Σ f G N     k     →N     1   . In general, C N     1     →N     k   ≠C N     k     →N     1   , due to the directionality of the causal relations. 
     The target of studying temporal causality between audio and visual codewords in this invention is to identify strongly correlated AVGs, where the direction of the relations is usually not important. For example, a dog can start barking at any time during the video, and one would like to find the AVG that contains correlated codewords describing the foreground dog barking sound and the visual dog point tracks. The direction of whether the barking sound is captured before or after the visual tracks is irrelevant. Therefore, for a pair of codewords represented by point processes N k   S     k   (t) and N 1   S     1    (t) (where s k  or s 1  is one of the following f−v, f−a, b−v, and b−a, indicating the vocabularies the codeword comes from), the nonparametric Granger causality scores from both directions C N     1     →N     k    and C N     k     →N     k    are summed together to generate the final similarity between these two codewords: 
         C ( N   k   S     k     ,N   1   S     k   )− C   N     k     →N     1     +C   N     1     →N     k   .  (10)
 
     Then, for a pair of audio and visual vocabularies, e.g., V f−v  and V f−a , we have a 2D×2D symmetric causal matrix  410 : 
     
       
         
           
             
               
                 
                   
                     [ 
                     
                       
                         
                           
                             C 
                             
                               
                                 f 
                                 - 
                                 v 
                               
                               , 
                               
                                 f 
                                 - 
                                 v 
                               
                             
                           
                         
                         
                           
                             C 
                             
                               
                                 f 
                                 - 
                                 v 
                               
                               , 
                               
                                 f 
                                 - 
                                 a 
                               
                             
                           
                         
                       
                       
                         
                           
                             C 
                             
                               
                                 f 
                                 - 
                                 a 
                               
                               , 
                               
                                 f 
                                 - 
                                 v 
                               
                             
                           
                         
                         
                           
                             C 
                             
                               
                                 f 
                                 - 
                                 a 
                               
                               , 
                               
                                 f 
                                 - 
                                 v 
                               
                             
                           
                         
                       
                     
                     ] 
                   
                   , 
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     where C f−v,f−v , C f−a,f−a , and C f−v,f−a  are D×D matrices with entries C(N k   f−v , N 1   f−v ), C(N k   f−a , N 1   f−a ), and C(N k   f−v , N 1   f−a ), respectively. 
     A spectral clustering step  420  can be applied directly based on the causal matrices  410  to identify clusters of codewords that have high correlations. Each resulting cluster is called an AVG, and codewords in an AVG can come from both audio and visual vocabularies. The AVGs capture temporally correlated audio and visual codewords that statistically interact over time. Each AVG can be treated as an audio-visual pattern, and all AVGs form an audio-visual dictionary. For a given video V j , the original SIFT point tracks and audio features can be mapped to AVGs and generate an audio-visual dictionary-based feature. The simplest way to compute this dictionary-based feature is to aggregate the original BoW features over individual visual and audio vocabularies. For instance, for an AVG containing codewords w 1   f−a , . . . , w n   f−a  from V f−a , and w 1   f−v , . . . , w m   f−v  from V f−v , the value over the corresponding bin in the dictionary-based feature of video V j  is: Σ p=1   n F jp   f−a +Σ p=1   m F jp   f−v . F jp   f−a  (or F jp   f−v ) is the value over the p-th bin in the audio foreground BoW features  380  (F j   f−a ) (or the visual foreground BoW features  270  (F j   f−v ) over the audio foreground vocabulary  370  (V f−a ) (or the visual foreground vocabulary  260  (V f−v ). Classifiers such as the well-known Support Vector Machine (SVM) classifiers can be trained using this feature for concept classification. 
     In a preferred embodiment, a total of four audio-visual dictionaries  180  are generated by analyzing the temporal causal relations between different types of audio and visual codewords. They are: a visual foreground audio foreground dictionary  430  (D f−v,f−a ) formed by correlating the visual foreground vocabulary  260  (V f−v ) and the audio foreground vocabulary  370  (V f−a ), a visual background audio background dictionary  460  (D b−v,b−a ) formed by correlating the visual background vocabulary  265  (V b−v ) and the audio background vocabulary  330  (V b−a ), a visual foreground audio background dictionary  440  (D f−v,b−a ) formed by correlating the visual foreground vocabulary  260  (V f−v ) and the audio background vocabulary  330  (V b−a ), and a visual background audio foreground dictionary  450  (D b−v,f−a ) formed by correlating the visual background vocabulary  265  (V b−v ) and the audio foreground vocabulary  370  (V f−a ). All of these correlations reveal useful audio-visual patterns for classifying semantic concepts. 
     The individual audio-visual dictionaries  180  can be directly used to train classifiers for detecting semantic concepts. Also, these audio-visual dictionaries  180  can be combined (i.e., the dictionary-based features can be concatenated into a long vector) to train the semantic concept classifiers  190 . 
     A preferred embodiment of a method for determining a semantic concept classification for an input video  500  including a temporal sequence of image frames  505  and a corresponding audio soundtrack  510  will now be discussed with reference to  FIG. 6 . SIFT features  520  are extracted from a set of sampled image frames selected from the image frames  505  (e.g., uniformly sampled image frames) using the earlier described SIFT feature extraction step  200 . The sampling rate is determined by considering both the computation cost and the representativeness of images to cover the video content. In addition to the SIFT features  520 , motion features  525  represented by an HOM motion feature vector is computed over each extracted SIFT interest point using a motion feature extraction step  515 . The SIFT features  520  and the corresponding motion features  525  are concatenated to provide visual features  540  that describes each local SIFT interest point. 
     Relative to the audio soundtrack  510 , MFCCs features  530  represented by a 13-dim MFCCs feature vector is extracted from the audio soundtrack  510  using the earlier described MFCC feature extraction step  300  using uniformly spaced short windows (e.g., 25 ms windows with a hop size of 10 ms). Also, transient features  535  are determined using the earlier described transient event feature extraction step  350 . 
     D-dim visual foreground BoW features  545  are generated by matching the visual features  540  to codewords in the visual foreground vocabulary  260 . In a preferred embodiment, the visual foreground BoW features  545  are generated in an analogous manner to the method used to compute the visual foreground BoW features  270  described earlier, where a soft weighting scheme is also used to alleviate the quantization effects. Likewise, visual background BoW features  550  are generated by matching the visual features  540  to codewords in the visual background vocabulary  265 ; audio foreground BoW features  555  are generated by matching the audio transient features  535  to codewords in the audio foreground vocabulary  370 ; and audio background BoW features  560  are generated by matching the MFCCs features  530  to codewords in the audio background vocabulary  330 . 
     By using the visual foreground BoW features  545 , the visual background BoW features  550 , the audio foreground BoW features  555 , and the audio background BoW features  560 , dictionary-based similarity scores  565  can be computed based on the audio-visual dictionaries  180  (i.e., similarity scores based on the visual foreground audio foreground dictionary  430  (D f−v,f−a ), similarity scores based on the visual background audio background dictionary  460  (D b−v,b−a ), similarity scores based on the visual foreground audio background dictionary  440  (D f−v,b−a ), and similarity scores based on the visual background audio foreground dictionary  450  (D b−v,f−a )). Each dictionary-based similarity score  565  indicates the similarity between the input video  500  and the AVGs in the various audio-visual dictionaries. Then the dictionary-based similarity scores  565  can be fed into a concept classification step  570  which applies the semantic concept classifiers  190  to determine a semantic concept  580  for the input video  500 . 
     The dictionary-based similarity scores  565  can be determined in a variety of ways in accordance with the present invention. One such method is illustrated in  FIG. 7A , which shows a set of AVGs  650  in one of the audio-visual dictionaries  180  (FIG.  5 )—specifically, AVGs  650  from the visual foreground audio foreground dictionary  430  that were constructed by correlating the visual foreground vocabulary  260  ( FIG. 3 ) and the audio foreground vocabulary  370  ( FIG. 4 ). Each AVG  650  contains a set of visual foreground and audio foreground codewords. The structures of other types of AVGs (i.e., the AVGs from the visual foreground audio background dictionary  440 , the AVGs from the visual background audio foreground dictionary  450 , and the AVGs from the visual background audio background dictionary  460 ) have forms similar to the example AVGs  650  illustrated for the visual foreground audio foreground dictionary  430 , where the visual and audio codewords are either foreground or background codewords as appropriate. 
     In the illustrated example, an input wedding ceremony video clip  600  includes a time sequence of image frames  605  and a corresponding audio soundtrack  610 . The image frames  605  include various visual foreground  615  elements (e.g., the bride and groom) and visual background  620  elements (e.g., curtains and windows). Similarly, the audio soundtrack  610  includes audio foreground  625  elements (e.g., speech and cheering), as well as audio background  630  elements. 
     A set of codeword similarity scores  640  are determined corresponding to the visual and audio features extracted from the video clip  600 . As was discussed relative to  FIG. 6 , the image frames  605  are analyzed to determine visual foreground BoW features  545  ( FIG. 6 ) corresponding to visual foreground codewords in the visual foreground vocabulary  260  ( FIG. 6 ). Similarity scores are determined for each of the visual foreground codewords providing an indication of the similarity between the visual foreground  615  feature in the image frames  605  and the visual foreground codewords in the visual foreground vocabulary  260  (e.g., visual foreground codeword # 1  is assigned a similarity score of “6” in the example shown in  FIG. 7A ). Each codeword similarity score  640  for a visual foreground  615  feature gives the value of the corresponding bin in the visual foreground BoW feature  545  ( FIG. 5 ) For example, the bin corresponding to visual foreground codeword # 1  in the visual foreground BoW feature  545  has value “6” in the illustrated example. 
     Similarly, the audio soundtrack  610  analyzed to determine audio foreground BoW features  555  ( FIG. 6 ) corresponding to audio foreground codewords in the audio foreground vocabulary  370  ( FIG. 6 ). Similarity scores are determined for each of the audio foreground codewords providing an indication of similarity between the audio foreground  625  feature in the audio soundtrack  610  and the audio foreground codewords in the audio foreground vocabulary  370  (e.g., audio foreground codeword # 1  is assigned a similarity score of “7” in the example shown in  FIG. 7A ). Each audio foreground codeword similarity score gives the value of the corresponding bin in the audio foreground BoW feature  555  (e.g., the bin corresponding to audio foreground codeword # 1  in the audio foreground BoW feature  555  has value “7” in the illustrated example. 
     The codeword similarity scores  640  are then analyzed relative to the AVGs  650  from the visual foreground audio foreground dictionary  430  to determine a set of dictionary-based similarity scores  565  representing the similarity between the video clip  600  and the AVGs  650 . The dictionary-based similarity scores  565  can be computed from the codeword similarity scores  640  using any method known in the art. In one embodiment, the dictionary-based similarity score  565  for a particular AVG  650  is determined by computing average of the codeword similarity scores for the codewords that are included in the AVG  650 . For example, the dictionary-based similarity score for AVG # 1  can be determined by averaging the codeword similarity scores for visual foreground codeword # 1 , visual foreground codeword # 3 , audio foreground codeword # 2 , audio foreground codeword # 4 , and audio foreground codeword # 5 . In this case, AVG # 1  is rounded to a similarity score of “3.” 
     A similar process can be used to determine the dictionary-based similarity scores  565  for the other audio-visual dictionaries  180  ( FIG. 6 ). Then the resulting set of dictionary-based similarity scores  565  can be fed into the concept classification step  570  ( FIG. 6 ) to determine the semantic concept  580  for the input video clip  600  (e.g., the dictionary-based similarity scores  565  can be used to generate kernel matrices for SVM classification). 
     Returning to a discussion of  FIG. 6 , in an alternate embodiment the dictionary based similarity scores  565  are also responsive to reference video codeword similarity scores  590  for a set of reference video clips. In this case the set of reference videos are analyzed to determine corresponding visual foreground BoW features  545 , visual background BoW features  550 , audio foreground BoW features  555  and audio background BoW features  560 . These features are then compared to the codewords in the audio-visual dictionaries  180  to determine the reference video codeword similarity scores  590 . In some embodiments, the reference video codeword similarity scores  590  can be determined in a manner analogous to the method used to determine the codeword similarity scores  640  ( FIG. 7A ). 
     In some implementations, the set of reference video clips include some or all of the video clips in the set of training video clips that were analyzed during the formation of the audio-visual dictionaries  180 . In some implementations, the set of reference video clips may also include video clips that were not included in the set of training video clips. It is generally desirable that the set of reference video clips include video clips that span the range of different semantic concepts  580  that the semantic concept classifiers  190  are trained to recognize. 
     According to this alternate embodiment, the dictionary-based similarity score  565  are reference video similarity scores that are determined relative to each reference video clip representing the similarity between the video  500  and the reference video clip responsive to the audio-visual grouplets, the codeword similarity scores and the reference video codeword similarity scores. The semantic concept classifiers  190  are trained to determine the semantic concept  580  responsive to the reference video similarity scores. 
     There are a variety of ways that the dictionary-based similarity score  565  for each reference video clip can be determined in accordance to this alternate embodiment. One such method is illustrated in  FIG. 7B . As in the method of  FIG. 7A , an input video  500  is analyzed to determine visual features and audio features, which are in turn used to determine a set of codeword similarity scores  640  for each audio-visual codeword in the audio-visual dictionaries  180  ( FIG. 6 ). Likewise, each reference video  700  is analyzed to determine visual features and audio features, which are in turn used to determine a set of reference video codeword similarity scores  710 . 
     The codeword similarity scores  640  and the reference video codeword similarity scores  710  are then compared relative to the AVGs in the audio-visual dictionaries  180  ( FIG. 6 ) to determine a set of AVG similarity scores  720 . Each AVG similarity score  720  provides an indication of the difference between the codeword similarity scores  640  and the reference video codeword similarity scores  710  for the bins corresponding to the codewords that are included in a particular AVG. For example, AVG # 1  shown in  FIG. 7A  includes visual foreground codeword # 1 , visual foreground codeword # 3 , audio foreground codeword # 2 , audio foreground codeword # 4  and audio foreground codeword # 5 . These five AVG # 1  codewords  715  are indicated with dashed outlines in  FIG. 7B . The corresponding codeword similarity scores  640  and the reference video codeword similarity scores  710  for the AVG # 1  codewords are then be compared to determine an AVG # 1  similarity score  722 . This process is repeated for each of the AVGs to determine a set of AVG similarity scores  720  (AVG # 1  similarity score  722  through AVG #n similarity score  724 ) that together provide a comparison of the input video  500  to the reference video  700 . 
     The AVG similarity scores  720  can be determined using any method known in the art. In one embodiment, the AVG similarity scores  720  are determined by computing a Euclidean distance between the codeword similarity scores for the codewords that are in a particular AVG. For example, the AVG# 1  similarity score (S 1 ) would be given by: 
         s   1 =√{square root over ((6−8) 2 +(1−3) 2 +(3−2) 2 +(1−4) 2 +(2−3) 2 )}{square root over ((6−8) 2 +(1−3) 2 +(3−2) 2 +(1−4) 2 +(2−3) 2 )}{square root over ((6−8) 2 +(1−3) 2 +(3−2) 2 +(1−4) 2 +(2−3) 2 )}{square root over ((6−8) 2 +(1−3) 2 +(3−2) 2 +(1−4) 2 +(2−3) 2 )}{square root over ((6−8) 2 +(1−3) 2 +(3−2) 2 +(1−4) 2 +(2−3) 2 )}=4.3.  (12)
 
     In this example, a low AVG similarity score value provides an indication that the video  500  and the reference video  700  are similar relative to the features associated with the particular AVG. 
     In other embodiments, the AVG similarity scores  720  can be determined using other methods for characterizing differences between sets of numbers. For example, AVG similarity scores  720  can be determined by computing an average of the codeword similarity score differences, or by computing a sum of the codeword similarity score differences. 
     This set of AVG similarity scores  720  determined with respect to the particular reference video  700  can then be aggregated to determine the dictionary-based similarity score  565  for the reference video  700  (i.e., a “reference video similarity score”). The reference video similarity score provides a measure of the similarity between the input video  500  and the reference video  700 . 
     The AVG similarity scores  720  can be combined in any way known in the art to determine the reference video similarity score. For example, the AVG similarity scores  720  can be combined using a simple average. In a preferred embodiment, the AVG similarity scores  720  are combined using a weighted average. The weights associated with each AVG can be determined as part of a training process in order to provide reference video similarity scores that correlate with known similarity values for a set of training videos. 
     The process shown in  FIG. 7B  is repeated for each reference video  700  in a predefined set of reference videos to determine a set of dictionary-based similarity scores  565  (i.e., reference video similarity scores). The determined dictionary-based similarity scores  565  can then be fed into the semantic concept classifiers  190  ( FIG. 6 ) to determine one or more semantic concepts  580  ( FIG. 6 ) for the video  500 . The similarity of the video  500  to reference videos  700  having known semantic concepts can provide a strong indication of appropriate semantic concepts  580  that should be assigned to the video  500 . For example, if the reference video similarity scores indicate that the video  500  has a high degree of similarity to reference videos  700  that were classified as “music performance” then the semantic concept classifiers  190  would tend to classify the video  500  using this semantic concept classification. 
     The current invention is different from other works such as U.S. Patent Application Publication 2011/0081082 to Jiang et al., entitled “Video concept classification using audio-visual atoms,” in the following aspects. The present invention extracts an Audio-Visual Grouplet (AVG) representation by analyzing the temporal correlation between audio and visual codewords. While the method of U.S. Patent Application Publication 2011/0081082 learns audio-visual atoms by concatenating static audio and visual features. The current invention learns statistical temporal correlation between audio and visual codewords over a set of training videos. While the method of U.S. Patent Application Publication 2011/0081082 pursues coarse-level audio-visual synchronization within an individual video. In addition, the present invention conducts foreground/background separation in both visual and audio channels while the method of U.S. Patent Application Publication 2011/0081082 does not distinguish foreground and background information. 
     The preferred embodiment of the current invention was evaluated over the large-scale CCV set described in the aforementioned article by Jiang, et al. entitled “Consumer video understanding: A benchmark database and an evaluation of human and machine performance,” which contains 9317 consumer videos collected from YouTube. These videos were captured by ordinary users under unrestricted challenging conditions, without post-editing. The original audio soundtracks are preserved, in contrast to other commonly used datasets that include news or movie videos. This enables studying legitimate audio-visual correlations. Each video in the large-scale CCV set was manually labeled to 20 semantic concepts by using Amazon Mechanical Turk. More details about the data set and category definitions can be found in the aforementioned article by Jiang et al. The experiment used similar settings to those used by Jiang et al. That is, the same training (4659 videos) and test (4658 videos) sets were used, and the same one-versus-all χ 2  kernel SVM classifier was used for the semantic concept classifiers  190 . The performance was measured by Average Precision (AP, the area under un-interpolated PR curve) and Mean AP (MAP, averaged AP across concepts). 
     To demonstrate the effectiveness of the proposed method, the preferred embodiment of the invention is compared with the state-of-the-art BoW representations using individual foreground and background audio and visual vocabularies (V f−v , V b−v , V f−a  and V b−a ), as well as their early-fusion combinations. The AP results are shown in  FIG. 8 . In accordance with the preferred embodiment, the four types of audio-visual dictionary-based features (D f−v,f−a , D b−v,f−a , D f−v,b−a  and D b−v,b−a ) are concatenated together to train semantic concept classifiers  190 , which is the “All A-V Dictionaries” approach shown in  FIG. 8 . 
     From  FIG. 8 , for individual vocabularies, the visual foreground by), vocabulary (V f−v ) performs better than the visual background vocabulary (V b−v ), in general, while the audio background vocabulary (V b−a ) performs better than the audio foreground vocabulary (V f−a ). Such results are reasonable, because of the importance of the visual foreground in classifying objects and activities, as well as the effectiveness of audio background environmental sounds in classifying general concepts as shown by previous literatures. Compared with the visual foreground, visual background wins for the “wedding ceremony” and “non-music performance” semantic concepts; this is because of the importance of the background settings for these concepts (e.g., the arch, flower boutique, and seated crowd for “wedding ceremony,” and the stadium or stage setting for “non-music performance”). In the audio aspect, audio foreground outperforms audio background over three concepts: “dog,” “birthday,” and “music performance”; this is because of the usefulness of capturing consistent foreground sounds for these semantic concepts. By combining the individual features via early fusion (“SIFT+MFCCs+Trans”), compared with individual features, all concepts show improvements in the AP scores, and the MAP is improved by over 33% on a relative basis. 
     Through exploiting the temporal audio-visual correlation, the audio-visual dictionaries generally outperform the corresponding individual audio or visual vocabularies. For example, the MAP of the visual foreground audio foreground dictionary (D f−v,f−a ) outperforms those of visual foreground vocabulary (V f−v ) and the audio foreground vocabulary (V f−a ) by more than 10%, the MAP of the visual background audio background dictionary (D b−v,b−a ) outperforms those of the visual background vocabulary (V b−v ) and the audio background vocabulary (V b−a ) by more than 20%. By combining all types of dictionaries together, all concepts get consistent, significant AP improvements compared with individual audio or visual vocabularies, and the overall MAP is improved by more than 50%. In addition, compared with direct multi-modal fusion without temporal audio-visual correlation (i.e., “SIFT+MFCCs+Trans”), the “All A-V Dictionaries” approach developed in accordance with the present invention has consistent AP gains over all concepts, with a 12% MAP gain overall. For 12 concepts (e.g., “baseball,” and “ice skating”) the APs are improved by more than 10%. The results demonstrate the effectiveness of the method of the present invention relative to prior art methods for determining semantic concepts for videos. 
       FIG. 9  is a high-level flow diagram illustrating a preferred embodiment of the training process for determining a grouplet dictionary including a plurality of temporally-correlated grouplets  965 , a distance metric  975 , and semantic concept classifiers  990 , in accordance with the present invention. 
     Given an input digital record  900 , including an image sequence  905  comprising a temporal sequence of video frames, a corresponding audio soundtrack  910 , and a corresponding temporal sequence of textual information  915 . The textual information  915  can take various forms, such as a closed caption information corresponding to the audio soundtrack, or overlay text provided over the image sequence  905 . An extract visual features step  920  is used to compute visual features  925  based on the image sequence  905 , an extract audio features step  921  is used to compute audio features  930  based on the audio soundtrack  910 , and an extract textual features step  922  is used to compute textual features  935  based on the temporal sequence of textual information  915 . 
     A variety of methods for performing the feature extraction steps are known in the digital multimedia record analysis art, and any such method can be used in accordance with the present invention. In some embodiments, the visual features  925  are the visual foreground temporal features  140  and visual background temporal features  145  discussed in  FIG. 3 , and the audio features  930  are the audio foreground temporal features  160  and audio background temporal features  165  discussed in  FIG. 4 . 
     In a preferred embodiment, a stream of textual characters are extracted from the textual information  915 . The extract textual features step  922  then determines the textual features  935  by forming words from groups of textual characters. Each word can then be treated as a textual feature  935 . The textual features  935  are temporal in that they occur at a particular time in the video sequence (i.e., they correspond to particular video frames). In some embodiments, additional processing steps can be used to reduce the dimensionality of the textual feature space. For example, less important words such as “a” and “the” can be eliminated, and words can be grouped together to form common phrases. In some embodiments, principle components analysis can be performed to further reduce the dimensionality of the textual feature space. In other embodiments, latent semantic indexing techniques can be used to determine the textual features  935 . 
     A visual codebook construction step  940  is used to generate visual vocabulary  945 , based on the visual features  925 . Likewise, an audio codebook construction step  941  is used to generate audio vocabulary  950  based on the audio features  930 , and a textual codebook construction step  942  is used to generate textual vocabulary  955  based on the textual features  935 . A variety of methods for performing the codebook construction steps are known in the digital multimedia record analysis art, for example, through clustering techniques, and any such method can be used in accordance with the present invention. 
     A grouplet construction step  960  is used to generate temporally-correlated grouplets  965  based on the visual vocabulary  945 , audio vocabulary  950  and textual vocabulary  955 . Each of the temporally-correlated grouplets includes at least one textual codeword (from the textual vocabulary  955 ) and at least one visual codeword (from the visual vocabulary  945 ) or audio codeword (from the audio vocabulary  950 ). 
     A distance metric learning step  970  is used to determine a distance metric  975  that can be used for computing pair-wise distances  980  between two digital records  900  based on the temporally-correlated grouplets  965 . A compute pair-wise distances step  976  is used to compute pair-wise distances  980  over set of a training data  977 . A train classifiers step  985  is used to train semantic concept classifiers  990  based on the pair-wise distances  980  computed over the set of training data  977 . The semantic concept classifiers  990  are adopted to classify semantic concepts in digital records  900 . In a preferred embodiment, the semantic concept classifiers  990  are well-known Support Vector Machine (SVM) classifiers. Methods for training SVM classifiers are well-known in the image and video analysis art. 
     Additional details for a preferred embodiment of the grouplet construction step  960  in  FIG. 9  will now be discussed with reference to  FIG. 10 . Similar to the process of generating visual foreground and background temporal features  140  and  145  in  FIG. 3 , visual temporal features  1010  are computed over time corresponding to the visual vocabulary  945 . Given a digital record  900  (video V j ), each visual feature  925  extracted from the image sequence  905  in the video is labeled with the visual codeword in the visual vocabulary  945  that is closest to the visual feature  925  in the visual feature space. Next, for each video frame I ji  in the video, the frequency of occurrence is counted for each visual codeword labeled to the visual features  925  corresponding to this video frame, and a histogram H j1   v  is generated. After this computation, for each video V j , visual temporal features  1010  are obtained as {H j1   v , H j2   v , . . . }. 
     Similar to the process of generating audio foreground and background temporal features  160  and  165  in  FIG. 4 , audio temporal features  1020  are computed over time corresponding to the audio vocabulary  950 . Given the audio soundtrack  910  for video V j , each audio feature  930  extracted from the soundtrack is labeled with the audio codeword in the audio vocabulary  950  that is closest to the audio feature  930  in the audio feature space. Next, for a short window surrounding each video frame I ji  in the audio soundtrack  910 , the frequency of occurrence is counted for each audio codeword labeled to the audio features  930  corresponding to the short window, and a histogram H j1  a is generated. After this computation, for each video V j , audio temporal features  1020  are obtained as {H j1   a , H j2   a , . . . }. 
     Given the temporal textual information  915  corresponding to video V j , each textual feature  935  extracted from the textual information  915  is labeled with the textual codeword in the textual vocabulary  955  that is closest to the feature in the textual feature space. Next, for a short window surrounding each video frame I ji  in the temporal textual information, the frequency of occurrence is counted for each textual codeword labeled to the textual features  935  corresponding to the short window, and a histogram H j1   t  is generated. After this computation, for each video V j , textual temporal features  1030  are obtained as {H j1   t , H j2   t , . . . }. 
     For each vocabulary (i.e., the visual vocabulary  945 , the audio vocabulary  950 , and the textual vocabulary  955 ), each codeword w k  in the vocabulary can be treated as a point process, N k   f−v (t), which counts the number of occurrences of w k  in the interval (0, t]. Point processes generated by all codewords of the vocabulary form a multivariate point process. Given each video V j , the visual temporal features  1010 , audio temporal features  1020 , and textual temporal features  1030  give one realization (trial) of the multivariate point process, respectively, corresponding to the visual vocabulary  945 , audio vocabulary  950 , and textual vocabulary  955 . 
     The pair-wise nonparametric Granger causality are then computed between each pair of textual and visual codewords, and each pair of textual and audio codewords, using a compute nonparametric Granger causality step  1040 . In a preferred embodiment, the compute nonparametric Granger causality step  1040  uses the method developed in the aforementioned article by Nedungadi et al. entitled “Analyzing multiple spike trains with nonparametric granger causality.”. Then for the textual vocabulary  955  and the visual vocabulary  945  pair, a causal matrix  1050  is computed, where each entry of the causal matrix  1050  is the nonparametric Granger causality between the corresponding pair of textual and visual codewords. Similarly for the textual vocabulary  955  and audio vocabulary  950  pair, a causal matrix  1050  is computed. A causal matrix  1050  can also be computed corresponsive to all the three of the audio vocabulary  950 , the visual vocabulary  945  and the textual vocabulary  955 , which captures the temporal causal relation between each pair of audio, visual, and textual codewords. 
     Next, a spectral clustering step  1060  is applied directly based on the causal matrices  1050  to identify clusters of codewords that have high temporal correlations. Each resulting cluster is called a temporally-correlated grouplet  965 . Each temporally-correlated grouplet  965  will contain textual codewords and either visual codewords or audio codewords or both. Each temporally-correlated grouplet can be treated as a multi-modal pattern, and all grouplets form a grouplet dictionary. 
     Additional details for a preferred embodiment of the distance metric learning step  970  in  FIG. 9  will now be discussed with reference to  FIG. 11 . Assume that there are K temporally-correlated grouplets  965 , denoted by G k  (k=1, . . . , K). Let D k   G (x i , x j ) denote the grouplet-based distance  1100  between data x i  and x j  computed based on the grouplet G k . The overall pair-wise distance  980  D(x i ,x j ) between data x i  and x j  is given by: 
     
       
         
           
             
               
                 
                   
                     D 
                      
                     
                       ( 
                       
                         
                           x 
                           i 
                         
                          
                         
                           x 
                           j 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         k 
                         = 
                         1 
                       
                       K 
                     
                      
                     
                       
                         v 
                         k 
                       
                        
                       
                         
                           
                             D 
                             k 
                             G 
                           
                            
                           
                             ( 
                             
                               
                                 x 
                                 i 
                               
                               , 
                               
                                 x 
                                 j 
                               
                             
                             ) 
                           
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
     The well-known SVM classifiers are generally used as the concept classifiers due to their good performances in classifying generic videos, and the RBF-like kernels shown in Eq. (14) are found to provide state-of-the-art performances in several semantic concept classification tasks: 
         K ( x   i   ,x   j )=exp{−γ D ( x   i   ,x   j )}.  (14)
 
     For example, the chi-square RBF kernel usually performs well with histogram-like features, where distance D(x i ,x j ) in Eq. (14) is the chi-square distance. 
     It is not trivial, however, to directly learn the optimal weights v k  (k=1, . . . , K) in the SVM optimization setting, due to the exponential function in RBF-like kernels. In the current embodiment of the invention, an iterative QP problem is formulated to learn optimal weights v k . The basic concept is to incorporate the Large-Margin Nearest Neighbor (LMNN) setting for distance metric learning that is described by Weinberger et al. in the article “Distance metric learning for large margin nearest neighbor classification” (Journal of Machine Learning Research, Vol. 10, pp. 207-244 2009). The rationale is that the role of large margin in LMNN is inspired by its role in SVMs, and LMNN should inherit various strengths of SVMs. Therefore, although weights v k  are not directly optimized in the SVM optimization setting, the final optimal weights can still provide reasonably good performance for SVM concept classifiers. 
     A set of training data  1110  is provided, denoted by (x i , y i ) (i=1, . . . , N), where y i ε{ 1 , . . . , c}, and c is the number of classes. For LMNN classification, the training process has two steps. First, n k  similarly labeled target neighbors are identified for each input training datum x i . The target neighbors are selected by using prior knowledge or by simply computing the n k  nearest (similarly labeled) neighbors using the Euclidean distance. Let η ji =1 (or 0) denote that x j  is a target neighbor of x i  (or not). In the second step, the distance metric is adapted so that these target neighbors are closer to x i  than all other differently labeled training data. Let y j1 ε{0,1} indicate whether inputs x i  and x 1  have the same class label. ε ji1  is the amount by which a differently labeled input x 1  invades the “perimeter” around the input x i  defined by its target neighbor x j . By defining D(x i ,x j )=[D 1   G (x i ,x j ), . . . , D K   G (x i ,x j )] T  and v=[v 1 , . . . , v K ] T , the following optimization problem can be formulated: 
     
       
         
           
             
               
                 
                   
                     
                       
                         min 
                         v 
                       
                        
                       J 
                     
                     = 
                     
                       
                         min 
                         v 
                       
                        
                       
                         { 
                         
                           
                             
                               
                                  
                                 v 
                                  
                               
                               2 
                               2 
                             
                             2 
                           
                           + 
                           
                             
                               C 
                               0 
                             
                              
                             
                               
                                 ∑ 
                                 ij 
                               
                                
                               
                                 
                                   η 
                                   ij 
                                 
                                  
                                 
                                   v 
                                   T 
                                 
                                  
                                 
                                   D 
                                    
                                   
                                     ( 
                                     
                                       
                                         x 
                                         i 
                                       
                                       , 
                                       
                                         x 
                                         j 
                                       
                                     
                                     ) 
                                   
                                 
                               
                             
                           
                           + 
                           
                             C 
                              
                             
                               
                                 ∑ 
                                 
                                   ij 
                                    
                                   
                                       
                                   
                                    
                                   1 
                                 
                               
                                
                               
                                 
                                   
                                     η 
                                     ij 
                                   
                                    
                                   
                                     ( 
                                     
                                       1 
                                       - 
                                       
                                         y 
                                         
                                           i 
                                            
                                           
                                               
                                           
                                            
                                           1 
                                         
                                       
                                     
                                     ) 
                                   
                                 
                                  
                                 
                                   ɛ 
                                   
                                     ij 
                                      
                                     
                                         
                                     
                                      
                                     1 
                                   
                                 
                               
                             
                           
                         
                         } 
                       
                     
                   
                    
                   
                     
 
                   
                    
                   
                       
                   
                    
                   
                     
                       
                         
                           
                             s 
                             . 
                             t 
                             . 
                             
                                 
                             
                              
                             
                               v 
                               T 
                             
                           
                            
                           
                             D 
                              
                             
                               ( 
                               
                                 
                                   x 
                                   i 
                                 
                                 , 
                                 
                                   x 
                                   1 
                                 
                               
                               ) 
                             
                           
                         
                         - 
                         
                           
                             v 
                             T 
                           
                            
                           
                             D 
                              
                             
                               ( 
                               
                                 
                                   x 
                                   i 
                                 
                                 , 
                                 
                                   x 
                                   j 
                                 
                               
                               ) 
                             
                           
                         
                       
                       ≥ 
                       
                         1 
                         - 
                         
                           ɛ 
                           
                             ij 
                              
                             
                                 
                             
                              
                             1 
                           
                         
                       
                     
                     , 
                     
                       
                         ɛ 
                         
                           ij 
                            
                           
                               
                           
                            
                           1 
                         
                       
                       ≥ 
                       0 
                     
                     , 
                     
                       
                         v 
                         k 
                       
                       ≥ 
                       0 
                     
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
           
         
       
     
     where ∥v 2 ∥ 2  is the L 2  regularization that controls the complexity of v. By introducing Lagrangian multipliers μ ij1 ≧0,γ ij1 ≧0, and σ k ≧0, the following optimization formulation can be obtained: 
     
       
         
           
             
               
                 
                   
                     
                       
                         min 
                         v 
                       
                        
                       J 
                     
                     = 
                     
                       
                         min 
                         v 
                       
                        
                       
                         
                           { 
                           
                             
                               
                                 
                                   
                                     
                                       
                                          
                                         v 
                                          
                                       
                                       2 
                                       2 
                                     
                                     2 
                                   
                                   + 
                                   
                                     
                                       C 
                                       0 
                                     
                                      
                                     
                                       
                                         ∑ 
                                         ij 
                                       
                                        
                                       
                                         
                                           η 
                                           ij 
                                         
                                          
                                         
                                           v 
                                           T 
                                         
                                          
                                         D 
                                          
                                         
                                           ( 
                                           
                                             
                                               x 
                                               i 
                                             
                                             , 
                                             
                                               x 
                                               j 
                                             
                                           
                                           ) 
                                         
                                       
                                     
                                   
                                   + 
                                   
                                     C 
                                      
                                     
                                       
                                         ∑ 
                                         
                                           ij 
                                            
                                           
                                               
                                           
                                            
                                           1 
                                         
                                       
                                        
                                       
                                         
                                           
                                             η 
                                             ij 
                                           
                                            
                                           
                                             ( 
                                             
                                               1 
                                               - 
                                               
                                                 y 
                                                 
                                                   i 
                                                    
                                                   
                                                       
                                                   
                                                    
                                                   1 
                                                 
                                               
                                             
                                             ) 
                                           
                                         
                                          
                                         
                                           ɛ 
                                           
                                             ij 
                                              
                                             
                                                 
                                             
                                              
                                             1 
                                           
                                         
                                       
                                     
                                   
                                   - 
                                 
                               
                             
                             
                               
                                 
                                   
                                     
                                       ∑ 
                                       
                                         ij 
                                          
                                         
                                             
                                         
                                          
                                         1 
                                       
                                     
                                      
                                     
                                       
                                         μ 
                                         ij 
                                       
                                        
                                       
                                         
                                           η 
                                           ij 
                                         
                                          
                                         
                                           [ 
                                           
                                             
                                               
                                                 v 
                                                 T 
                                               
                                                
                                               
                                                 D 
                                                  
                                                 
                                                   ( 
                                                   
                                                     
                                                       x 
                                                       i 
                                                     
                                                     , 
                                                     
                                                       x 
                                                       1 
                                                     
                                                   
                                                   ) 
                                                 
                                               
                                             
                                             - 
                                             
                                               
                                                 v 
                                                 T 
                                               
                                                
                                               
                                                 D 
                                                  
                                                 
                                                   ( 
                                                   
                                                     
                                                       x 
                                                       i 
                                                     
                                                     , 
                                                     
                                                       x 
                                                       j 
                                                     
                                                   
                                                   ) 
                                                 
                                               
                                             
                                             - 
                                             1 
                                             + 
                                             
                                               ɛ 
                                               
                                                 ij 
                                                  
                                                 
                                                     
                                                 
                                                  
                                                 1 
                                               
                                             
                                           
                                           ] 
                                         
                                       
                                     
                                   
                                   - 
                                 
                               
                             
                             
                               
                                 
                                   
                                     
                                       ∑ 
                                       
                                         ij 
                                          
                                         
                                             
                                         
                                          
                                         1 
                                       
                                     
                                      
                                     
                                       
                                         γ 
                                         
                                           ij 
                                            
                                           
                                               
                                           
                                            
                                           1 
                                         
                                       
                                        
                                       
                                         η 
                                         
                                           ij 
                                            
                                           
                                               
                                           
                                            
                                           1 
                                         
                                       
                                        
                                       
                                         ɛ 
                                         
                                           ij 
                                            
                                           
                                               
                                           
                                            
                                           1 
                                         
                                       
                                     
                                   
                                   - 
                                   
                                     
                                       ∑ 
                                       k 
                                     
                                      
                                     
                                       
                                         σ 
                                         k 
                                       
                                        
                                       
                                         v 
                                         k 
                                       
                                     
                                   
                                 
                               
                             
                           
                           } 
                         
                         . 
                         
                           
 
                         
                          
                         
                             
                         
                          
                         Next 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
             
               
                 
                   
                       
                   
                    
                   
                     
                       
                         ∂ 
                         J 
                       
                       
                         ∂ 
                         
                           ɛ 
                           
                             ij 
                              
                             
                                 
                             
                              
                             1 
                           
                         
                       
                     
                     = 
                     
                       
                         0 
                         ⇒ 
                         
                           
                             C 
                              
                             
                                 
                             
                              
                             
                               
                                 η 
                                 ij 
                               
                                
                               
                                 ( 
                                 
                                   1 
                                   - 
                                   
                                     y 
                                     
                                       i 
                                        
                                       
                                           
                                       
                                        
                                       1 
                                     
                                   
                                 
                                 ) 
                               
                             
                           
                           - 
                           
                             
                               μ 
                               
                                 ij 
                                  
                                 
                                     
                                 
                                  
                                 1 
                               
                             
                              
                             
                               η 
                               ij 
                             
                           
                           - 
                           
                             
                               y 
                               
                                 ij 
                                  
                                 
                                     
                                 
                                  
                                 1 
                               
                             
                              
                             
                               η 
                               ij 
                             
                           
                         
                       
                       = 
                       0. 
                     
                   
                 
               
               
                 
                   ( 
                   17 
                   ) 
                 
               
             
           
         
       
     
     That is, for any pair of x i  and its target neighbor x j , since only those x i  with y i1 =0 are considered, 0≦μ i1 ≦C. Based on Eq. (17), Eq. (16) turns into: 
     
       
         
           
             
               
                 
                   
                     
                       
                         min 
                         v 
                       
                        
                       J 
                     
                     = 
                     
                       
                         min 
                         v 
                       
                        
                       
                         { 
                         
                           
                             
                               
                                 
                                   
                                     
                                        
                                       v 
                                        
                                     
                                     2 
                                     2 
                                   
                                   2 
                                 
                                 + 
                                 
                                   
                                     C 
                                     0 
                                   
                                    
                                   
                                     
                                       ∑ 
                                       ij 
                                     
                                      
                                     
                                       
                                         η 
                                         ij 
                                       
                                        
                                       
                                         v 
                                         T 
                                       
                                        
                                       D 
                                        
                                       
                                         ( 
                                         
                                           
                                             x 
                                             i 
                                           
                                           , 
                                           
                                             x 
                                             j 
                                           
                                         
                                         ) 
                                       
                                     
                                   
                                 
                                 - 
                               
                             
                           
                           
                             
                               
                                 
                                   
                                     ∑ 
                                     
                                       ij 
                                        
                                       
                                           
                                       
                                        
                                       1 
                                     
                                   
                                    
                                   
                                     
                                       μ 
                                       ij 
                                     
                                      
                                     
                                       
                                         η 
                                         ij 
                                       
                                        
                                       
                                         [ 
                                         
                                           
                                             
                                               v 
                                               T 
                                             
                                              
                                             
                                               D 
                                                
                                               
                                                 ( 
                                                 
                                                   
                                                     x 
                                                     i 
                                                   
                                                   , 
                                                   
                                                     x 
                                                     1 
                                                   
                                                 
                                                 ) 
                                               
                                             
                                           
                                           - 
                                           
                                             
                                               v 
                                               T 
                                             
                                              
                                             
                                               D 
                                                
                                               
                                                 ( 
                                                 
                                                   
                                                     x 
                                                     i 
                                                   
                                                   , 
                                                   
                                                     x 
                                                     j 
                                                   
                                                 
                                                 ) 
                                               
                                             
                                           
                                         
                                         ] 
                                       
                                     
                                   
                                 
                                 - 
                                 
                                   
                                     v 
                                     T 
                                   
                                    
                                   σ 
                                 
                               
                             
                           
                         
                         } 
                       
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                     
                       where 
                        
                       
                           
                       
                        
                       σ 
                     
                     = 
                     
                       
                         
                           [ 
                           
                             
                               σ 
                               1 
                             
                             , 
                             … 
                              
                             
                                 
                             
                             , 
                             
                               σ 
                               k 
                             
                           
                           ] 
                         
                         T 
                       
                       . 
                       
                           
                       
                        
                       Thus 
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   18 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         
                           
                             ∂ 
                             J 
                           
                           
                             ∂ 
                             v 
                           
                         
                         = 
                           
                          
                         0 
                       
                     
                   
                   
                     
                       
                         ⇒ 
                           
                          
                         v 
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                               ∑ 
                               
                                 ij 
                                  
                                 
                                     
                                 
                                  
                                 1 
                               
                             
                              
                             
                               
                                 μ 
                                 
                                   ij 
                                    
                                   
                                       
                                   
                                    
                                   1 
                                 
                               
                                
                               
                                 
                                   η 
                                   ij 
                                 
                                  
                                 
                                   [ 
                                   
                                     
                                       D 
                                        
                                       
                                         ( 
                                         
                                           
                                             x 
                                             i 
                                           
                                           , 
                                           
                                             x 
                                             1 
                                           
                                         
                                         ) 
                                       
                                     
                                     - 
                                     
                                       D 
                                        
                                       
                                         ( 
                                         
                                           
                                             x 
                                             i 
                                           
                                           , 
                                           
                                             x 
                                             j 
                                           
                                         
                                         ) 
                                       
                                     
                                   
                                   ] 
                                 
                               
                             
                           
                           + 
                           σ 
                           - 
                           
                             
                               C 
                               0 
                             
                              
                             
                               
                                 ∑ 
                                 ij 
                               
                                
                               
                                 
                                   η 
                                   ij 
                                 
                                  
                                 
                                   D 
                                    
                                   
                                     ( 
                                     
                                       
                                         x 
                                         i 
                                       
                                        
                                       
                                         x 
                                         j 
                                       
                                     
                                     ) 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
     Define set P  1120  as the set of indexes i,j, 1  that satisfy the conditions of η ji =1, y i1 =0 and that x 1  invades the “perimeter” around the input x i  defined by its target neighbor x j , i.e., 0≦D(x i , x 1 )−D(x i , x j )≦1. Define set Q  1130  as the set of indexes i,j that satisfy η ji =1. Next, use μ p ,pεP to replace the original notation μ ij1 , use D P   p , pεP to replace the corresponding D(x i ,x 1 )−D(x i ,x j ) in define D P   p  step  1140 , and use D Q   q , qεQ to replace the corresponding D(x i ,x j ) in define D Q   p  step  1150 . Next, define u=[μ 1 , . . . , μ |P| ] T , |P|×K matrix D P =∥D P   1 , . . . , D P   |P| ] T , and |Q|ΔK matrix D Q =[D Q   1 , . . . , D Q   |Q| ] T . Through some derivation, the dual of Eq. (18) can be obtained as follows: 
     
       
         
           
             
               
                 
                   
                     
                       max 
                       
                         σ 
                         , 
                         u 
                       
                     
                      
                     
                       { 
                       
                         
                           
                             
                               
                                 
                                   - 
                                   
                                     1 
                                     2 
                                   
                                 
                                  
                                 
                                   u 
                                   T 
                                 
                                  
                                 
                                   D 
                                   P 
                                 
                                  
                                 
                                   D 
                                   P 
                                   T 
                                 
                                  
                                 u 
                               
                               + 
                               
                                 
                                   C 
                                   0 
                                 
                                  
                                 
                                   u 
                                   T 
                                 
                                  
                                 
                                   D 
                                   P 
                                 
                                  
                                 
                                   D 
                                   Q 
                                   T 
                                 
                                  
                                 
                                   1 
                                   Q 
                                 
                               
                               + 
                             
                           
                         
                         
                           
                             
                               
                                 
                                   u 
                                   T 
                                 
                                  
                                 
                                   1 
                                   P 
                                 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                  
                                 
                                   σ 
                                   T 
                                 
                                  
                                 σ 
                               
                               - 
                               
                                 
                                   u 
                                   T 
                                 
                                  
                                 
                                   D 
                                   P 
                                 
                                  
                                 σ 
                               
                               + 
                               
                                 
                                   C 
                                   0 
                                 
                                  
                                 
                                   σ 
                                   T 
                                 
                                  
                                 
                                   D 
                                   Q 
                                   T 
                                 
                                  
                                 
                                   1 
                                   Q 
                                 
                               
                             
                           
                         
                       
                       } 
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   20 
                   ) 
                 
               
             
           
         
       
     
     where  1   Q  is a |Q|-dim vector whose elements are all ones, and  1   P  is a |P|-dim vector whose elements are all ones. 
     When σ is fixed, Eq. (20) can be further rewritten to the following QP problem: 
     
       
         
           
             
               
                 
                   
                     
                       max 
                       u 
                     
                      
                     
                       { 
                       
                         
                           
                             - 
                             
                               1 
                               2 
                             
                           
                            
                           
                             u 
                             T 
                           
                            
                           
                             D 
                             P 
                           
                            
                           
                             D 
                             P 
                             T 
                           
                            
                           u 
                         
                         + 
                         
                           
                             u 
                             T 
                           
                            
                           
                             ( 
                             
                               
                                 
                                   C 
                                   0 
                                 
                                  
                                 
                                   D 
                                   P 
                                 
                                  
                                 
                                   D 
                                   Q 
                                   T 
                                 
                                  
                                 
                                   1 
                                   Q 
                                 
                               
                               + 
                               
                                 1 
                                 P 
                               
                               - 
                               
                                 
                                   D 
                                   P 
                                 
                                  
                                 σ 
                               
                             
                             ) 
                           
                         
                       
                       } 
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                     s 
                     . 
                     t 
                     . 
                     
                         
                     
                      
                     
                       ∀ 
                       
                         p 
                         ∈ 
                         P 
                       
                     
                   
                   , 
                   
                     0 
                     ≤ 
                     
                       μ 
                       p 
                     
                     ≤ 
                     
                       C 
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   21 
                   ) 
                 
               
             
           
         
       
     
     On the other hand, when u is fixed, Eq. (20) turns into the following QP problem: 
     
       
         
           
             
               
                 
                   
                     
                       max 
                       u 
                     
                      
                     
                       { 
                       
                         
                           
                             - 
                             
                               1 
                               2 
                             
                           
                            
                           
                             σ 
                             T 
                           
                            
                           σ 
                         
                         + 
                         
                           
                             σ 
                             T 
                           
                            
                           
                             ( 
                             
                               
                                 
                                   C 
                                   0 
                                 
                                  
                                 
                                   D 
                                   Q 
                                   T 
                                 
                                  
                                 
                                   1 
                                   Q 
                                 
                               
                               + 
                               
                                 
                                   D 
                                   P 
                                   T 
                                 
                                  
                                 u 
                               
                             
                             ) 
                           
                         
                       
                       } 
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                     
                       s 
                       . 
                       t 
                       . 
                       
                           
                       
                        
                       
                         ∀ 
                         k 
                       
                     
                     = 
                     1 
                   
                   , 
                   … 
                    
                   
                       
                   
                   , 
                   K 
                   , 
                   
                     
                       σ 
                       p 
                     
                     ≤ 
                     0. 
                   
                 
               
               
                 
                   ( 
                   22 
                   ) 
                 
               
             
           
         
       
     
     Therefore, the QP problems of Eq. (21) and Eq. (22) can be iteratively solved in iterative QP optimization module  1160 , and the desired weights  1170  (v) can be computed through Eq. (19) based on the optimal σ and u from the iterative optimization. 
     A variety of methods for computing the grouplet-based distance  1100  between data points are known in the art. For example, computing the Euclidean distance, or the chi-square distance, or any such method can be used in accordance with the present invention. 
     A preferred embodiment of a method for determining a semantic concept classification  1265  for an input digital record  1200  including an image sequence  1205  comprising a temporal sequence of video frames, a corresponding audio soundtrack  1210 , and a corresponding temporal sequence of textual information  1215  will now be discussed with reference to  FIG. 12 . An extract visual features step  1220  is used to extract visual features  1225  from the image sequence  1205 ; an extract audio features step  1221  is used to extract audio features  1230  from the audio soundtrack  1210 ; and an extract textual features step  1222  is used to extract textual features  1235  from the textual information  1215  using methods analogous to the equivalent steps discussed relative to  FIG. 9 . 
     Next, codeword similarity scores  1240  are computed for the input digital record  1200  by comparing the visual features  1225  to visual codewords in the visual vocabulary  945 , comparing the audio features  1230  to audio codewords in the audio vocabulary  950 , and comparing the textual features  1235  to textual codewords in the textual vocabulary  955 . 
     Reference digital record similarity scores  1250  are then determined using a determine reference digital record similarity scores step  1245  by computing distances between the input digital record  1200  and a set of reference digital records. Reference video codeword similarity scores  1255  are determined for the set of reference digital records using the same approach that was described for determining the codeword similarity scores  1240  from the input digital records  1200 . The reference digital record similarity score  1250  for each reference digital record is determined based on computing a distance between the codeword similarity scores  1240  and the reference video codeword similarity scores  1255 , responsive to the temporally-correlated grouplets  965  and the distance metric  975 . The resulting reference digital record similarity scores  1250  are then fed into the semantic concept classifiers  990  using a determine concept classification step  1260  to determine a semantic concept classification  1265  for the input digital record  1200  (e.g., the reference digital record similarity scores  1250  can be used to generate kernel matrices for SVM classification). In some cases, a plurality of different concept classifications can be assigned to the input digital record  1200 . In some embodiments, indications of the determined semantic concept classifications  1265  are stored as metadata in association with the input digital record  1200 . 
     The preferred embodiment of the current invention was evaluated over the large-scale CCV set described in the aforementioned article by Jiang, et al. entitled “Consumer video understanding: A benchmark database and an evaluation of human and machine performance.” The experiment used similar settings to those used by Jiang et al. That is, the same training set (including 4659 videos) and test set (including 4658 videos) were used, and the same one-versus-all χ 2  kernel SVM classifier was used for the semantic concept classifiers  990 . 
     Five different approaches were computed: the aggregated BoW (“agg-BoW”) that was used for experiments in  FIG. 8 ; the standard chi-square RBF kernel that does not use any grouplet information (“χ 2 -RBF”); the chi-square RBF kernel that uses the inverse document frequency (idf) information but does not use any grouplet information (“idf-χ 2 -RBF”); the weighted chi-square RBF kernel with the proposed distance metric learning that uses the grouplets (“w-χ 2 -RBF”); and the weighted chi-square RBF kernel with the proposed distance metric learning that uses both the idf information and the grouplets (“w-idf-χ 2 -RBF”). For both “χ 2 -RBF” and “w-χ 2 -RBF”, the grouplet-based distance  1100  is computed as the following chi-square distance: 
     
       
         
           
             
               
                 
                   
                     
                       
                         D 
                         k 
                         G 
                       
                        
                       
                         ( 
                         
                           
                             x 
                             i 
                           
                           , 
                           
                             x 
                             j 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         ∑ 
                         
                           
                             w 
                             m 
                           
                           ∈ 
                           
                             G 
                             k 
                           
                         
                       
                        
                       
                         
                           
                             [ 
                             
                               
                                 
                                   f 
                                   
                                     w 
                                     m 
                                   
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     i 
                                   
                                   ) 
                                 
                               
                               - 
                               
                                 
                                   f 
                                   
                                     w 
                                     m 
                                   
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                             
                             ] 
                           
                           2 
                         
                         
                           
                             1 
                             2 
                           
                            
                           
                             [ 
                             
                               
                                 
                                   f 
                                   
                                     w 
                                     m 
                                   
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     i 
                                   
                                   ) 
                                 
                               
                               + 
                               
                                 
                                   f 
                                   
                                     w 
                                     m 
                                   
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                             
                             ] 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   23 
                   ) 
                 
               
             
           
         
       
     
     where f w     m    (x i ) is the feature of x i  corresponding to the codeword w m  in the grouplet G k . For both “idf-χ 2 -RBF” and “w-idf-χ 2 -RBF”, the grouplet-based distance  1100  is computed as follows: 
     
       
         
           
             
               
                 
                   
                     
                       
                         D 
                         k 
                         G 
                       
                        
                       
                         ( 
                         
                           
                             x 
                             i 
                           
                           , 
                           
                             x 
                             j 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         1 
                         
                           
                             ∑ 
                             
                               
                                 w 
                                 m 
                               
                               ∈ 
                               
                                 G 
                                 k 
                               
                             
                           
                            
                           
                             idf 
                              
                             
                               ( 
                               
                                 w 
                                 m 
                               
                               ) 
                             
                           
                         
                       
                        
                       
                         
                           ∑ 
                           
                             
                               w 
                               m 
                             
                             ∈ 
                             
                               G 
                               k 
                             
                           
                         
                          
                         
                           
                             idf 
                              
                             
                               ( 
                               
                                 w 
                                 m 
                               
                               ) 
                             
                           
                            
                           
                             
                               
                                 [ 
                                 
                                   
                                     
                                       f 
                                       
                                         w 
                                         m 
                                       
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         i 
                                       
                                       ) 
                                     
                                   
                                   - 
                                   
                                     
                                       f 
                                       
                                         w 
                                         m 
                                       
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         
                                           j 
                                            
                                           
                                               
                                           
                                         
                                       
                                       ) 
                                     
                                   
                                 
                                 ] 
                               
                               2 
                             
                             
                               
                                 1 
                                 2 
                               
                                
                               
                                 [ 
                                 
                                   
                                     
                                       f 
                                       
                                         w 
                                         m 
                                       
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         i 
                                       
                                       ) 
                                     
                                   
                                   + 
                                   
                                     
                                       f 
                                       
                                         w 
                                         m 
                                       
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         j 
                                       
                                       ) 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   24 
                   ) 
                 
               
             
           
         
       
     
     where idf(w m ) is computed as the total number of occurrences of all codewords in the training corpus divided by the total number of occurrences of codeword w m  in the training corpus. 
       FIG. 13  compares the performance of the various approaches described above. It can be seen that the “w-χ 2 -RBF” and the “w-idf-χ 2 -RBF” approaches, which correspond to the method of  FIG. 12 , outperform all of the other methods. The improvements provided by the use of the grouplets can be seen from the fact that the “w-χ 2 -RBF” approach outperforms the “x 2 -RBF” approach, and the “w-idf-χ 2 -RBF” approach outperforms the “idf-χ 2 -RBF” approach. The advantage of the proposed “w-idf-χ 2 -RBF” is quite apparent (i.e., it performs the most efficiently in every case). Compared to the naive “agg-BoW” approach, the “w-idf-χ 2 -RBF” approach improves the final MAP by 26% on a relative basis. 
     A computer program product can include one or more non-transitory, tangible, computer readable storage medium, for example; magnetic storage media such as magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as optical disk, optical tape, or machine readable bar code; solid-state electronic storage devices such as random access memory (RAM), or read-only memory (ROM); or any other physical device or media employed to store a computer program having instructions for controlling one or more computers to practice the method according to the present invention. 
     The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention. 
     PARTS LIST 
     
         
           10  data processing system 
           20  peripheral system 
           30  user interface system 
           40  data storage system 
           100  video 
           105  shot boundary detection step 
           110  bad shot elimination step 
           115  video shot 
           120  image sequence 
           125  audio soundtrack 
           130  visual processing module 
           140  visual foreground temporal feature 
           145  visual background temporal feature 
           150  audio processing module 
           160  audio foreground temporal feature 
           165  audio background temporal feature 
           170  audio-visual temporal causal analysis module 
           180  audio-visual dictionaries 
           182  extract features step 
           185  dictionary-based features 
           187  train classifiers step 
           190  semantic concept classifiers 
           200  SIFT feature extraction step 
           205  SIFT feature matching step 
           210  sift point tracks 
           215  feature vectors 
           220  hierarchical clustering step 
           225  candidate foreground SIFT point tracks 
           230  candidate background SIFT point tracks 
           235  refine SIFT point tracks step 
           240  spatiotemporal image representation 
           245  foreground SIFT point tracks 
           250  background SIFT point tracks 
           255  clustering step 
           256  clustering step 
           260  visual foreground vocabulary 
           265  visual background vocabulary 
           270  visual foreground BoW features 
           275  visual background BoW features 
           300  MFCCs feature extraction step 
           310  MFCCs feature 
           320  clustering step 
           330  audio background vocabulary 
           340  audio background BoW features 
           350  transient event feature extraction step 
           360  transient feature 
           365  clustering step 
           370  audio foreground vocabulary 
           380  audio foreground BoW features 
           400  compute nonparametric Granger causality step 
           410  causal matrices 
           420  spectral clustering step 
           430  visual foreground audio foreground dictionary 
           440  visual foreground audio background dictionary 
           450  visual background audio foreground dictionary 
           460  visual background audio background dictionary 
           500  video 
           505  image frames 
           510  audio soundtrack 
           515  motion feature extraction step 
           520  SIFT features 
           525  motion features 
           530  MFCCs features 
           535  transient features 
           540  visual features 
           545  visual foreground BoW features 
           550  visual background BoW features 
           555  audio foreground BoW features 
           560  audio background BoW features 
           565  dictionary-based similarity scores 
           570  concept classification step 
           580  semantic concept 
           590  reference video codeword similarity scores 
           600  video clip 
           605  image frames 
           610  audio soundtrack 
           615  visual foreground 
           620  visual background 
           625  audio foreground 
           630  audio background 
           640  codeword similarity scores 
           650  AVG 
           700  reference video 
           710  reference video codeword similarity scores 
           715  AVG # 1  codewords 
           720  AVG similarity scores
         722  AVG # 1  similarity score   
     
           724  AVG #n similarity score 
           900  digital record 
           905  image sequence 
           910  audio soundtrack 
           915  textual information 
           920  extract visual features step 
           921  extract audio features step 
           922  extract textual features step 
           925  visual features 
           930  audio features 
           935  textual features 
           940  visual codebook construction step 
           941  audio codebook construction step 
           942  textual codebook construction step 
           945  visual vocabulary 
           950  audio vocabulary 
           955  textual vocabulary 
           960  grouplet construction step 
           965  temporally-correlated grouplets 
           970  distance metric learning step 
           975  distance metric 
           976  compute pair-wise distances 
           977  set of training data 
           980  pair-wise distances 
           985  train classifier step 
           990  semantic concept classifiers 
           1010  visual temporal features 
           1020  audio temporal features 
           1030  textual temporal features 
           1040  compute nonparametric Granger causality step 
           1050  causal matrices 
           1060  spectral clustering step 
           1100  grouplet-based distance 
           1110  training data 
           1120  set P 
           1130  set Q 
           1140  define D P   p  step 
           1150  define D Q   p  step 
           1160  iterative QP optimization module 
           1170  weights 
           1200  input digital record 
           1205  image sequence 
           1210  audio soundtrack 
           1215  textual information 
           1220  extract visual features step 
           1221  extract audio features step 
           1222  extract textual features step 
           1225  visual features 
           1230  audio features 
           1235  textual features 
           1240  codeword similarity scores 
           1245  determine reference digital record similarity scores step 
           1250  reference digital record similarity scores 
           1255  reference video codeword similarity scores 
           1260  determine concept classification step 
           1265  concept classification