Patent Description:
Feature detection and extraction was developed for image processing to identify objects (features) within an image. This may be useful as a part of computer vision, image search, object recognition, and other such image-analysis systems. The act of feature detection involves identifying points or areas of potential interest in an image (e.g. edges, corners, etc.), often using gradient analysis. The act of feature extraction involves generating a multi-dimensional vector representation of a small patch of the image containing the feature of interest. The resulting output of feature extraction is a feature descriptor.

Feature descriptors are then used, for example, in object recognition or image matching processes. For instance, the set of feature descriptors from an image may be stored in a query database containing sets of feature descriptors for a large number of images. This can enable such functions such as facial recognition, for example.

The concept of feature detection and extraction may also be extended to video, which may be considered a series of pictures. The Motion Picture Experts Group (MPEG) intends to standardize the descriptors for video as Compact Descriptors for Video Analysis (CDVA). MPEG is exploring tools for video processing and analysis with the goal of reducing "the data amount for the resulting video descriptors transmission or storage" and achieving "interoperability of implementations, applications and databases". Note that in MPEG CDVA, descriptors for a video sequence are extracted in a picture by picture (or frame by frame) manner, and then compressed.

The large number of feature descriptors that may be extracted from each picture of a video leads to a large data burden in identifying and storing and transmitting feature descriptors, even when compressed. It would be advantageous to provide for a system and method of feature descriptor extraction that maintains or improves the discriminative power (i.e. accuracy) of the feature descriptors while reducing or maintaining the number and complexity of the descriptors.

The present application describes methods and devices for extracting feature descriptors from a video. The methods and devices rely on temporal redundancy to identify "persistent descriptors", where a persistent descriptor is a feature descriptor of a feature that appears in multiple consecutive frames. Such a feature may be identified by determining that extracted feature descriptors from multiple consecutive frames are a sufficient match to each other that they likely relate to the same feature. Motion consistency checks may be used to ensure that persistent descriptors are consistent with the motion of the objects in the video. In some cases, persistent descriptors may be identified by matching extracted descriptors for two keyframes relating to a segment of video, where the keyframes are pictures at either end of the segment. The present invention is as defined in the appended set of claims.

In a first aspect, the present application describes a method of extracting feature descriptors for a video, in a video feature descriptor extractor, the video including a sequence of pictures. The method includes identifying a first key picture and a second key picture later in the sequence than the first key picture; extracting a first set of feature descriptors from the first key picture and a second set of feature descriptors from the second key picture; identifying a set of pairs of feature descriptors, where each pair includes one descriptor from the first set and one descriptor from the second set; generating motion information describing the motion field between the first key picture and the second key picture; and filtering the set of pairs of feature descriptors based on correlation with the motion information to produce and output a subset of persistent descriptors.

In a further aspect, the present application describes a video feature extractor configured to implement such methods of video feature descriptor extraction.

In yet a further aspect, the present application describes non-transitory computer-readable media storing computer-executable program instructions which, when executed, cause one or more processors to perform the described methods of video feature descriptor extraction.

In the description that follows, some example embodiments may make reference to various feature extraction processes, i.e. algorithms for detecting and/or representing a localized feature in an image. A common example of such processes include the Scale-Invariant Feature Transform (SIFT). Other examples include Speeded-Up Robust Features (SURF), Gradient Location and Orientation Histogram (GLOH), Local Energy-based Shape Histogram (LESH), Histogram of Oriented Gradients (HOG). In general, a feature extractor or feature extraction process produces a feature descriptor, which is a multi-dimensional representation of the image data around a certain local feature. Determining which features to extract or describe by way of feature descriptors may rely on any one or more of a number of feature detection processes or algorithms. Example feature detection processes include edge detection, corner detection (sometime termed "interest" detection), blob detection, ridge detection, or other algorithms designed to identify areas of potential interest in an image that may be significance in recognizing a particular object in that image. Many such algorithms may use gradient analysis to identify points or areas of potential interest or significance. When the present application makes reference to "feature extraction", that is analyzing an image or picture to extract one or more feature descriptors, it is not intended to refer only to extraction (determining the descriptor) and may include feature detection as a part of the extraction process.

A SIFT descriptor, for example, is a region around a detected potential point of interest. The region is divided into sixteen rectangles from which <NUM> histograms of the orientation and magnitude of the gradients inside each rectangle are constructed. The descriptor is then formed as the concatenation of those sixteen histograms in a prescribed scan order of the rectangles.

<FIG> shows an example of a picture <NUM> and a portion of an example SIFT descriptor extracted from that picture <NUM>. A feature is identified and the region around it is divided into sixteen rectangles b1, b2,. A portion of the concatenated histograms (specifically, the example histograms for b1 and b2) is illustrated.

In the description that follows, when referring to video the terms frame, picture, slice, tile, quantization group, and rectangular slice group may be used somewhat interchangeably. Those of skill in the art will appreciate that a picture or frame may contain one or more slices or segments. A consecutive series of frames/pictures may be called a "sequence" in some cases. Other terms may be used in other video implementations.

Feature detection and extraction was developed for image processing to identify objects (features) within an image. This may be useful as a part of computer vision, image search, object recognition, and other such image-analysis systems. The act of feature detection involves identifying points or areas of potential interest in an image (e.g. edges, corners, etc.), often using gradient analysis. The act of feature extraction involves generating a multi-dimensional vector representation of a small patch of the image containing the feature of interest. The resulting output of feature extraction is a feature descriptor. As noted above, SIFT is one example of a feature extractor.

To locate and extract descriptors from a picture, SIFT uses the following steps:.

Given an image as input, SIFT might generate a large number of key descriptors. For example, hundreds of key descriptors may be extracted from an image of size 384x512 (<NUM>,<NUM> pixels).

Those feature descriptors are then used, for example, in object recognition or image matching processes. For instance, the set of feature descriptors from an image may be stored in a query database containing set of feature descriptors for a large number of images. A new image may undergo feature extraction and a query system may attempt to find a match for those features in the query database by comparing the newly extracted feature descriptors from the new image to the stored sets of feature descriptors in the database. This can enable such functions as facial recognition, for example.

The concept of feature detection and extraction may also be extended to video, which is typically a series of pictures. However, as will be appreciated, video includes a large number of pictures, resulting in a large number of feature descriptors. MPEG intends to standardize the descriptors for video as Compact Descriptors for Video Analysis (CDVA). MPEG is exploring tools for video processing and analysis with the goal of reducing "the data amount for the resulting video descriptors transmission or storage" and achieving "interoperability of implementations, applications and databases". Note that in MPEG CDVA, descriptors for a video sequence are extracted in a picture by picture (or frame by frame) manner, and then compressed. That is, each picture in the video sequence is treated as a single image from which descriptors are located and extracted by using methods similar to SIFT. Inter-picture correlation is used in compression of descriptors after they are extracted from individual pictures.

<FIG> shows, in block diagram form, an example of a CDVA system <NUM>. The CDVA system <NUM> uses SIFT-like feature descriptor extractor <NUM> and treat the video as a series of video frames or pictures. The output from the feature descriptor extractor <NUM> is a set of descriptors <NUM> for each frame/picture. Each set of descriptors is then, in this example, encoded by a descriptor encoder <NUM> to produce a bitstream of compact descriptors <NUM>, which may then be stored in a database or other memory <NUM>.

The system <NUM> may include a feedback loop in the encoding process to improve the compression of the descriptors through predictive coding. In such an example implementation the system <NUM> includes a descriptor decoder <NUM> that decodes the compact descriptors of previous pictures in the sequence. In <FIG>, the dashed line indicates that it is optional to use descriptors from previous picture xi-<NUM> in coding of descriptors extracted from the current picture xi. Further note that in <FIG> coding of descriptors may be lossy and involve quantization in some examples.

It could be advantageous to improve the balance of accuracy (discriminative power) and compression of feature descriptors. Accuracy of feature descriptors can be measured by an evaluation framework. For example, MPEG released a unified evaluation framework for video descriptors in <NUM>. In MPEG's example framework, discriminative accuracy (i.e. performance) may be evaluated by comparing two ratios. One ratio assesses true positives to false negatives. The other ratio assesses false positives to true negatives. An improved accuracy leads to improved query response accuracy. In addition, to the extent that accuracy may be maintained or improved while reducing the number of feature descriptors per image or per video results in greater performance and savings in bandwidth (for transmitting feature descriptors), computational resources (for processing queries), and storage requirements (for a query database or local image representations), as examples.

The present application proposes methods and systems aimed at improving the discriminative accuracy of feature descriptors for video and potentially reducing the quantity of such feature descriptors in characterizing a video sequence. To this end, the present application proposes to exploit the temporal nature of video in the feature extraction process in order to identify and extract more robust and discriminative feature descriptors.

The present application introduces the concept of "persistent" feature descriptors. A persistent feature descriptor (also termed "persistent descriptor" herein) is one which persists through two or more consecutive pictures in a video sequence. That is, a sequence of descriptors from successive video frames are collectively called a persistent descriptor for these video frames if the descriptors in the sequence match each other (i.e. they are within a prescribed distance of each other). In some embodiments, the persistent feature descriptors may further be tested against the motion field of the image and those that deviate from the motion field are excluded as persistent descriptors. Experimental results on typical CDVA test video sequences show that a substantial portion of descriptors extracted from a video sequence are persistent.

<FIG> shows, in block diagram form, one example of a video feature extractor <NUM> for producing a set of persistent descriptors characterizing a video sequence. The video feature extractor <NUM> includes an extraction component <NUM> and an encoding component <NUM>. In some embodiments, the extractor <NUM> may exclude the encoding component <NUM> and may simply output unencoded sets of descriptors for storage, locally or remotely, or for generating one or more queries.

The encoding component <NUM> includes a descriptor encoder <NUM>, a local storage or memory <NUM>, and a descriptor decoder <NUM>. The feedback loop containing the descriptor decoder <NUM> enables the encoding process to encode the descriptors predictively. That is the descriptor encoder <NUM> may encode current descriptors from a current image, based on decoded descriptors from an earlier image in the sequence. Note that this predictive coding operation relates to exploiting the temporal nature of the video pictures so as to improve the coding and compression of the descriptors themselves after they have been identified and extracted.

The extraction component <NUM> includes a descriptor extractor <NUM> and a picture buffer <NUM>. The picture buffer <NUM> makes multiple pictures available to the descriptor extractor <NUM>, thereby enabling the descriptor extractor <NUM> to perform descriptor matching across multiple successive pictures so as to identify persistent descriptors.

In one embodiment, a direct method of finding persistent descriptors in a video sequence is implemented by the descriptor extractor <NUM>:.

This two-step method represents a simple and accurate approach to obtaining a set of persistent descriptors for a given video sequence. It locates and identifies descriptors based on spatiotemporal features and relevance to the content; however, it is computationally expensive. The complexity comes as a result of the need to process every frame independently to extract all feature descriptors and to search through the extracted descriptors for matches.

In another embodiment, rather than searching every frame or picture of a video sequence, the sequence is first analyzed to identify keyframes. By analyzing frame correlation, the sequence is divided into segments exhibiting high frame correlation among the frames of the segment. The keyframes may be the first and last of the pictures in the segment. In another embodiment, the keyframes are the first of the pictures in a segment, such that for any one segment, the keyframes for that segment are its first picture and the first picture from the next segment in the sequence. High frame correlation suggests that two frames could produce a set that encompasses the majority of the persistent descriptors. This has been shown experimentally to be true. Hence, this solution extracts descriptors from two frames in a segment (referred to as the keyframes), not all of them. The search for persistent descriptors may then be carried out between the descriptors of these two keyframes. The search may employ both geometric and motion information in some embodiments. The terms "keyframes", "key frames" and "key pictures" are used interchangeably herein.

Persistent descriptors are usually detected on different objects in the scene, some of which are moving and others are stationary. Descriptor-tracking experiments have shown that, in both cases, persistent descriptors exhibit the same motion field as that of their objects, e.g., if a descriptor is detected on a stationary object appearing in few consecutive frames, it will not exhibit any location change throughout these frames. Such motion compliance helps detect and preserve descriptors pertaining to salient objects - salient objects are those existing in multiple frames and conveying the main information of the scene.

Reference is now briefly made to <FIG>, which illustrates an example of a persistent descriptor. <FIG> features two pictures: a first picture <NUM> and a second picture <NUM>. In some examples, the first picture <NUM> and the second picture <NUM> are consecutive pictures in the video. In some examples, the first picture <NUM> may be a first keyframe and the second picture may be a second keyframe <NUM> for a given segment of the video. The identified feature and its regions in the two respective pictures <NUM>, <NUM> are shown below the pictures. The beginning portion of the feature descriptors (e.g. SIFT) extracted from the respective pictures <NUM>, <NUM> is shown as Descriptor <NUM> and Descriptor <NUM>. Descriptor <NUM> is matched (paired) to Descriptor <NUM> on the basis of its similarity to Descriptor <NUM> (non-identical, but within a certain Euclidean distance). The pairing of the two descriptors as relating to the same feature may be tested for consistency with the motion field to validate the pairing.

In this example embodiment, the process of feature extraction implemented by the feature extractor <NUM>, is given by:.

Note that in Step S1 above, the selection of xk<NUM> and xk<NUM> might depend upon motion information to make sure motion field is not interrupted between the selected pair of keyframes. In other words, S1 might include scene change detection to select keyframes.

In S3 above, motion information from xk<NUM> to xk<NUM> may be generated by using traditional block-based motion estimation methods typically used in video coding, where a picture is partitioned into blocks and a motion vector is identified for each block. In some embodiments, optic flow algorithms may be used to determine motion information to be used in S4.

The above-described solution analyses the motion field and defines descriptors with respect to both spatial and temporal features by following the motion field. As such, the solution enables extraction of generalized descriptors that capture temporal structures. By using generalized descriptors the solution allows for locating and identifying key regions, e.g., flat areas with temporal fluctuations, that otherwise may not be located by using image-based descriptor extraction schemes. A motion field is a 2D vector field for describing the motion of objects in a video sequence or segment. The motion field may be determined or estimated using an optical flow algorithm in some cases.

As noted above, the set of persistent descriptors output by the video feature extractor <NUM> may be used in a query system. Reference is now made to <FIG>, which shows, in block diagram form, one example embodiment of a query system <NUM> for recognition in video. The system <NUM>, in this example, includes a video database <NUM> containing sets of persistent feature descriptors for a number of video segments. The system <NUM> may be for determining whether a video segment has a match in the database <NUM>; that is, whether there is a video segment that shows the same features. This can be used for object recognition in some implementations.

The system <NUM> includes a query processor <NUM> that receives the query data, which in this case includes the persistent descriptors for the video segment that is the subject of the query. The system <NUM> may include a decoder <NUM> if the persistent descriptors are encoded. The query processor <NUM> compares the received (and decoded, if need be) persistent descriptors that form the query with persistent descriptors in the database to determine whether there is a sufficiently close match to declare a match between to two video segments. The process of comparing may include identifying pairs of persistent descriptors between the query segment and the database segment, determining Euclidean distance between paired descriptors, and assessing whether the differences between persistent descriptors is lower than a threshold level. That threshold level may be set based on the degree of concordance demanded by a particular application.

It will be appreciated that local descriptors extracted from a picture may be summarily represented as a single vector called the global descriptor for the picture. For example, each element in a global descriptor for a picture may indicate the number of local descriptors from the picture that are quantized to a member in a known set of descriptors, i.e. a vocabulary. In that case, the global descriptor is constructed as a histogram of the local descriptors given a vocabulary. For example, in some implementations, the global descriptor is formed using difference vectors obtained from every local descriptor and its match in the vocabulary set. The global descriptor is the final matrix reshaped to a long vector.

The benefit of having global descriptors for pictures is to allow a two-stage process in searching for a match in a large database, where simpler, shorter global descriptors are used exclusively in the first stage and local descriptors are used only if a match is found in the first stage.

The concept of a global descriptor for a picture may be extended to a global descriptor for a video segment. For example, given a vocabulary for a video segment, a histogram of the persistent descriptors in the segment may be constructed and used as the global descriptor for the video segment. Correspondingly, the following process may be used to search for a match:.

Reference is now made to <FIG>, which shows, in block diagram form, one example embodiment of a video feature extractor <NUM>. This example embodiment relies on SIFT-based feature identification and extraction and optical flow-based motion estimation, but it will be appreciated that other implementations may use other feature extraction techniques and other motion estimation techniques.

The example video feature extractor <NUM> receives the video sequence as an input. A segmentation process <NUM> divides the video sequence into segments and identifies keyframes within each segment. A feature extractor, such as a SIFT extractor <NUM>, then identifies features in the keyframes and extracts their descriptors. A matching process <NUM> is used to pair feature descriptors from the two keyframes of a segment. Meanwhile, a motion estimation process <NUM> determines the motion field between the two keyframes identified by the segmentation process <NUM>. The motion field is then used to evaluate the pairs of feature descriptors for consistency with the motion field in a motion-consistency check process <NUM>. If a pair of feature descriptors deviates from the motion field, then the pair may be discarded as a valid feature descriptor pair. In that sense, the motion-consistency check process <NUM> filters the set of pairs of feature descriptors generated by the matching process <NUM> to output a subset of pairs of feature descriptors that is consistent with the motion field of the segment.

The output subset of feature descriptors may then be stored in memory such as a local database or other data structure, or, as indicated in this example, may be encoded by a descriptor encoder <NUM> to produce an encoded bitstream of feature descriptors for the video.

Reference will now be made to <FIG>, which shows, in flowchart form, one example process <NUM> of extracting feature descriptors for a video. In this example process <NUM>, the video sequence is first divided into segments, as indicated by operation <NUM>. As noted above, scene change detection and other such analysis may be used to partition the sequence into segments. In some embodiments, the segments each contain a predetermined number of pictures/frames per segment.

In operation <NUM>, keyframes are identified for each segment. In one example implementation, the keyframes (or "key pictures") are the first frame/picture of the segment and the last frame/picture of the segment. In another example implementation, the keyframes for a segment are the first frame/picture of the segment and the first frame/picture of the next segment in the sequence (the consecutive segment). The two keyframes of a segment may be designated KF<NUM> and KF<NUM> respectively, in the following description.

In operation <NUM>, features descriptors are extracted from the keyframes. The extraction process may include SIFT or any other such image feature descriptor extracting process. Then, in operation <NUM>, pairs of matching descriptors are found. In this example, for each descriptor in the first keyframe, the system finds whether a matching descriptor appears in the second keyframe. The matching may include testing descriptors in the second keyframe to determine whether there is a descriptor in that keyframe within a particular search window and that is sufficiently similar to the descriptor in the first keyframe. Similarity, in the case of a descriptor that is a multi-dimensional vector, may be measured in terms of Euclidean distance, in some cases. In an example implementation, pairs of descriptors are found by, for each descriptor extracted from the first keyframe, identifying a descriptor from the second key frame based on a minimizing Euclidean distance between them. In another example implementation, a ratio test is applied to determine if a match exists. For example, the ratio test may involve:.

It will be appreciated that sensitivity of the determination of whether there is a match may be adjusted by adjusting the ratio outcome that is designated as a match. In the above example, a matching descriptor is identified if the ratio is < <NUM>, but other preset maximums may be used, such as <NUM> or <NUM>, depending on how selective the system is intended to be in identifying matches. The maximum ratio may be selected through empirical testing in some cases.

The example matching process in which a matching descriptor in the second keyframe is sought for every descriptor in the first keyframe results in the possibility that more than one descriptor in the first keyframe may be matched to the same descriptor in the second keyframe, as indicated by operation <NUM>. In the event that there are multiple pairs involving a match to the same descriptor from the second keyframe, in operation <NUM> the system determines the best match involving the same descriptor from the second keyframe and discards the other pairings. The best match may be evaluated in a number of ways. In one example, the quality of each of the matches is determined and the best of them is kept. The best match may rely on a comparison of Euclidean differences between the descriptors of the first keyframe and the descriptor in the second keyframe. In one example, the pairs are each given a score based on the ratio test of operation <NUM>. For instance, each pair may be scored based on the following expression in which r is the result of the ratio test in operation <NUM>: <MAT>.

The descriptor from KF<NUM> that has the highest score β with the descriptor from KF<NUM> is declared as the best match. Thus, unique pairs of descriptors are generated, i.e., a descriptor from KF<NUM> matches only one descriptor from KF<NUM>.

In operation <NUM>, a geometric consistency test may be applied to validate the pairs of descriptors. Those that do not pass the geometric consistency test are eliminated from the set of pairs of descriptors for the segment. An example of a geometric consistency test is provided in N14961, "Test Model <NUM>: compact descriptors for visual search," <NUM>, the contents of which are hereby incorporated by reference. This test checks whether the pair of descriptors likely represent the same regions in two frames. After finding the initial set of M matching pairs, Sinitial_matches ={ (dKF1, dKF2)<NUM> ,. , (dKF1, dKF2)M }, a matrix of Log Distance Ratios (LDR) called Z may be constructed. Every entry in Z represents the following ratio: <MAT>.

In the above expression, xi and yi are the coordinates of the ith matching pair in Sinitial_matches, and xj and yj are the coordinates of the jth matching pair in the same set Sinitial_matches. A histogram of the values in the matrix Z may be constructed and checked against a known-outlier probability distribution using, for example, Pearson's goodness-of-fit test.

In operation <NUM>, the system may determine the motion field from the first keyframe to the second keyframe and may evaluate each pair in the set of pairs of descriptors for consistency with the determined motion field. Pairs that are too inconsistent with the motion field are discarded, resulting in a subset of pairs of descriptors that may be designated as the persistent descriptors for that segment of the video. That subset of persistent descriptors, in particular the first descriptor in each pair, is then output in operation <NUM>. It will be appreciated that it is the descriptors of the first keyframe that are output as the persistent descriptors for a particular segment.

<FIG> shows, in flowchart form, one example implementation of operation <NUM> relating to motion-consistency testing. In general, motion-consistency testing involves assessing whether individual pairs of descriptors generally conform (in terms of their relative locations in the frames) to the motion field between the frames. In this example, the system uses an optical flow algorithm to determine the motion field of the segment, as indicated by operation <NUM>-<NUM>. One example of an optical flow algorithm is described in B. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," in International Joint Conference on Artificial Intelligence, <NUM>, but other such algorithms may be used.

In operation <NUM>-<NUM>, the system starts with a descriptor in the first keyframe and uses the motion field to pinpoint an estimated location for that feature location in the second keyframe. A search window is established around that estimated location in the second keyframe, as indicated by operation <NUM>-<NUM>. Then, in operation <NUM>-<NUM>, the system determines whether the matching descriptor from the pair falls inside of the search window. If so, then it generally conforms to the motion field. If not, then it deviates too far from the motion field and, in operation <NUM>-<NUM>, it is discarded. This is thought of as filtering the set of pairs of descriptors to remove those that are too inconsistent with the motion field. As indicated by operation <NUM>-<NUM>, the operation is performed for each pair of descriptors identified for the segment.

As noted above, criteria may be set for evaluating the pairwise matching capability of a feature descriptor extraction scheme. One example uses True Positive (TP) and False Positive (FP) Rates. They are given by the following two equations:
<MAT>
<MAT>
where:.

An embodiment of the presently-described solution achieves good TPR and FNR without imposing any restrictions on the average descriptor lengths per second at the output of the video feature extractor (no upper bound on the output bit rate). The experiments were implemented with the assumption that the approximate persistent set is perfectly recoverable at the decoder, i.e., a lossless encoding scheme is used. The recorded TPR and FPR were <NUM>% and less than <NUM>%, respectively, in testing.

Reference is now made to <FIG>, which shows a simplified block diagram of an example embodiment of a video feature extractor <NUM>. The encoder <NUM> includes a processor <NUM>, memory <NUM>, and a video feature descriptor extraction application <NUM>. The application <NUM> may include a computer program or application stored in memory <NUM> and containing instructions that, when executed, cause the processor <NUM> to perform operations such as those described herein. For example, the application <NUM> may identify and extract sets of persistent feature descriptors in accordance with the processes described herein. In some implementations, the application <NUM> may encode the sets of persistent feature descriptors and output them as an encoded bitstream. It will be understood that the application <NUM> may be stored in on a non-transitory computer-readable medium, such as a compact disc, flash memory device, random access memory, hard drive, etc. When the instructions are executed, the processor <NUM> carries out the operations and functions specified in the instructions so as to operate as a special-purpose processor that implements the described process(es). Such a processor may be referred to as a "processor circuit" or "processor circuitry" in some examples.

It will be appreciated that the video feature extractor according to the present application may be implemented in a number of computing devices, including, without limitation, servers, suitably-programmed general purpose computers, audio/video encoding devices, and mobile devices. The video feature extractor may be implemented by way of software containing instructions for configuring a processor or processors to carry out the functions described herein. The software instructions may be stored on any suitable non-transitory computer-readable memory, including CDs, RAM, ROM, Flash memory, etc..

It will be understood that the video feature extractor described herein and the module, routine, process, thread, or other software component implementing the described method/process for configuring the video feature extractor may be realized using standard computer programming techniques and languages. The present application is not limited to particular processors, computer languages, computer programming conventions, data structures, other such implementation details. Those skilled in the art will recognize that the described processes may be implemented as a part of computer-executable code stored in volatile or non-volatile memory, as part of an application-specific integrated chip (ASIC), etc..

Claim 1:
A method of extracting feature descriptors for a video, the video including a sequence of pictures, the method comprising:
identifying a first key picture and a second key picture later in the sequence than the first key picture;
extracting a first set of feature descriptors from the first key picture and a second set of feature descriptors from the second key picture;
identifying a set of pairs of feature descriptors, where each pair includes one descriptor from the first set and one descriptor from the second set;
generating motion information describing a motion field between the first key picture and the second key picture; and
filtering the set of pairs of feature descriptors based on correlation with the motion information to produce and output a subset of persistent descriptors,
wherein identifying the first and second key pictures comprises analyzing picture correlation of the sequence of pictures and dividing the sequence of pictures into segments exhibiting high picture correlation among respective pictures of a segment, each segment having a respective first key picture and a respective second key picture,
wherein filtering the set of pairs of feature descriptors includes, for each pair,
determining, based on the motion information and a location of pair's descriptor from the first key picture, an estimated location in the second key picture;
determining whether the pair's descriptor from the second key picture is located within a search window centered on the estimated location; and
if so, retaining the pair in the subset of persistent descriptors, and
if not, excluding the pair from the subset of persistent descriptors.