Patent Publication Number: US-9432702-B2

Title: System and method for video program recognition

Description:
FIELD OF THE INVENTION 
     The present invention relates to the field of video processing technology and, more particularly, to video systems and methods for recognizing video program using a mobile device. 
     BACKGROUND 
     With the development of the Internet and video delivery technologies, currently video has been a significant part of the Internet traffic. Also, people are used to multi-screen experiences when watching videos. For example, nearly half of US population use smart phones when watching TV, and it is observed that 47% of users use mobile phones to engage in TV related activities. 
     However, the human interaction with video is still quite limited due to the complexity of video structure. That is to say, the interactions between mobile phones and TVs are still quite limited. With existing technology, the mobile phone can be used to identify products, CDs, books, and other print media, but it is difficult or impractical for such mobile phone applications in the video area that deals with high-volume data. 
     For example, considering a TV program with 200 channels, that is, the throughput of the program is 6000 frames per second. If a 10-minute lagging period is allowed, the system needs to be able to recognize 3.6 Million pictures (=6000×10×60) with very low latency. Hence, the difficulty is twofold: high throughput to handle, and large image database for content recognition. Typically, the content recognition involves high computation amount, but the computation is generally related to the number of frames involved in classification/recognition. The more frames involved, the higher classification/recognition accuracy, but also higher amount of computation. 
     The disclosed methods and systems are directed to solve one or more problems set forth above and other problems. 
     BRIEF SUMMARY OF THE DISCLOSURE 
     One aspect of the present disclosure includes a method for video program recognition on a server system. The method includes recording video sequences of each of a plurality of channels and obtaining a set of features of an image captured by a first user device of a video frame from one of the plurality of channels and played on a second user device different from the first user device. The method also includes performing a camera registration operation on the video sequences of each of the plurality of channels to select a plurality of video frames representing the recorded video sequences of each of the plurality of channels based on key video frames from a plurality of registered cameras associated with the recorded video sequences. Further, the method includes extracting features of at least some of the selected plurality of video frames of each of the plurality of channels, and storing the extracted features of each of the plurality of channels. The method also includes comparing the obtained set of features of the image with the stored features of each of the plurality of channels using an image classifier to determine whether there is a match between the obtained set of features of the image and features of a video frame from a specific channel and, when there is a match between the obtained set of features of the image and features of a video frame from a specific channel, determining a channel number of the specific channel as a recognized channel number. 
     Another aspect of the present disclosure includes a video program recognition system. The system includes a recorder module, an image classifier module, a camera registration module, a feature extraction module, a feature extraction module, and a feature database. The recorder module is configured to record video sequences of each of a plurality of channels. The image classifier module is configured to obtain a set of features of an image captured by a first user device of a video frame from one of the plurality of channels and played on a second user device different from the first user device. Further, the camera registration module is configured to perform a camera registration operation on the video sequences of each of the plurality of channels to select a plurality of video frames representing the recorded video sequences of each of the plurality of channels based on key video frames from a plurality of registered cameras associated with the recorded video sequences. The feature extraction module is configured to extract features of at least some of the selected plurality of video frames of each of the plurality of channels, and the feature database configured to store the extracted features of each of the plurality of channels. Further, the image classifier is configured to compare the obtained set of features of the image with the stored features of each of the plurality of channels to determine whether there is a match between the obtained set of features of the image and features of a video frame from a specific channel and, when there is a match between the obtained set of features of the image and features of a video frame from a specific channel, to determine a channel number of the specific channel as a recognized channel number. 
     Other aspects of the present disclosure can be understood by those skilled in the art in light of the description, the claims, and the drawings of the present disclosure. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an exemplary environment incorporating certain embodiments of the present invention; 
         FIG. 2  illustrates an exemplary computing system consistent with the disclosed embodiments; 
         FIG. 3  illustrates an exemplary video program recognition system consistent with the disclosed embodiments; 
         FIG. 4  illustrates an exemplary video structure consistent with the disclosed embodiments; 
         FIG. 5  illustrates an exemplary operating process consistent with the disclosed embodiments; 
         FIG. 6  illustrates an example of camera take editing results from a movie; 
         FIG. 7  illustrates exemplary camera take detection and registration consistent with the disclosed embodiments; 
         FIG. 8  illustrates an exemplary flow control operation of an adaptive controller consistent with the disclosed embodiments; 
         FIG. 9  illustrates an exemplary interaction between an adaptive controller and a camera registration module consistent with the disclosed embodiments; 
         FIG. 10  illustrates an exemplary image classification using global features consistent with the disclosed embodiments; 
         FIG. 11  illustrates an exemplary image classification using local features consistent with the disclosed embodiments; and 
         FIG. 12  illustrates an exemplary operating process of a mobile device consistent with the disclosed embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to exemplary embodiments of the invention, which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. 
       FIG. 1  illustrates an exemplary environment  100  incorporating certain embodiments of the present invention. As shown in  FIG. 1 , environment  100  includes a television set (TV)  102 , a mobile device  104 , a notebook computer  106 , a cloud or network  120 , and a server  108 . Certain devices may be omitted and other devices may be added. 
     TV  102  may include any appropriate type of TV, such as plasma TV, LCD TV, projection TV, non-smart TV, smart TV, or mobile TV. TV  102  may also include any devices that with display functionality. Mobile device  104  may include any appropriate type of mobile computing device, such as a mobile phone, a smart phone, a tablet, and a digital personal assistant (PDA). Notebook computer  106  may include any appropriate type of computer, such as laptop computer or other type of personal computer. TV  102 , mobile device  104 , and notebook computer  106  may be considered user devices of multi-screens in certain multi-screen applications. Any number and other types of user devices may be included. 
     Server  108  may include any appropriate type of server computer or computers providing server functionalities. Cloud  120  may include any number of servers connected via a computer network, such as the Internet, to provide services to devices connected to the cloud  120 . Server  108  may be a part of cloud  120 , and server  108  and cloud  120  may be used interchangeably with respect to certain functionalities or services provided by the cloud  120 . Optionally, a traditional client-server architecture may be used, and cloud  120  may simply be referred as a computer network, such as the Internet. 
       FIG. 2  shows a block diagram of an exemplary computing system  200  capable of implementing TV  102 , mobile device  104 , notebook computer  106 , and/or server  108 . As shown in  FIG. 2 , computing system  200  may include a processor  202 , a storage medium  204 , a monitor  206 , a communication module  208 , a database  210 , and peripherals  212 . Certain devices may be omitted and other devices may be included. 
     Processor  202  may include any appropriate processor or processors. Further, processor  202  can include multiple cores for multi-thread or parallel processing. Storage medium  204  may include memory modules, such as ROM, RAM, flash memory modules, and mass storages, such as CD-ROM and hard disk, etc. Storage medium  204  may store computer programs for implementing various processes, when the computer programs are executed by processor  202 . 
     Further, peripherals  212  may include various sensors and other I/O devices, such as keyboard and mouse, and communication module  208  may include certain network interface devices for establishing connections through communication networks. Database  210  may include one or more databases for storing certain data and for performing certain operations on the stored data, such as database searching. Particularly, when computing system  200  implements the mobile device  104 , peripherals  212  may also include a camera for capturing photo images for the mobile device  104 . 
     In operation, the mobile device  104  and/or the cloud  120 /server  108  may perform a video recognition process in a multi-screen application.  FIG. 3  illustrates an exemplary video program recognition system consistent with the disclosed embodiments. 
     As shown in  FIG. 3 , for illustrative purposes, mobile device  104  may include a photo capturer module  302 , a feature extraction module  304 , and an output module  306  for outputting video recognition results. The cloud  120  may include a recorder module  312  for recording video content, a camera registration module  314 , an adaptive controller  316 , a feature extraction module  318 , an image classifier  322 , and a video feature database  320 . Certain components may be omitted and other components may be added. 
     In operation, a video stream may be provided to one or more user devices and cloud  120 . For example, a broadcasting system may be streaming video as a subscription service over the Internet. A video stream may be divided into different video components. As shown in  FIG. 4 , a video stream may be divided into scenes, a scene may be divided into shots, and a shot may be divided into frames, as. The frame can be further divided into objects and features of the video frame may be extracted for further processing. 
     A user device (e.g., a TV, PC, laptop, Pad, or mobile device, etc.) may playback the video. The video in playback may be displayed on a screen of the user device. Further, referring to  FIG. 3 , the mobile device  104  may capture a video frame using the photo capturer. For example, the mobile device  104  may capture a video frame from the screen of the user device using a camera. That is, the camera may take a snapshot of the screen to capture the video frame. Optionally, other methods of capturing the video frame may also be used. 
     Further, the mobile device  104  may extract features of the video frame. That is, the mobile device  104  may extract certain characteristics from the video frame for image classification. The extracted feature set may be provided to cloud  120  or, more specifically, to the image classifier of the cloud  120 . 
     At the same time, the cloud  120  may record a number of channels of the broadcasting streaming video (channel  1 , channel  2 , . . . , channel n) using the recorder module. The recording of the channels of the broadcast may be set for a predetermined time period, such as 1 minutes, 5 minutes, 10 minutes, or other time values. The recording may be used for video program recognition or other purposes. 
     Further, the cloud  120  may perform a camera registration operation on the recorded video frames. Camera registration, as used herein, may refer to registration of different cameras capturing video frames in a video sequence/stream. The concept of camera registration is based on the camera takes in reconstruction of video edits. A typical video sequence is an interleaved format of a number of camera shots, and a camera take is a continuous recorded performance with a given camera setup. By registering each camera from the incoming video frames, the original interleaved format can be separated into a number of sequences with each corresponding to a registered camera that is aligned to the original camera setup. 
     After the camera registration operation by the camera registration module, the near-duplicated images in the same camera take may be ignored, reducing the throughput while maintain an acceptable recognition accuracy. That is, the camera registration module may be configured to register certain different camera takes in the video frames to simplify the video frames and to provide the simplified video frames for next step of processing. 
     The simplified video frames are then processed by feature extraction module in cloud  120 , with an addition of the adaptive controller providing a computational adaptive flow control mechanism to maintain the video processing flow in a stable condition. 
     The cloud  120  may then perform feature extraction of the video frames from the camera registration module using the feature extraction module. The feature extraction module of the cloud  120  may use a same feature extraction algorithm as the feature extraction module of the mobile device  104 . That is, the output representation and/or calculation algorithm may be the same for both feature extraction modules in the cloud  120  and in the mobile device  104 . The detailed implementations for both feature extraction modules in the cloud  120  and in the mobile device  104  may be different, or may be same, depending on specific applications. 
     The extracted features of the video frames from the various channels may then be stored in the feature database. The feature database may include any appropriate commercial or customized database, or any appropriate storage specifically configured for image retrieval. 
     Further, the image classifier compares the feature set submitted from the mobile device  104  with features from the feature database to determine whether the feature set from the mobile device  104  matches any feature set provided by the cloud  120  from a particular channel. If the image classifier determines a match, the particular channel is recognized, and the particular channel number is provided to the mobile device  104 , which may also generate video recognition results by the output module. The channel number is also feedback to the adaptive controller module to improve the performance of the adaptive controller. 
     The adaptive controller may control the video frame processing based on certain predetermined algorithms. For example, the adaptive controller may determine which frame is to be dropped and which frame goes to the next step of processing (i.e., feature extraction). That is, the adaptive controller may control the throughput of the video frame processing such that the accuracy of video recognition and the computation amount may be balanced to reach a desired implementation. For example, the adaptive controller may control the camera registration module to reduce the high throughput of multiple video channels (e.g., 6000 fps) to a lower throughput that matches the computational capacity of the feature extraction module (e.g., 60 fps). 
     More specifically,  FIG. 5  illustrates an exemplary operating process of the cloud  120 /server  108 . As shown in  FIG. 5 , at beginning, the server  108  (e.g., processor  202 ) may cause a recorder module to record a plurality of channels of the broadcasting streaming video (channel  1 , channel  2 , . . . , channel n) using the recorder module (S 502 ). 
     Further, the recorded video sequences from the plurality of channels are provided to a camera registration module for performing a camera registration operation (S 504 ). 
     A camera take is a series of consecutive frames taken by a camera. It can be cut into a sequence of segments and interleaved with other camera takes to form a scene which completes an event or a story in a video sequence.  FIG. 6  shows an example of camera take editing results from a movie. Each picture in the  FIG. 6  is a key frame selected from a shot of the video scene, thus frames (a) to (h) represent consecutive shots, composing a scene. 
     Further, frames (a), (c) and (f) are from the same camera take, frames (b) and (e) are from the same camera take, and frames (d) and (g) are from the same camera take. Thus, the shots from the same camera take can be grouped together and represented by one or more frames. The number of video frames need to be processed can be substantially reduced and the throughput for further processing can also be reduced. 
     To perform the camera registration operation, the server (e.g., the camera registration module) detects and maintains camera takes for each channel. The detected and maintained camera take can form a registered camera pool. That is, each channel follows the same procedure of camera take detection and contributes to the registered cameras pool. Specifically, as shown in  FIG. 7 , for each channel (Channel  1 , Channel  2 , . . . , Channel S), it may take following steps for camera take detection. 
     Frame difference calculation: based on the assumption that two consecutive frames in a video shot should have high similarity in terms of visual content, the frame difference is calculated using a similarity indicator, such as a color histogram (or raw pixel values for saving computational cost), as a measurement of similarity between two frames. Optionally, a plurality of similarity calculation algorithms may be implemented with different computational complexity and computation speeds. 
     Shot detection: if the frame difference is above a preset threshold value, then a new shot is detected. The selection of the threshold, if not selected properly, may cause over segmentation or down segmentation depending on the types of video programs (e.g., action, or drama, etc.). To determine a proper threshold value and further refine the detection results, certain constraints may apply, such as video shot duration, i.e., the detection of the shot has to be within the constraint. 
     Key frame selection: a key frame is selected to represent the visual content of a shot using a predetermined algorithm. For example, the first frame of a shot is selected as the key frame for later processing. Other methods may also be used to select (or generate) the most representative key frame(s). 
     Camera take detection: each detected shot (represented by a key frame) may be matched with the last shot in each detected camera take based on predetermined matching criteria. If the matching criterion is satisfied, then the current shot is added to the end of the matched camera take. Such operation may be based on the assumption that a shot is most related to the one with closest temporal relationship. 
     Initially, within a certain time period, the first shot may be used as a camera take. In addition, a camera retirement strategy may be used. That is, after a certain period time that a camera take has no matched image, the camera is consider “retried” and removed from the list of registered cameras. As shown in  FIG. 7 , camera take detection is performed video frames from individual channels, and after the camera registration operation, channel  1  has registered cameras cam  1  and cam  2 , channel  2  has registered cameras cam  3 , cam  4 , and cam  5 , . . . , and channel S has registered cam k-1 and cam k. 
     Returning to  FIG. 5 , after performing the camera registration operation (S 504 ), a feature extraction operation is performed on the selected video frames representing the video sequences in each of the plurality of channels (S 506 ). Further, an adaptive flow control operation may be performed by an adaptive controller to maintain a desired processing flow of the video frame processing and to interact with the camera registration operation to select frames for further processing (S 508 ).  FIG. 8  illustrates an exemplary flow control operation of the adaptive controller. 
     As shown in  FIG. 8 , the adaptive controller may receive M fps (frame-per-second) input image candidates (i.e., video frames) from the camera registration module and selects n fps winners for further processing (M and n are integers and M&gt;&gt;n). The adaptive controller may also drop the unselected video frames by interacting with the camera registration module. 
     The camera registration module may receive the selection outcome (e.g., n fps) of the adaptive controller and updates its data and/or algorithm. For example, the camera registration module may maintain a list of K live cameras (K is a changing integer) for the S channels, and each channel may have more than one associated live cameras, and M=30S if the video stream runs 30 frame per second). 
     For each camera, there is a last selected frame recorded, so that the current frame can be compared to generate a similarity score. Once a frame is selected by the adaptive controller, the camera registration module may replace the last selected frame with the selected frame for the associated camera.  FIG. 9  illustrates an exemplary interaction between the adaptive controller and the camera registration module. 
     As shown in  FIG. 9 , the camera registration module interacts with the adaptive control module for selecting a reference frame. Using one channel (Channel  1 ) as an example, the adaptive controller receives the channel recognition results (recognized channel number) and calculates the similarity score of a current frame with the last selected frame (LSF) of each camera take. The LSF with a highest score is then replaced by the current frame. 
     In addition, a flow control mechanism may be implemented inside the camera registration module. For example, the camera registration module may include a plurality of implementations of algorithms for frame detection and selection with different precision and computation speed, from a lowest precision but fastest speed to a highest precision but lowest speed. Server  108  may manage buffers for input frames to the camera registration module and provide the buffer fullness to the camera registration module. Based on the buffer fullness, the camera registration module may select a specific implementation with the precision and computation speed matching the video frame flow. 
     For example, if the buffer is close to completely full, i.e., a processing congestion condition, which means the processing speed needs to be accelerated, the camera registration module may dynamically switch to a simpler implementation with lower precision but faster computation speed. On the other hand, if the buffer is close to empty, a processing non-congestion condition, which means the processing speed can be slowed down, the camera registration module may switch to a more complex implementation with higher precision but slower computation speed. The number of the implementations may be configured in advance or in real-time depending on specific applications. 
     Returning to  FIG. 8 , in addition to the interaction with the camera registration module, the adaptive controller may also receive the selected channel number (i.e., the result) as a feedback from the mobile device  104 . 
     Thus, the problem for the adaptive controller to solve may be similar to a Knapsack problem, i.e., selecting n objects from a bag of M objects. Specifically, the nature logic of such selection is based on the followings. 
     For channels that no one cares (i.e., no inquires), the priority for selection can be low. That is, the user preference of channels may be considered as an indication of selection priority. If the user(s) does not make any inquires on a particular channel, it can mean that such channel is less likely to be selected. 
     For an existing camera, if the similarity between the current frame and the last selected frame gets low, then priority of selection can be raised. For a new camera added by the camera registration module, the priority of selection is high. Further, for a channel without any selection for a certain time period, the priority for selection gets high. 
     Provided that I s  denotes the number of inquiries of channel s (s=1, 2, . . . , S) within a short recent period T (e.g., 10 seconds), L s  denotes the number of inquiries of channel s (s=1, 2, . . . , S) within a longer recent period (e.g., 5 minutes), f denotes the current frame in examination, C(f) denotes its associated channel of the current frame in examination, Sim(f) denotes the similarity score between f and last selected frame of C(f) (if the last selected frame does not exist, then Sim(f)=0), t(f) denotes the time period that the channel of f has no new frame selection, then the score for current frame is obtained by: 
               W   ⁡     (   f   )       =     {             1.0   -     Sim   ⁡     (   f   )           β   +     ⅇ       α   ⁢           ⁢     I     C   ⁡     (   f   )           +       (     1   -   α     )     ⁢     L     C   ⁡     (   f   )                           if   ⁢           ⁢     t   ⁡     (   f   )         &lt;   T             1.0       otherwise                 
where α and β are parameters to be trained ahead of time, and the impact of channel inquires is modeled with a power function. By ranking the scores of incoming frames, the adaptive controller selects the frames with highest scores for further processing.
 
     Returning to  FIG. 5 , after the flow control operation and features of selected video frames are extracted, the extracted features are stored into a feature database (S 510 ). 
     The extracted features may be divided into two categories, global features and local features. Global features include those features that describe an entire image. Most shape and texture descriptors fall into this category. On the other hand, local features include descriptors of local image neighborhoods computed at multiple interest points. The feature database may store global features and local features separately for efficient retrieval. 
     Further, an image classification operation may be performed on the extracted features based on the image features provided by the mobile device using an image classifier (S 512 ). The image classifier may compare the provided features with the extracted features to determine whether there is a match. If there is a match, the channel of the extracted features can be determined. 
     In addition, a flow control mechanism may be implemented inside the image classifier module. For example, the image classifier may include two types of classifications, one for high throughput situations, e.g., buffer being close to full, and the other for low throughput situations, e.g., buffer being close to empty, and may dynamically chooses between the two. 
     More particularly, the high-throughput classification may use a low-complexity algorithm using global features, and the low-throughput classification may use a high-complexity algorithm using local features. 
     As shown in  FIG. 10 , the high-throughput classification use extracted global features of the frames from the plurality channels. Global features may produce compact representations of images, where each image corresponds to a point in a high dimensional feature space. The machine learning of the high-throughput classification may use support vector machine (SVM), which supervised learning model with associated learning algorithms that analyze data and recognize patterns, for classification and regression analysis. In other words, given labeled training data, the algorithm outputs an optimal hyper plane which categorizes new examples. 
     Typically, supervised learning has a training stage. In the training stage, after feature extraction, all the frame features from one channel are grouped as a class {C 1 , C 2 , . . . , Cn}, and the SVM multi-classifier is trained. In operation, the global image/photo features uploaded from the mobile device directly go to the trained SVM multi-classifier, which finally determines the channel number. 
     As shown in  FIG. 10 , the low-throughput classification use local features, which are descriptors of local image neighborhoods computed at multiple interest points. Typically, interest points are detected at multiple scales and are expected to be repeatable across different views of an object. For example, certain feature types, such as the Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF), may be used. 
     Descriptors extracted from local affine invariant regions may be quantized into visual words, which are defined by vocabulary tree that is built by hierarchical k-means clustering. A large set of representative descriptor vectors are used in the unsupervised training of the tree. The collections of visual words are used in Term Frequency Inverse Document Frequency (TF-IDF) scoring of the relevance of an image to the query. Every node in the vocabulary tree is associated with an inverted file, such that large databases can be scored efficiently. 
     Returning to  FIG. 5 , after the image classification operation, the determined channel number is provided to the mobile device  104  (S 514 ). The channel number is also feedback to the adaptive controller for flow control operations, and a balance between computation amount and accuracy. 
     Correspondingly,  FIG. 12  illustrates an exemplary operation process  1200  for the mobile device  104 . The mobile device  104  may be connected with server  108 . As shown in  FIG. 12 , at the beginning, the mobile device may capture a photo of a video stream showing on a screen of a user device using a camera (S 1202 ). 
     Further, the mobile device may extract features from the photo image (S 1204 ). The mobile device may extract global features and/or local features of the image. Further, the mobile terminal may send the extracted features to the server (S 1206 ). 
     The mobile device may send both global features and local features to the server (e.g., the image classifier of the server). Alternatively, the mobile terminal may receive an instruction from the server and send global features only, local features only, or global features and local features based on the instruction, such that flow control in the image classifier may be performed. 
     Further, as explained above, the server may perform a video program recognition process to recognize a channel number for the photo image sent by the mobile device, and the mobile device may receive the recognized channel number from the server (S 1208 ). The channel number indicates that the video stream showing on the screen of the user device is from the channel with the channel number. 
     Further, the mobile device may present the recognition result to a user of the mobile device (S 1210 ). For example, the mobile device may show the recognized channel number to the user and may further ask the user whether a particular action needs to be taken. In certain embodiments, the user may request the same video from that channel number to be played back on the mobile device. The mobile device may also present the result to another software program for further processing. Other actions may also be performed. 
     Thus, by using the disclosed systems and methods, a computational adaptive video recognition application can be implemented, enabling dynamically selection of a list of reference images for channel detection. A camera registration mechanism can be included to automatically detect the camera take of the current video frame in each channel, thus the images similar to the reference image with a same camera take can be discard from further processing, reducing the size of list of reference images significantly. The decision of frame dropping is controlled by an adaptive controller that maintains the flow throughput to meet the computation capability of the image recognition process, but also maintains the recognition accuracy. To secure a high accuracy of recognition, a user behavior adaptation model can be applied to adjust the reference image list according to the users currently favored channels. In order to maintain flow from the high-throughput source to the low-throughput destiny, specially-designed flow control mechanism can be used the camera registration, adaptive controller, and feature extraction modules. 
     The disclosed systems and methods may also be used to other multi-screen applications enabling the real-time recognition of a TV program with time-shift feature. Other applications, advantages, alternations, modifications, or equivalents to the disclosed embodiments are obvious to those skilled in the art.