Abstract:
An image processing apparatus detects a representative frame of a moving image. The image processing apparatus includes a holding section configured to hold the moving image which is inputted, a detecting section configured to detect a peak of zooming that occurs in the inputted moving image, and an extracting section configured to extract the representative frame corresponding to the detected peak from a plurality of frames constituting the held moving image.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention relates to an image processing apparatus, an image processing method, and a program, especially relates to an image processing apparatus, an image processing method, and a program suitable for automatically detecting a noteworthy frame among a plurality of frames constituting a moving image and automatically detecting a noteworthy object in a moving image. 
         [0003]    2. Description of the Related Art 
         [0004]    There is a technique called digest reproduction which enables a viewer to grasp an outline of a moving image by watching and listening to a part of the moving image without watching and listening to all of the moving image. In the digest reproduction, scenes that seem important are detected from the entire moving image, and only the scenes that seem important are reproduced sequentially. 
         [0005]    As a method for detecting scenes that seem important from the entire moving image, there are a method which detects a so-called scene change and detects a frame after the scene change as an important scene, and a method which highlights a moving image sequence (frame) detected by a time-series learning apparatus using HMM (hidden Markov model), in other words which detects the moving image sequence as an important scene (for example, refer to Japanese Unexamined Patent Application Publication No. 2008-21225). 
         [0006]    Further, for a moving image in which a TV program of a sports game is recorded, there is a method which detects a frame played in slow motion, a replayed frame, and the like as an important scene. 
       SUMMARY OF THE INVENTION 
       [0007]    However, in the method which detects an important scene on the basis of a scene change, for example, when an important subject is slowly zoomed in on, it is difficult to detect such a scene as an important scene. 
         [0008]    Any of the methods described above is effective when applying to a moving image created by a professional of video shooting and editing, for example, such as a TV program. However, for example, these methods are not necessarily effective for a moving image shot by an ordinary user of a home video camera, and an important scene may be difficult to be detected. 
         [0009]    It is desirable to detect an important scene from a moving image shot by an ordinary user of a home video camera. 
         [0010]    An image processing apparatus according to an embodiment of the present invention includes, in the image processing apparatus which detects a representative frame of a moving image, holding means configured to hold the moving image which is inputted, detecting means configured to detect a peak of zooming that occurs in the inputted moving image, and an extracting means configured to extract the representative frame corresponding to the detected peak from a plurality of frames constituting the held moving image. 
         [0011]    The image processing apparatus according to an embodiment of the present invention can further include calculating means configured to calculate an optical flow of the inputted moving image and calculate a scale parameter indicating a zoom state of each frame on the basis of the calculated optical flow, wherein the detecting means can detect an extreme value of the scale parameter as a peak of the zoom of the inputted moving image. 
         [0012]    The extracting means can extract the representative frame corresponding to the detected peak from a plurality of frames constituting the held moving image, and output the representative frame as a digest image. 
         [0013]    The extracting means can extract the representative frame corresponding to the detected peak and a predetermined number of frames before and after the representative frame from a plurality of frames constituting the held moving image, and output the representative frame and the predetermined number of frames as training image candidates in object recognition. 
         [0014]    An image processing method according to an embodiment of the present invention includes the steps of, in the image processing method which detects a representative frame of a moving image, holding the moving image which is inputted, detecting a peak of zooming that occurs in the inputted moving image, and extracting the representative frame corresponding to the detected peak from a plurality of frames constituting the held moving image. 
         [0015]    A program according to an embodiment of the present invention is a control program of an image processing apparatus for detecting a representative frame of a moving image, and the program causes a computer of an image processing apparatus to execute processing including the steps of holding the moving image which is inputted, detecting a peak of zooming that occurs in the inputted moving image, and extracting the representative frame corresponding to the detected peak from a plurality of frames constituting the held moving image. 
         [0016]    In an embodiment of the present invention, an inputted moving image is held, and a peak of zooming that occurs in the inputted moving image is detected. Further, a representative frame corresponding to the detected peak is extracted from a plurality of frames constituting the held moving image. 
         [0017]    According to an embodiment of the present invention, a scene that seems important can be detected from a moving image. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0018]      FIG. 1  is a view for explaining a principle according to an embodiment of the present invention; 
           [0019]      FIG. 2  is a block diagram illustrating a configuration example of an image processing apparatus to which an embodiment of the present invention is applied; 
           [0020]      FIG. 3  is a view for explaining an optical flow; 
           [0021]      FIG. 4  is a view for explaining a peak of a scale parameter; 
           [0022]      FIG. 5  is a flowchart for explaining digest reproduction image creation processing; 
           [0023]      FIG. 6  is a flowchart for explaining a training image candidate creation processing; and 
           [0024]      FIG. 7  is a block diagram illustrating a configuration example of a general purpose computer. 
       
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0025]    Hereinafter, a preferred embodiment (hereinafter referred to as embodiment) of the present invention will be described in detail with reference to the drawings. The embodiment will be described in the following order. 
         [0026]    1. First Embodiment 
       1. First Embodiment 
     Configuration Example of Image Processing Apparatus 
       [0027]    An image processing apparatus according to an embodiment of the present invention detects a scene that seems important from a moving image. Generally, in many cases of video taking operations, a subject observed by a person taking a video is zoomed in on, and thereafter zoomed out from in a moving image. Therefore, in this image processing apparatus, a frame in which the subject observed by the person taking the video is enlarged is detected as an important scene. 
         [0028]    Specifically, when a subject (dog) is gradually zoomed in on, and thereafter zoomed out from as illustrated in a series of moving images in  FIG. 1 , the frame  4  which is most enlarged is detected as an important scene. 
         [0029]      FIG. 2  is a block diagram illustrating a configuration example of the image processing apparatus according to the embodiment of the present invention. 
         [0030]    This image processing apparatus  10  includes a moving image obtaining section  11 , a holding section  12 , an optical flow calculating section  13 , a peak detecting section  14 , and a frame extracting section  15 . 
         [0031]    The moving image obtaining section  11  obtains a moving image outputted from an external apparatus (for example, video camera, video recorder, and the like, which are not illustrated in the figures) connected to the image processing apparatus  10 , and outputs the moving image to the holding section  12  and the optical flow detecting section  13 . 
         [0032]    The holding section  12  holds the moving image inputted from the moving image obtaining section  11 , and provides a frame corresponding to a request from a next stage of frame extracting section  15  to the frame extracting section  15 . 
         [0033]    The optical flow calculating section  13  calculates an optical flow, calculates a scale parameter s indicating a change of zoom-in and zoom-out in the moving image from the calculated optical flow, and outputs the scale parameter s to the peak detecting section  14 . 
         [0034]    Here, the optical flow corresponds to how a pixel indicating the same point in the subject moves between the frames of the moving image, and specifically corresponds to a motion vector of the point on the subject. 
         [0035]    To calculate the optical flow between the frames (to calculate the motion vector), it is widely known that a calculation formula of Lucas-Kanade optical flow, which is also called the gradient method shown by the formula (1) below, can be applied. 
         [0000]    
       
         
           
             
               E 
               = 
               
                 
                   ∑ 
                   
                     x 
                     ∈ 
                     R 
                   
                 
                  
                 
                     
                 
                  
                 
                   
                     [ 
                     
                       
                         F 
                          
                         
                           ( 
                           
                             x 
                             + 
                             h 
                           
                           ) 
                         
                       
                       - 
                       
                         G 
                          
                         
                           ( 
                           x 
                           ) 
                         
                       
                     
                     ] 
                   
                   2 
                 
               
             
             , 
           
         
       
     
         [0000]    
       
         
           
             
               
                 F 
                  
                 
                   ( 
                   
                     x 
                     + 
                     h 
                   
                   ) 
                 
               
               ≈ 
               
                 
                   F 
                    
                   
                     ( 
                     x 
                     ) 
                   
                 
                 + 
                 
                   h 
                    
                   
                     ∂ 
                     
                       ∂ 
                       x 
                     
                   
                    
                   
                     F 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
     
         [0000]    true when h is sufficiently small 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         
                           o 
                           = 
                             
                            
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 h 
                               
                             
                              
                             E 
                           
                         
                       
                     
                     
                       
                         
                           ≈ 
                             
                            
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 h 
                               
                             
                              
                             
                               
                                 ∑ 
                                 x 
                               
                                
                               
                                 
                                   [ 
                                   
                                     
                                       F 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                     + 
                                     
                                       h 
                                        
                                       
                                         
                                           ∂ 
                                           F 
                                         
                                         
                                           ∂ 
                                           x 
                                         
                                       
                                     
                                     - 
                                     
                                       G 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                   
                                   ] 
                                 
                                 2 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             = 
                               
                              
                             
                               
                                 ∑ 
                                 x 
                               
                                
                               
                                 2 
                                  
                                 
                                   
                                     
                                       ∂ 
                                       F 
                                     
                                     
                                       ∂ 
                                       x 
                                     
                                   
                                   [ 
                                   
                                     
                                       F 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                     + 
                                     
                                       h 
                                        
                                       
                                         
                                           ∂ 
                                           F 
                                         
                                         
                                           ∂ 
                                           x 
                                         
                                       
                                     
                                     - 
                                     
                                       G 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                   
                                   ] 
                                 
                               
                             
                           
                           , 
                         
                       
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       h 
                       ≈ 
                       
                         
                           
                             [ 
                             
                               
                                 ∑ 
                                 x 
                               
                                
                               
                                 
                                   
                                     ( 
                                     
                                       
                                         ∂ 
                                         F 
                                       
                                       
                                         ∂ 
                                         x 
                                       
                                     
                                     ) 
                                   
                                   T 
                                 
                                  
                                 
                                   [ 
                                   
                                     
                                       G 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                     - 
                                     
                                       F 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                   
                                   ] 
                                 
                               
                             
                             ] 
                           
                           [ 
                           
                             
                               ∑ 
                               x 
                             
                              
                             
                               
                                 
                                   ( 
                                   
                                     
                                       ∂ 
                                       F 
                                     
                                     
                                       ∂ 
                                       x 
                                     
                                   
                                   ) 
                                 
                                 T 
                               
                                
                               
                                 ( 
                                 
                                   
                                     ∂ 
                                     F 
                                   
                                   
                                     ∂ 
                                     x 
                                   
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                         
                           - 
                           1 
                         
                       
                     
                     , 
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0036]    The calculation formula of Lucas-Kanade optical flow is a publicly known technique described in “An Iterative Image Registration Technique with an Application to Stereo Vision” Bruce D. Lucas &amp; Takeo Kanade, 7th International Joint Conference on Artificial Intelligence (IJCAI), 1981, pp. 674-679. However, the optical flow can be calculated by using a formula other than the formula (1) described above. 
         [0037]    For example, when calculating the optical flow from the moving image illustrated in  FIG. 1 , the direction of the optical flow is toward the center of the subject (dog) while zooming in, and the direction is away from the center of the subject (dog) to the outside while zooming out. 
         [0038]    To calculate a zoom change (size) from the calculated optical flow, an affine transformation matrix (or projective transformation matrix) is obtained from a pair of points (x, y) and (x′, y′) corresponding to each other between frames by using the optical flow. 
         [0039]    Generally, the affine matrix is represented by the following formula (2). 
         [0000]    
       
         
           
             
               
                 
                   
                     [ 
                     
                       
                         
                           
                             x 
                             ′ 
                           
                         
                       
                       
                         
                           
                             y 
                             ′ 
                           
                         
                       
                     
                     ] 
                   
                   = 
                   
                     
                       
                         [ 
                         
                           
                             
                               a 
                             
                             
                               b 
                             
                           
                           
                             
                               c 
                             
                             
                               d 
                             
                           
                         
                         ] 
                       
                        
                       
                         [ 
                         
                           
                             
                               x 
                             
                           
                           
                             
                               y 
                             
                           
                         
                         ] 
                       
                     
                     + 
                     
                       [ 
                       
                         
                           
                             tx 
                           
                         
                         
                           
                             ty 
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0040]    When there is no rotational or translational component in the pair of points corresponding to each other between frames, and the points are only zoomed in on (or zoomed out from), the zoom change (size) appears as a scale parameter s as shown in the following formula (3). 
         [0000]    
       
         
           
             
               
                 
                   
                     [ 
                     
                       
                         
                           
                             x 
                             ′ 
                           
                         
                       
                       
                         
                           
                             y 
                             ′ 
                           
                         
                       
                     
                     ] 
                   
                   = 
                   
                     
                       [ 
                       
                         
                           
                             s 
                           
                           
                             0 
                           
                         
                         
                           
                             0 
                           
                           
                             s 
                           
                         
                       
                       ] 
                     
                      
                     
                       [ 
                       
                         
                           
                             x 
                           
                         
                         
                           
                             y 
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0041]    This scale parameter s is outputted to the peak detecting section  14 . 
         [0042]    Return to  FIG. 2 . As illustrated in  FIG. 4 , the peak detecting section  14  detects an extreme value (hereinafter also referred to as zooming peak) of the scale parameter s inputted from the optical flow calculating section  14 , and transmits the detection result to the frame extracting section  15 . 
         [0043]    The frame extracting section  15  obtains a frame corresponding to the extreme value of the scale parameter s from the holding section  12  on the basis of the detection result from the peak detecting section  14 , and outputs the frame to the next stage. 
       [Operation Explanation] 
       [0044]    Next, operation of the image processing apparatus  10  will be described. 
         [0045]      FIG. 5  is a flowchart explaining digest reproduction image creation processing corresponding to the inputted moving image by the image processing apparatus  10 . In the digest reproduction image creation processing, a frame that seems important is outputted as a digest reproduction image from the frames constituting the moving image. 
         [0046]    In step S 1 , the moving image obtaining section  11  obtains a moving image outputted from an external apparatus connected to the image processing apparatus  10 , and provides the moving image to the holding section  12  and the optical flow detecting section  13 . The holding section  12  holds the moving image inputted from the moving image obtaining section  11 . 
         [0047]    In step S 2 , the optical flow calculating section  13  calculates an optical flow of the moving image provided from the moving image obtaining section  11 , calculates a scale parameter s from the calculated optical flow, and outputs the scale parameter s to the peak detecting section  14 . The peak detecting section  14  holds the scale parameters s inputted sequentially. 
         [0048]    In step S 3 , the moving image obtaining section  11  determines whether or not the input of the moving image from the external apparatus ends, and until the input of the moving image from the external apparatus ends, the moving image obtaining section  11  returns the process to step S 1  and continues to provide the moving image to the holding section  12  and the optical flow detecting section  13 . 
         [0049]    In step S 3 , when the input of the moving image from the external device is determined to end, the process proceeds to step S 4 . In step S 4 , the peak detecting section  14  detects an extreme value of the scale parameter s inputted from the optical flow calculating section  13 , and transmits the detection result to the frame extracting section  15 . 
         [0050]    In step S 5 , the frame extracting section  15  obtains a frame corresponding to the extreme value of the scale parameter s from the holding section  12  on the basis of the detection result from the peak detecting section  14 , and outputs the frame as the digest reproduction image to the next stage. Then, the digest reproduction image creation processing ends. 
         [0051]    According to the digest reproduction image creation processing as described above, a frame that seems important in which a subject observed by a person taking a video is enlarged can be outputted as the digest reproduction image. 
         [0052]    Although, in the above described digest reproduction image creation processing, the entire moving image is held by the holding section  12 , and the extreme value of the scale parameter s is detected from the entire moving image by the peak detection section  14 , the moving image may be divided by a predetermined time unit to be processed. By doing so, the capacity of the holding section  12  can be reduced and processing of the peak detection section  14  can be lightened. 
         [0053]    The frame that seems important which is extracted by the image processing apparatus  10  can be applied not only to the digest reproduction image, but also to a training image in object recognition. 
         [0054]    Here, object recognition is a technique in which only a specific subject (for example, the face of a person) is detected from a moving image, and in object recognition of the related art, a training image has to be prepared by manually cutting out the specific subject to be recognized from the moving image. 
         [0055]    On the other hand, when using the image processing apparatus  10  in processing for creating a training image for object recognition, a frame that seems important in which a subject observed by a person taking a video is enlarged can be used as the training image. When using not only the frame corresponding to the zooming peak, but also several frames before and after the frame as the training images, it is considered that the object recognition system can be developed to be an object recognition system having high robustness to image enlargement/reduction, parallel movement, rotation, and the like. 
         [0056]    Next,  FIG. 6  is a flowchart explaining processing for creating a training image for object recognition from an inputted moving image (hereinafter referred to as training image creation processing) by the image processing apparatus  10 . In the training image creation processing, a frame that seems important and several frames before and after the frame are outputted as training image candidates from the frames constituting the moving image. 
         [0057]    In step S 11 , the moving image obtaining section  11  obtains a moving image outputted from an external apparatus connected to the image processing apparatus  10 , and provides the moving image to the holding section  12  and the optical flow detecting section  13 . The holding section  12  holds the moving image inputted from the moving image obtaining section  11 . 
         [0058]    In step S 12 , the optical flow calculating section  13  calculates an optical flow of the moving image provided from the moving image obtaining section  11 , calculates a scale parameter s from the calculated optical flow, and outputs the scale parameter s to the peak detecting section  14 . The peak detecting section  14  holds the scale parameters s inputted sequentially. 
         [0059]    In step S 13 , the moving image obtaining section  11  determines whether or not the input of the moving image from the external apparatus ends, and until the input of the moving image from the external apparatus ends, the moving image obtaining section  11  returns the process to step S 11  and continues to provide the moving image to the holding section  12  and the optical flow detecting section  13 . 
         [0060]    In step S 13 , when the input of the moving image from the external device is determined to end, the process proceeds to step S 14 . In step S 14 , the peak detecting section  14  detects an extreme value of the scale parameter s inputted from the optical flow calculating section  13 , and transmits the detection result to the frame extracting section  15 . 
         [0061]    In step S 15 , the frame extracting section  15  obtains a frame corresponding to the extreme value of the scale parameter s and a predetermined number of frames before and after the frame from the holding section  12  on the basis of the detection result from the peak detecting section  14 , and outputs the obtained frames as training image candidates to the next stage. In the object recognition system located in the next stage, all the training image candidates may be used for learning, or the object recognition system may cause a user to select a training image used for learning from the training image candidates. Then, the training image creation processing ends. 
         [0062]    According to the training image creation processing described above, a frame that seems important in which a subject observed by a person taking a video is enlarged, and several frames before and after the frame can be outputted as the training image candidates. 
         [0063]    In the same way as the digest reproduction image creation processing, in the training image creation processing, the moving image may be divided into a predetermined units of time and processed. By doing so, the capacity of the holding section  12  can be reduced and the processing load of the peak detection section  14  can be reduced. 
         [0064]    The series of processing operations described above can be implemented by hardware, and also implemented by software. When the series of processing operations is implemented by software, a computer in which a program constituting the software is installed in dedicated hardware is used, or the program constituting the software is installed in, for example, a general purpose personal computer which can perform various functions by installing various programs from a program recording medium. 
         [0065]      FIG. 7  is a block diagram illustrating a hardware configuration example of a computer which performs the series of processing operations described above by executing a program. 
         [0066]    In this computer  100 , a CPU (Central Processing Unit)  101 , a ROM (Read Only Memory)  102 , and a RAM (Random Access Memory)  103  are connected to one another by a bus  104 . 
         [0067]    An input/output interface  105  is further connected to the bus  104 . An input section  106  including a keyboard, a mouse, a microphone, and the like, an output section  107  including a display, a speaker, and the like, a storage section  108  including a hard disk, a non-volatile memory, and the like, a communication section  109  including a network interface and the like, and a drive  110  for driving a magnetic disk, an optical disk, an optical magnetic disk, or a removable medium  111  such as a semiconductor memory are connected to the input/output interface  105 . 
         [0068]    In a computer having a configuration as described above, the CPU  101  loads a program stored in the storage section  108  into the RAM  103  via the input/output interface  105  and the bus  104 , and executes the program, so that the series of processing operations described above is performed. 
         [0069]    For example, the program executed by the computer (CPU  101 ) is provided by being recorded in a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), and the like), an optical magnetic disk, or a removable medium  111  that is a package medium including a semiconductor memory, or provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting. 
         [0070]    The program can be installed in the storage section  108  via the input/output interface  105  by mounting the removable medium  111  in the drive  110 . Also, the program can be installed in the storage section  108  by receiving the program by the communication section  109  via a wired or wireless transmission medium. In addition, the program can be installed in the ROM  102  or the storage section  108  in advance. 
         [0071]    In this description, the system represents an entire apparatus including a plurality of apparatuses. 
         [0072]    The embodiment of the present invention is not limited to the embodiment described above, and various modifications may be made without departing from the scope of the present invention. 
         [0073]    The present application contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2009-073141 filed in the Japan Patent Office on Mar. 25, 2009, the entire content of which is hereby incorporated by reference. 
         [0074]    It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.