Abstract:
This invention provides a device and a method of multi-view image calibration which can easily acquire calibration parameters (intrinsic/extrinsic parameter) of multi-image input device in wide space without separate calibration tool.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit of Korean Patent Application No. 10-2015-0019206 filed with the Korean Intellectual Property Office on Feb. 9, 2015, the entire disclosure of which is incorporated herein by reference. 
       BACKGROUND 
       [0002]    1. Technical Field 
         [0003]    The present invention relates to a device and a method for multi-view image calibration, and more particularly, a method for multi-view image calibration for producing calibration parameters without calibration tool. 
         [0004]    2. Description of the Related Art 
         [0005]    According to a market report, the market size of the three-dimensional animation is expected to get bigger each year, and motion capture technology and model restoration technology for 3D animation have been increasingly developed. For motion capture for 3D animation, a plurality of cameras are used to take the multi-view image. The plurality of cameras cause the problem that vertical error and irregular time interval may occur. 
         [0006]    The related arts are disclosed in Published Korean Patent Application 10-2002-0014456 (published on Feb. 25, 2002) “A METHOD AND SYSTEM OF REVISION FOR 3-DIMENSIONAL IMAGE.” 
       SUMMARY OF THE INVENTION 
       [0007]    This invention thus provides with a device and a method for multi-view image calibration which can produce calibration parameters of multi-view image without special calibration tool. 
         [0008]    The objects of this invention are not limited to those mention above, another objects not mentioned can be apparently understood by the following description. 
         [0009]    An aspect of this invention provides a multi-view image calibration device including: an image input synchronizing means for synchronizing a multi-image input device which takes pictures of an object at multiple view points; an image capturing means for inputting multi-view images in synchronization from the multi-image input device; a characteristic point detecting means for detecting characteristic points of the multi-view images; a corresponding point set generating means for selecting corresponding point(s) among the detected characteristic points and for generating a corresponding point set; and a calibration parameter generating means for producing calibration parameters of the multi-image input device from the generated corresponding point set. 
         [0010]    Another aspect of this invention provides a multi-view image calibration method including the steps of: inputting multi-view images in synchronization from a multi-image input device; detecting characteristic points with respect to the multi-view images; selecting corresponding points among the detected characteristic points so as to generate a corresponding point set; and generating calibration parameters of the multi-image input device from the generated corresponding point set. 
         [0011]    The present invention has the advantage of reducing the consumption of time and material resources, because the constraint of space is comparatively less than that of the calibration using a calibration tool, and there is no need to make a separate calibration tool. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0012]      FIG. 1  and  FIG. 2  are drawings for illustrating the multi-view image calibration system according to one embodiment of this invention; 
           [0013]      FIG. 3  and  FIG. 4  are drawings for illustrating the multi-view image calibration device according to one embodiment of this invention; 
           [0014]      FIG. 5  through  FIG. 10  are drawings for illustrating the multi-view image calibration method according to one embodiment of this invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0015]    Hereinafter the embodiments of this invention will be in detail explained with reference to the attached drawings, so that those skilled in this art should implement this invention. The present invention, however, can be implemented in various types and is not limited to the following embodiment. In addition, the expression of “comprising” means that it could be interpreted as “further includes” but it could not be interpreted as “exclude additional component”, unless otherwise mentioned. 
         [0016]      FIG. 1  and  FIG. 2  are drawings for illustrating the multi-view image calibration system according to one embodiment of this invention. 
         [0017]    Referring to  FIG. 1 , the multi-view image calibration system comprises a multi-image input device  100  and a multi-view image calibration device  200 . 
         [0018]    The multi-image input device  100  is a device of image input device for taking pictures of object at multiple viewpoints. The multi-image input device  100  takes pictures of moving object in synchronization and produces multi-viewpoints images. The multi-image input device  100  can be, for example, a camera device. 
         [0019]    Referring to  FIG. 2 , the multi-image input device  100  can comprises at least three camera devices. In the multi-image input device  100 , the more of the images captured at the same time causes the higher accuracy of calibration but the slower processing speed of input image. Also, in the multi-image input device  100 , the less of images captured at the same time causes the lower calibration accuracy but the higher processing speed is. Thus, in the multi-image input device, the number of camera device can be controlled in accordance with the calibration accuracy and the image processing speed of the multi-view image calibration device  200 . 
         [0020]    The multi-view image calibration device  200  processes the synchronized image of multi-viewpoints inputted through the multi-image input device  100  and produces calibration parameters of the multi-image input device  100 . The multi-view image calibration device  200  may be a computer apparatus physically including a processor, a memory, an input/output device, and a communication interface. The multi-view image calibration device  200  may be an image processing device for executing a multi-view image calibration program. 
         [0021]    The multi-view image calibration device  200  controls the multi-image input device  100  such that the multi-image input device take synchronized pictures of object, and detects the characteristic point(s) in the inputted multi-view image(s). Then, the multi-view image calibration device  200  selects corresponding point(s) in the multi-view image(s), tracks the corresponding point(s) of the every image inputted and processed, and generates a corresponding point set. Then the multi-view image calibration device  200  generates calibration parameter(s) of the multi-image input device  100  using the generated corresponding point set. 
         [0022]      FIG. 3  and  FIG. 4  are drawings for illustrating the multi-view image calibration device according to one embodiment of this invention. 
         [0023]    Referring to  FIG. 3 , the multi-view image calibration device  200  includes an image input synchronizing unit  210 , an image acquisition unit  220 , a characteristic point detection unit  230 , a corresponding point set generating unit  240 , and a calibration parameter generating unit  250 . 
         [0024]    The image input synchronizing unit  210  synchronizes multi-image input device(s)  100  which take pictures of object(s) so as to synchronize the multi-view images inputted. 
         [0025]    The image acquisition unit  220  inputs the synchronized multi-view image generated from, the multi-image input device  100 . 
         [0026]    The characteristic point detection unit  230  detects characteristic point(s) among the multi-view image inputted. Here, the characteristic point(s) means the point(s) that can be distinguished from ambient background and can be easily identified because the point is little affected by different viewpoints and lights, to be selected as reference point(s) for matching every multi-view image. The characteristic point detection unit  230  performs color-conversion of every multi-view image inputted for selecting the characteristic point(s) and performs the corner-detection method. Here, the color-conversion can be HSV (Hue-Saturation-Value) color conversion, and the corner detection method can use the corner detection method based on brightness and contour. Referring to  FIG. 4 , the characteristic point detection unit  230  can detect a plurality of characteristic points  410  to  430  in the multi-view image through the color-conversion and the corner detection method. 
         [0027]    The corresponding point set generating unit  240  selects the characteristic point(s) little affected by different viewpoints among the characteristic point(s) as corresponding point(s), and tracks the selected corresponding point(s) of every multi-view image inputted so as to generate corresponding point set. The corresponding point set generating unit  240  can select at least three corresponding points per one viewpoint for the sake of accuracy. Also, the corresponding point set generating unit  240  can eliminate the erroneously detected corresponding point(s) among the selected corresponding point(s) and can add the missed corresponding point(s) by interpolation. For this, the corresponding point set generating unit  240  can use, for example, RANSAC (RANdom Sample Consensus) method or the like. 
         [0028]    The calibration parameter generating unit  250  performs matrix-factorization from the selected corresponding point set to produce calibration parameter(s) of the multi-image input device  100 . 
         [0029]      FIG. 5  to  FIG. 10  are drawings for illustrating, the multi-view image calibration method according to one embodiment of this invention. 
         [0030]    Referring to  FIG. 5 , the multi-view image calibration device  200  acquires multi-view images from the synchronized multi-image input device  100  in step S 510 . 
         [0031]    The multi-view image calibration device  200  detects the characteristic points of the every multi-view image inputted, in step S 520 . The multi-view image calibration device  200 , as mentioned above, can detect the point(s) which can be distinguished from ambient background and can be easily identified due to less effect of viewpoint and light, through the HSV (Hue-Saturation-Value) color-conversion and the bright and contour based corner detection method. 
         [0032]    The multi-view image calibration device  200  selects corresponding points among the detected characteristic point(s) in step S 530 , and generates a corresponding point set by tracking the selected corresponding point(s) of every multi-view image inputted. This will be in more detail explained with reference to  FIG. 6  and  FIG. 7 . 
         [0033]    The multi-view image calibration device  200  generates calibration parameter(s) of the multi-image input device  100  through the corresponding point sets of consecutive frames in step S 540 . 
         [0034]    Hereinafter, the method of precisely selecting corresponding point(s) in every multi-view image inputted with reference to  FIG. 6 . 
         [0035]    The multi-view image calibration device  200  detects characteristic point(s) of the every multi-view image acquired, in step S 610 . 
         [0036]    The multi-view image calibration device  200  determines whether the image is the first frame image in which the detected characteristic point(s) would be selected as corresponding point(s), in step S 620 . 
         [0037]    In step S 630 , The multi-view image calibration device  200  selects corresponding point(s) among the characteristic point(s) in case the result of step S 620  is the first frame image in which the detected characteristic point(s) would be selected as corresponding point(s). Here, the corresponding point can be selected so as to be on the same three-dimensional space with respect to at least three view-points, and at least three corresponding points can be selected per one viewpoint for the sake of accuracy. Referring to  FIG. 4 , at least three corresponding points can be selected among the detected characteristic points. 
         [0038]    The multi-view image calibration device  200  determines whether corresponding points be selected for the every multi-view image inputted, in step S 640 . 
         [0039]    In step S 650 , the multi-view image calibration device  200  tracks the corresponding points among the characteristic point(s) of another multi-view image, based on the previously selected corresponding point(s), in case that the determination result is that corresponding point is not selected for the every multi-view image inputted. 
         [0040]    The multi-view image calibration device  200  determines whether or not the tracked corresponding point(s) is erroneously detected or there is any missing, by matching the synchronized multi-view images for every multi-view image, in step S 660 . 
         [0041]    The multi-view image calibration device  200  eliminates the erroneously detected corresponding point(s) and adds the missed corresponding point(s), in case the determination result is that the tracked corresponding point(s) is erroneously detected or missed in every multi-view image, in step S 670 . The multi-view image calibration device  200  can eliminate the noise included in the corresponding point(s) using Kalman Filter which is a recursive filter to track the state of a linear dynamic system including the noise. The multi-view image calibration device  200  can add the missed corresponding point using interpolation method. 
         [0042]    Hereinafter it will be explained the method for generating calibration parameters using the selected corresponding points in every multi-view image inputted, with reference to  FIG. 7 . 
         [0043]    The multi-view image calibration device  200  selects the corresponding points of the acquired every multi-view image, in step S 710 . 
         [0044]    The multi-view image calibration device  200  generates a corresponding point set using the selected corresponding points in multi-view image inputted in time sequence, in step S 720 . Referring to  FIG. 8  and  FIG. 9 , the corresponding point set can be set, for example P 1   t   1 , . . . , P 1   t   7  can be set as a first corresponding point set and, N 1   t   1 , . . . , N 1   t   7  can be set as a second corresponding point set. 
         [0045]    The multi-view image calibration device  200  generates a polynomial curve having least separation by interpolation method using the corresponding points of the generated corresponding point set as input values in step S 730 . Referring to  FIG. 8  and  FIG. 9 , a first polynomial curve  810  and a second polynomial curve  910  having least separation for the first corresponding point set and the second corresponding point set respectively can be generated. Here, the following equation 1 can be used for seeking the polynomial curve having least separation. 
         [0000]    
       
         
           
             
                 
             
              
             
               [ 
               
                 Equation 
                  
                 
                     
                 
                  
                 1 
               
               ] 
             
           
         
       
       
         
           
             min 
              
             
               ( 
               
                 
                   
                     
                       
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             M 
                           
                            
                           
                               
                           
                            
                           
                              
                             
                               
                                 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                   + 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                   + 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                 
                               
                               - 
                               
                                 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                   + 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                   + 
                                   
                                     P 
                                      
                                     
                                       ? 
                                     
                                   
                                 
                               
                             
                              
                           
                         
                         M 
                       
                       , 
                     
                   
                 
                 
                   
                     
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           M 
                         
                          
                         
                             
                         
                          
                         
                            
                           
                             
                               
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                                 + 
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                                 + 
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                               
                             
                             - 
                             
                               
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                                 + 
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                                 + 
                                 
                                   N 
                                    
                                   
                                     ? 
                                   
                                 
                               
                             
                           
                            
                         
                       
                       M 
                     
                   
                 
               
               ) 
             
           
         
       
       
         
           
             
               
                 ? 
               
                
               
                 indicates text missing or illegible when filed 
               
             
              
             
                 
             
           
         
       
     
         [0046]    Then the multi-view image calibration device  200  modifies the set of corresponding points included in the generated polynomial curve in step S 740 . Referring to  FIG. 8  and  FIG. 9 , the modified corresponding point set included in the polynomial curve can be set, for example, P 21   t   1 , . . . , P 2   t   7  as the first corresponding point set and N 2   t   1 , . . . , N 2   t   7  as the second corresponding point set. 
         [0047]    The multi-view image calibration device  200  generates calibration parameters of the multi-image input device  100  using the compensated corresponding point set in step S 750 . 
         [0048]    Hereinafter, referring to  FIG. 10 , it will be explained the generating method of calibration parameters of the multi-image input device  100  in the compensated corresponding point set. 
         [0049]    Referring to  FIG. 10 , in case there are m (natural number) multi-image input devices and n (natural number) corresponding points and X p ={x p ,y p ,z p ,1} T , p=1, . . . , n, the two-dimensional image point u p   i  can be written as the following Equation 2 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         λ 
                         p 
                         i 
                       
                        
                       
                         [ 
                         
                           
                             
                               
                                 x 
                                 p 
                                 i 
                               
                             
                           
                           
                             
                               
                                 y 
                                 p 
                                 i 
                               
                             
                           
                           
                             
                               1 
                             
                           
                         
                         ] 
                       
                     
                     = 
                     
                       
                         
                           λ 
                           p 
                           i 
                         
                          
                         
                           u 
                           p 
                           i 
                         
                       
                       = 
                       
                         
                           P 
                           i 
                         
                          
                         
                           X 
                           p 
                         
                       
                     
                   
                   , 
                   
                     
                       λ 
                       p 
                       i 
                     
                     ∈ 
                     
                       R 
                       + 
                     
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                      
                     
                         
                     
                      
                     2 
                   
                   ] 
                 
               
             
           
         
       
     
         [0050]    In Equation 2, P i  is a Projection Matrix and each of multi-image input devices  100  includes extrinsic parameter(s) and intrinsic parameter(s). 
         [0051]    Here, the extrinsic parameter is a parameter related to geometrical relationship with the external space of the multi-image input device such as the installation height and the direction (pan, tilt) etc. of the multi-image input device, which may include six information. The intrinsic parameter is an internal parameter such as focus length and center point etc. of the multi-image input device, which may include five information. Using the corresponding points u p   i  measured in previous step, the scale value λ p   i  is produced to be used for generating the projection matrix value P i . 
         [0052]    The scale value λ p   i , is set in two stereo images i and c, λ p   c =1 is set for every p corresponding to the point u p   c  in each of multi-image input device c, and an Unique Fundamental Matrix is produced for i≠ic≠c which can be used for producing the Fundamental Matrix F ic  as the following and can be used for producing an epipole e ic  which is the position of the multi-image input device shown in image, and depth value λ p   i , 
         [0000]    
       
         
           
             
               
                 
                   
                     λ 
                     p 
                     i 
                   
                   = 
                   
                     
                       
                         
                           ( 
                           
                             
                               e 
                               ic 
                             
                             × 
                             
                               u 
                               p 
                               i 
                             
                           
                           ) 
                         
                         · 
                         
                           ( 
                           
                             
                               F 
                               ic 
                             
                              
                             
                               u 
                               p 
                               c 
                             
                           
                           ) 
                         
                       
                       
                         
                            
                           
                             
                               e 
                               ic 
                             
                             × 
                             
                               u 
                               p 
                               i 
                             
                           
                            
                         
                         2 
                       
                     
                      
                     
                       λ 
                       p 
                       c 
                     
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                      
                     
                         
                     
                      
                     3 
                   
                   ] 
                 
               
             
           
         
       
     
         [0053]    Ws is obtained using the depth value of Equation 3 and P i  Projection Matrix may be produced using the Equation 4. 
         [0000]    
       
         
           
             
               
                 
                   
                     W 
                     s 
                   
                   = 
                   
                     
                       [ 
                       
                         
                           
                             
                               
                                 λ 
                                 1 
                                 1 
                               
                                
                               
                                 [ 
                                 
                                   
                                     
                                       
                                         x 
                                         1 
                                         1 
                                       
                                     
                                   
                                   
                                     
                                       
                                         y 
                                         1 
                                         1 
                                       
                                     
                                   
                                   
                                     
                                       1 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                           
                             … 
                           
                           
                             
                               
                                 λ 
                                 n 
                                 1 
                               
                                
                               
                                 [ 
                                 
                                   
                                     
                                       
                                         x 
                                         n 
                                         1 
                                       
                                     
                                   
                                   
                                     
                                       
                                         y 
                                         n 
                                         1 
                                       
                                     
                                   
                                   
                                     
                                       1 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                         
                           
                             ⋮ 
                           
                           
                             ⋱ 
                           
                           
                             ⋮ 
                           
                         
                         
                           
                             
                               
                                 λ 
                                 1 
                                 m 
                               
                                
                               
                                 [ 
                                 
                                   
                                     
                                       
                                         x 
                                         1 
                                         m 
                                       
                                     
                                   
                                   
                                     
                                       
                                         y 
                                         1 
                                         m 
                                       
                                     
                                   
                                   
                                     
                                       1 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                           
                             … 
                           
                           
                             
                               
                                 λ 
                                 n 
                                 m 
                               
                                
                               
                                 [ 
                                 
                                   
                                     
                                       
                                         x 
                                         n 
                                         m 
                                       
                                     
                                   
                                   
                                     
                                       
                                         y 
                                         n 
                                         m 
                                       
                                     
                                   
                                   
                                     
                                       1 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                       
                       ] 
                     
                     = 
                     
                       
                         
                           
                             [ 
                             
                               
                                 
                                   
                                     p 
                                     1 
                                   
                                 
                               
                               
                                 
                                   ⋮ 
                                 
                               
                               
                                 
                                   
                                     p 
                                     m 
                                   
                                 
                               
                             
                             ] 
                           
                           
                             3 
                              
                             
                                 
                             
                              
                             m 
                             × 
                             4 
                           
                         
                          
                         
                           [ 
                           
                             
                               X 
                               1 
                             
                              
                             
                                 
                             
                              
                             … 
                              
                             
                                 
                             
                              
                             
                               X 
                               n 
                             
                           
                           ] 
                         
                       
                       
                         4 
                         × 
                         n 
                       
                     
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                      
                     
                         
                     
                      
                     4 
                   
                   ] 
                 
               
             
           
         
       
     
         [0054]    Here, W s =PX, P[p1 . . . p m ] T , X=[X 1  . . . X n ] 
         [0055]    The multi-view image calibration method generates P i  projection matrix of the multi-image input device so as to produce calibration parameters of the multi-image input device. 
         [0056]    The multi-view image calibration method according to the various embodiments of this invention may be implemented in a program instruction that can be carried out via a variety of computer means such as a variety of servers. 
         [0057]    Also, the program for executing the multi-view image calibration method according to the present invention may be installed in the computer means and may be recorded in computer readable medium. The computer readable medium includes program command, data file, data structure etc. alone or in combination. 
         [0058]    The program command recorded in the computer readable medium may be specially designed and made for this invention or may be known to those skilled in the computer software field so as to be available. 
         [0059]    The examples of computer readable medium includes a magnetic media such as a hard-disk, a floppy-disk and a magnetic tape, an optical media such as CD-ROM and DVD, a magneto-optical media such as floptical disk, a hardware apparatus specially implemented for storing and performing the program command such as ROM, RAN, flash memory etc. Up to now the embodiments of this invention have been described. However, the description of the present invention described above are intended to be illustrative, those skilled in the art pertain to this invention can understand that this invention can be easily modified in other specific embodiments without changing the technical spirit or essential features of the invention. Therefore, the exemplary embodiments disclosed in the present invention and the accompanying drawings do not limit but describe the spirit of the present invention, and the scope of the present invention is not limited by the exemplary embodiments and accompanying drawings.