Abstract:
The present invention relates to a system ( 1 ) and method ( 100 ) for resolution enhancement which enables the depth resolution to be enhanced without using gimbal, performs depth description independent from lighting conditions and camera parameters in high resolution.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to a system and method for resolution enhancement which enables depth resolution to be increased in image processing techniques. 
       BACKGROUND OF THE INVENTION 
       [0002]    Structure from motion technique which deals with object motion analysis is frequently used in acquiring depth information. The aforementioned technique is based on performing depth analysis by associating images taken from different angles. However, point association is a cumbersome task in textureless regions and is a time consuming task. 
         [0003]    State of art reconstruction techniques such as structured lightening, stereo vision and laser scanning involve several parts or moving gimbals which complicate the deployment of the system they are used. Demand for time of flight (ToF) camera systems which are compact than aforementioned systems is gradually increasing. Although there has been progress in depth imaging and reconstruction techniques, existing systems can only acquire limited part of the depth within the working volume with limited resolution. However the systems to be used are expected to perform depth descriptions independent from the lighting conditions and camera parameters and produce high resolution output. 
         [0004]    Linked States patent document no U.S. Pat. No. 6,137,566, an application known in the state of the art, discloses as method and apparatus for processing signal for the laser radar receivers. The said apparatus comprises comparators which compares the output of the photo-detector with an adjustable threshold level. 
         [0005]    United States patent document no U.S. Pat. No. 7,511,801, an application known in the state of the art, discloses a system or method for automatic gain control of sensors in time-of-flight systems. In the said method, the overexposed and underexposed pixels are compared, and accordingly the light reaching the pixels is adjusted by changing at least one parameter. 
         [0006]    United States patent document no US2009073419, an application known in the state of the art, discloses a dual resolution, dual range sensor system and a method. It is disclosed in the said document that in an operation mode of the system, a low resolution profile of the region wanted to be scanned is developed and high resolution scanning is performed using this profile. 
         [0007]    The article titled “Depth Map Super Resolution” (Gevrekci et al), the article titled “Superresolution under Photometric Diversity of Images” (Gevrekci et al) and the article titled “High Resolution and High Dynamic Range Image Reconstruction from Differently Exposed Images” (Nakai et al), which are known in the state of the art, disclose about acquiring high resolution images by using low resolution images. 
       SUMMARY OF THE INVENTION 
       [0008]    The objective of the present invention is to realize a system and method for resolution enhancement which enables depth resolution to be increased in image processing techniques. 
         [0009]    A further objective of the present invention is to realize a system and method for resolution enhancement without using a gimbal or a mechanical part 
         [0010]    Another objective of the present invention is to realize a system and method for resolution enhancement wherein the depth information coming from the different parts of the working volume are combined using different camera integration times. 
         [0011]    Yet another objective of the present invention is to realize a system and method for resolution enhancement which performs depth descriptions independent from lighting conditions and camera parameters in high resolution. 
     
    
     
       DETAILED DESCRIPTION OF THE INVENTION 
         [0012]    The system and method for resolution enhancement developed to fulfill the objective of the present invention is illustrated in the accompanying figure wherein: 
           [0013]      FIG. 1  is the schematic view of the resolution enhancement system. 
           [0014]      FIG. 2  is the one possible sequence of integration time change with time index. 
           [0015]      FIG. 3  is the flowchart of the resolution enhancement method. 
           [0016]      FIG. 4  is the flowchart of the sub-steps of 104 th  step of the resolution enhancement method. 
           [0017]      FIG. 5  is the flowchart of the sub-steps of 125 th  step of the resolution enhancement method. 
       
    
    
       [0018]    The parts seen in the figures are each given a reference numeral where the numerals refer to the following: 
         [0019]      1 . Resolution enhancement system 
         [0020]      2 . Imaging unit 
         [0021]      3 . Data transfer unit
         31 . Exposure generation module     32 . Offset veneration module     33 . Data transfer module         
         [0025]      4 . Database 
         [0026]      5 . Motion unit 
         [0027]      6 . Control unit 
         [0028]      7 . Gradient buffer 
         [0029]      8 . Image buffer 
         [0030]    Different exposure times used during image acquisition can also be called as “Integration Time”. 
         [0031]    The inventive resolution enhancement system ( 1 ) comprises:
       at least one imaging unit ( 2 ) which enables to get image,   at least one data transfer unit ( 3 ) which has at least one exposure generation module ( 31 ) enabling different exposure times to be generated for the images to be taken, at least one offset generation module ( 32 ) enabling the data about the offset of the imaging unit ( 2 ) to be generated, and at least one data transfer module ( 33 ) enabling the data to be received and transferred,   at least one database ( 4 ) which enables the images acquired to be stored,   at least one motion unit ( 5 ) which enables the movement of the imaging unit ( 2 ) in tree dimension according to the data acquired by the offset generation module ( 32 ),   at least control unit ( 6 ) which enables the acquired images to be processed,   at least one gradient buffer ( 7 ) wherein the differences between the images are stored,   at least one image buffer ( 8 ) wherein the high resolution depth image is stored.       
 
         [0039]    The inventive resolution enhancement method ( 100 ) comprises the steps of initializing time gradient buffer ( 7 ) and the image buffer ( 101 ),
       acquiring low resolution image from the imaging unit ( 2 ) using data received from the exposure generation module ( 31 ) and offset generation module ( 32 ) ( 102 ),   registering the image received to the database ( 4 ) ( 103 ),   implementing resolution enhancement process ( 104 ),   displaying the final image to the user ( 105 ).       
 
         [0044]    In the preferred embodiment of the invention, the step of implementing resolution enhancement process ( 104 ) comprises the sub-steps of
       determining the sub-pixel motion of the low resolution image received last from the imaging unit ( 2 ) according to the previous low resolution image ( 120 ),   aligning the last low resolution image spatially with the previous low resolution image using the motion quantity data determined in 120 th  step ( 121 ),   aligning the last image on the image buffer ( 8 ) with the previous image using the sub-pixel level motion determined in 120 th  step ( 122 ),   aligning the last image on the gradient buffer ( 7 ) with the previous image by using the motion quantity estimated in 120 th  step ( 123 ),   determining a weight function by using saturated and noisy data in the aligned image acquired in step  123  ( 124 ),   computing the gradient buffer ( 7 ) and finding the difference between images in buffer ( 8 )( 125 ),   averaging differences found in step  125  in certain portions to the image aligned in step  123  ( 126 ),   storing the image acquired in step  126  in gradient buffer ( 7 ) ( 127 ),   combining the image acquired in step  121  and the image in the gradient buffer ( 7 ) in certain content ratio ( 128 ),   suppressing the high frequency elements in the acquired image ( 129 ),   registering the image acquired in step  129  to image buffer ( 8 ) ( 130 ).       
 
         [0056]    In the preferred embodiment of the invention, the step or finding the difference between the images in the image buffer ( 8 ) and computing the gradient buffer ( 7 ) ( 125 ) comprises the sub steps of
       acquiring low resolution image from the imaging unit ( 2 ) ( 150 ),   photometric adjustment of the image in the image huller ( 8 ) and the image acquired in step  150  ( 151 ),   downscaling the image in the imaging buffer ( 8 ) to the sizes of the image acquired in step  150  ( 152 ),   determining the depth difference between the image acquired in step  152  and the image acquired in step  150  ( 153 ),   upscaling the difference determined in step  153  to the sizes of the image in the image buffer ( 8 ) again ( 154 ),   multiplying the image acquired in step  154  with the weight function acquired in step  124  ( 155 ).       
 
         [0063]    Since the inventive resolution enhancement system ( 1 ) has no image information when it is activated, the gradient buffer ( 7 ) and the image buffer ( 8 ) are initialized to value 0 ( 101 ). Then, the motion unit ( 5 ) moves the imaging unit ( 2 ) by using the data generated by the offset generation module ( 32 ), meanwhile the imaging unit ( 2 ) takes low resolution images according to the data generated by the exposure generation module ( 31 ) ( 102 ). These images are recorded to the database ( 4 ) ( 103 ). The acquired images are processed with resolution enhancement process ( 104 ). Then, the final images produced by processing resolution enhancement are displayed to the user ( 105 ). 
         [0064]    In the preferred embodiment of the invention, the step of taking low resolution image from the imaging unit ( 2 ) by using the data received from the exposure generation module ( 31 ) and the offset generation module ( 32 ) (in other words step  102 ) is repeated by using a different exposure and offset information generated by exposure generation module ( 31 ) and the offset generation module ( 32 ), respectively. 
         [0065]    In a preferred embodiment of the invention, in step of implementing resolution enhancement processes (in other words step  104 ) the following processes are implemented. 
         [0066]    The control unit ( 6 ) compares the last low resolution image received from the imaging unit ( 2 ) and the last registered low resolution image in the database ( 4 ), and determines the motion quantity between these images at sub-pixel level ( 120 ). The last low resolution image received and the last low resolution image in the database ( 4 ) are aligned using the estimated motion ( 121 ). The high resolution image stored in buffer ( 8 ) is aligned with the image acquired previously using the offset quantity information determined in step  120  ( 122 ). The image stored in the gradient buffer ( 7 ) is aligned with the image acquired previously using the offset quantity information determined in step  120  ( 123 ). Then, the saturated and the noisy points in the image acquired in step  123  are extracted and a weight function is determined using these contaminated pixels ( 124 ). Then the differences between the images in the gradient buffer ( 7 ) and in the image buffer ( 8 ) are computed ( 125 ). The differences found in step  125  are implemented in certain portions to the image aligned in step  123  ( 126 ). The image acquired by implementing these differences is stored in gradient buffer ( 7 ) ( 127 ). Then, the image stored in the gradient buffer ( 7 ) and the image acquired in step  121  are combined using a step size ( 128 ). And then, the high frequency elements acquired in step  128  are suppressed ( 129 ). Next, the image acquired in step  129  is registered to image buffer ( 8 ) ( 130 ). 
         [0067]    In the preferred embodiment of the invention, the step of finding the difference between the images in the gradient buffer ( 7 ) and in the image buffer ( 8 ) (in other words step  125 ) the following steps are performed: 
         [0068]    First low resolution image is acquired from the imaging unit ( 2 ) ( 150 ). The acquired image is photometrically aligned with the image in the image buffer ( 8 ) ( 151 ). Then the image present in the image buffer ( 8 ) is downscaled to the size of the image acquired in step  150  ( 152 ). After that, the depth difference between the image acquired in step  152  and the image acquired in step  150  is determined ( 153 ). The acquired depth difference information is upscaled to the size of the image in the image buffer ( 8 ) again ( 154 ). Then these differences upscaled to the sizes of the image in the image buffer ( 8 ) are multiplied with the weight function acquired in step  124  ( 155 ). 
         [0069]    In the preferred embodiment of the invention, downscaling process in step  152  is realized by skipping certain amount of pixels located after each pixel taken from the image present in the image buffer ( 8 ). In the said embodiment, the number of the pixels to be skipped corresponds to resolution enhancement factor which is the ratio of the image size in the image buffer ( 8 ) and the size of the depth difference information. 
         [0070]    In the preferred embodiment of the invention, upscaling process in step  154  is realized by inserting zero-bit as much as the ratio of the image in the image buffer ( 8 ) and the sizes of the depth difference information between two pixels in the depth difference information. The said process is called as “upscaling with zero insertion”. 
         [0071]    The inventive resolution enhancement system ( 1 ) comprises at least one imaging unit ( 2 ) which enables image acquisition for depth reconstruction purposes. The exposure time which the imaging unit ( 2 ) will use during acquisition the said images is determined according to the exposure information generated by the exposure generation module ( 32 ) which the data transfer unit ( 3 ) has. The motion unit ( 5 ) that generates the motion of the imaging unit ( 2 ) in three dimensions moves the imaging unit ( 2 ) by using the information generated by the offset generation module ( 32 ). By this means, the imaging unit ( 2 ) acquires images in varying, exposure times and from different views. The acquired images are recorded to the database ( 4 ). The high resolution image acquired by processing the images according to the resolution enhancement method ( 100 ) steps is displayed to the user via a screen (not shown in the figures). 
         [0072]    The control unit ( 6 ) transfers the images acquired by the imaging unit ( 2 ) to the database ( 4 ) and these images are stored in the database ( 4 ). 
         [0073]    In the preferred embodiment of the invention, in determining the motion quantity of the low resolution image last received from the imaging unit ( 2 ) compared to the previous low resolution image (in other words step  120 ), the image is subjected to a contrast stretching process according to the pixel intensity set and then a corner (feature) detection algorithm such as Harris Corner Detector algorithm is applied to the said image in order to produce a three-dimensional corner response map. In this embodiment, the correlation matrix at a spatial located is computed as follow: 
         [0000]    
       
         
           
             
               A 
                
               
                 ( 
                 
                   x 
                   , 
                   
                     y 
                     ; 
                     
                       I 
                       c 
                     
                   
                 
                 ) 
               
             
             = 
             
               ( 
               
                 
                   
                     
                       
                         ∑ 
                         
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                           ∈ 
                           N 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 x 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                         2 
                       
                     
                   
                   
                     
                       
                         ∑ 
                         
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                           ∈ 
                           N 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 x 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                               [ 
                               
                                 m 
                                 , 
                                 n 
                               
                               ) 
                             
                           
                           ] 
                         
                          
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 y 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                 
                 
                   
                     
                       
                         ∑ 
                         
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                           ∈ 
                           N 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 x 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                          
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 y 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                   
                     
                       
                         ∑ 
                         
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                           ∈ 
                           N 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           [ 
                           
                             
                               ∂ 
                               
                                 ∂ 
                                 y 
                               
                             
                              
                             
                               
                                 l 
                                 c 
                               
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                         2 
                       
                     
                   
                 
               
               ) 
             
           
         
       
     
       Here, 
       [0074]    
       
         
           
             
               
                 ∂ 
                 
                   ∂ 
                   x 
                 
               
                
               
                 ( 
                 · 
                 ) 
               
             
             , 
             
               
                 ∂ 
                 
                   ∂ 
                   y 
                 
               
                
               
                 ( 
                 · 
                 ) 
               
             
           
         
       
     
         [0000]    are the gradient operators in vertical and horizontal direction, respectively. And then the cornerness response is computed according to the formula R(x,y;c)=det A(x,y;l c )−k(trace/l(x,y;l c )) 2 . In here, “k” is the cornerness response coefficient, and it is generally selected approximately 0.04. In here, (x,y) are the horizontal and vertical coordinates of the point which is computed for attribute, respectively. And (m,n) is the horizontal and vertical indices within the window neighborhood (N) taken around the feature point. In this embodiment, contrast stretching process is determined according to the following formula: 
         [0000]    
       
         
           
             
               
                 l 
                 c 
               
                
               
                 ( 
                 
                   m 
                   , 
                   n 
                 
                 ) 
               
             
             = 
             
               
                 
                   f 
                   c 
                 
                  
                 
                   ( 
                   
                     l 
                      
                     
                       ( 
                       
                         m 
                         , 
                         n 
                       
                       ) 
                     
                   
                   ) 
                 
               
               = 
               
                 1 
                 
                   1 
                   + 
                   
                      
                     
                       - 
                       
                         γ 
                          
                         
                           ( 
                           
                             
                               I 
                                
                               
                                 ( 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                             
                             - 
                             c 
                           
                           ) 
                         
                       
                     
                   
                 
               
             
           
         
       
     
         [0075]    l c  is the image to which the contrast stretching is applied, l is the raw image, and 
         [0076]    f c  symbolizes the contrast stretching function, whose slope is determined by 
         [0077]    γ and center is denoted as c, which lies within the range of [0,1]. 
         [0078]    In the preferred embodiment of the invention, the similarity of the corners is determined with normalized cross correlation formula. 
         [0000]    
       
         
           
             
               ρ 
               
                 A 
                 , 
                 B 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     i 
                     = 
                     1 
                   
                   n 
                 
                  
                 
                     
                 
                  
                 
                   
                     ( 
                     
                       
                         A 
                         i 
                       
                       - 
                       
                         A 
                         _ 
                       
                     
                     ) 
                   
                    
                   
                     ( 
                     
                       
                         B 
                         i 
                       
                       - 
                       
                         B 
                         _ 
                       
                     
                     ) 
                   
                 
               
               
                 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       n 
                     
                      
                     
                         
                     
                      
                     
                       
                         ( 
                         
                           
                             A 
                             i 
                           
                           - 
                           
                             A 
                             _ 
                           
                         
                         ) 
                       
                       2 
                     
                   
                 
                  
                 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       n 
                     
                      
                     
                         
                     
                      
                     
                       
                         ( 
                         
                           
                             B 
                             i 
                           
                           - 
                           
                             B 
                             _ 
                           
                         
                         ) 
                       
                       2 
                     
                   
                 
               
             
           
         
       
     
         [0079]    A and B represent the blocks to be taken around the feature in the images to be aligned. n corresponds to the number of the pixels inside blocks. 
         [0080]    In the preferred embodiment of the invention, at least two images are taken from the imaging unit ( 2 ) and the said images are stored in the database ( 4 ). The control unit ( 6 ) uses the former of the said images as geometric reference and computes the residue by aligning the latter with the said reference. In this embodiment the mean depth of the images is computed according to the following formula in step  153 . 
         [0000]    
       
         
           
             
               
                 FP 
                 i 
               
                
               
                 ( 
                 
                   
                     l 
                     1 
                   
                   , 
                   
                     l 
                     2 
                   
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 
                   
                     n 
                     1 
                   
                   , 
                   
                     n 
                     2 
                   
                 
               
                
               
                   
               
                
               
                 
                   
                     h 
                     i 
                   
                    
                   
                     ( 
                     
                       
                         l 
                         1 
                       
                       , 
                       
                         
                           l 
                           2 
                         
                         ; 
                         
                           n 
                           1 
                         
                       
                       , 
                       
                         n 
                         2 
                       
                     
                     ) 
                   
                 
                  
                 
                   
                     
                       q 
                       ^ 
                     
                     N 
                   
                    
                   
                     ( 
                     
                       
                         n 
                         1 
                       
                       , 
                       
                         n 
                         2 
                       
                     
                     ) 
                   
                 
               
             
           
         
       
     
         [0081]    Here, i is image index, l 1 ,l 2  are the spatial index of the low resolution image, n 1 ,n 2  are the spatial indexes in high resolution image, {circumflex over (q)} N  is the normalized irradiance value. Then a residual computation is performed according to the following formula by using mean depth information and raw depth D i . 
         [0000]        r   i (l 1   l   2 )= D   i (l 1 ,l 2 )− FP   i ( l   1   ,l   2 )
 
         [0082]    In the preferred embodiment of the invention, gradient buffer is accumulated using amnesic (in other words in 126 th  step) according to the formula 
         [0000]    
       
         
           
             
               
                 I 
                 _ 
               
               
                 m 
                 + 
                 1 
               
             
             = 
             
               
                 
                   ( 
                   
                     
                       m 
                       - 
                       l 
                     
                     
                       m 
                       + 
                       1 
                     
                   
                   ) 
                 
                  
                 
                   
                     I 
                     _ 
                   
                   m 
                 
               
               + 
               
                 
                   ( 
                   
                     
                       1 
                       + 
                       l 
                     
                     
                       m 
                       + 
                       1 
                     
                   
                   ) 
                 
                  
                 
                   
                     I 
                     
                       m 
                       + 
                       1 
                     
                   
                   . 
                 
               
             
           
         
       
     
         [0000]    In this formula,  l   m+1  is the resultant gradient value stored in gradient buffer, and l is the value computed according to the l−λm value, m is the time index, and λ is the amnesic factor which determines the forgetting factor. Amnesic mean adapts to the temporal change in gradient and ensures that gradients do not dominate the image mean. 
         [0083]    In the preferred embodiment of the present invention, in step of averaging the differences found in step  125  in certain ratios to the image aligned in step  123  (in other words in the step  126 ), the stated ratios are acquired using step size computed with the methods of “steepest descent” or “conjugate gradient” known in the state of the art. In this embodiment, the image buffer is updated according to x m+1 =x m +αĪ m+1  formula (in other words in the step  128 ). In the formula, a is the step size, x m+1  is the super-resolution image at time (m+1), x m  is the super-resolution image at time m. 
         [0084]    In the preferred embodiment of the present invention, in step off pressing the high frequency elements in the acquired image (in other words in step  129 ), the suppressing process is realized according to the following formula. 
         [0000]    
       
         
           
             
               
                 
                   q 
                   ^ 
                 
                 N 
               
                
               
                 ( 
                 
                   
                     n 
                     1 
                   
                   , 
                   
                     n 
                     2 
                   
                 
                 ) 
               
             
             = 
             
               
                 
                   f 
                   
                     - 
                     1 
                   
                 
                  
                 
                   ( 
                   
                     
                       q 
                       ^ 
                     
                      
                     
                       ( 
                       
                         
                           n 
                           1 
                         
                         , 
                         
                           n 
                           2 
                         
                       
                       ) 
                     
                   
                   ) 
                 
               
               
                 α 
                 i 
               
             
           
         
       
     
         [0085]    In the formula, q is the irradiance value, n1 and n2 are planar coordinates, α is the camera integration time and f is optoelectric circuit functions that also incorporates the weighting function computed in step  124  to suppress noisy and saturated sensor values. 
         [0086]    It is possible to develop a wide variety of embodiments of the inventive system ( 1 ) and method ( 100 ) for resolution enhancement. The invention cannot be limited to the examples described herein and it is essentially according, to the claims.