Patent Publication Number: US-2016224866-A1

Title: Imaging device and phase difference detection method

Description:
CROSS REFERENCE TO RELATED APPLICATION 
     This application is a continuation of International Patent Application No. PCT/JP2014/070303, having an international filing date of Aug. 1, 2014, which designated the United States, the entirety of which is incorporated herein by reference. Japanese Patent Application No. 2013-220023 filed on Oct. 23, 2013 is also incorporated herein by reference in its entirety. 
    
    
     BACKGROUND 
     The present invention relates to an imaging device, a phase difference detection method, and the like. 
     A method that compares two waveforms that are shifted in position (i.e., have a phase difference) to detect the phase difference is an important technique that is indispensable for a wide range of fields such as a parallax level (amount) detection process used for a stereo image 3D measurement process, and a phase difference detection process necessary for a control process (PLL control process) that adjusts the phase of an electrical signal to that of a reference signal. 
     Normally, the matching position is determined while shifting two similar comparison target waveforms, and the difference (shift amount) between the original position and the matching position is detected as the phase difference. A normalized cross-correlation calculation method (e.g., zero-mean normalized cross-correlation (ZNCC) method), a method that utilizes the sum of absolute differences (SAD), and the like have been proposed as a matching evaluation method that calculates the phase difference between two similar waveforms. 
     Such a matching evaluation process is affected by noise included in the comparison target waveforms (as described later). For example, JP-A-2001-22941 discloses a method that reduces the effects of noise by removing a noise component from the comparison target waveforms that include noise (i.e., the search target image and the template image in JP-A-2001-22941) through an estimation process. According to the method disclosed in JP-A-2001-22941, a micro-area within the target image is arbitrarily selected, and the variance within the selected micro-area is approximately calculated to be the variance of noise. Since a change in signal component is small (i.e., the signal component has an almost constant value) within the micro-area, the estimated variance is considered to approximately represent the variance of noise. A noise component is removed from the comparison target waveforms using the variance to define the matching evaluation value. 
     When using a known matching evaluation method (e.g., ZNCC or SAD), when the comparison target waveforms have a similar shape in the amplitude direction, the correlation coefficient has a maximum or minimum peak value in principle at a position at which the normalized comparison target waveforms coincide with each other. 
     SUMMARY 
     According to one aspect of the invention, there is provided an imaging device comprising: 
     an imager that captures a first object image and a second object image that have parallax with respect to an identical object; and 
     a processor comprising hardware, 
     the processor being configured to implement: 
     a phase difference detection process that calculates a correlation coefficient between a first image in which the first object image is captured, and a second image in which the second object image is captured, and detects a phase difference between the first image and the second image based on the correlation coefficient, 
     wherein the processor is configured to implement the phase difference detection process that subjects a pixel value of the first image and a pixel value of the second image to a normalization process, calculates an average value of the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process, and calculates the correlation coefficient based on a value obtained by adding up values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to a subtraction process within a fall interval in which the average value decreases, and a value obtained by adding up values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to the subtraction process within a rise interval in which the average value increases. 
     According to another aspect of the invention, there is provided a phase difference detection method comprising: 
     capturing a first object image and a second object image that have parallax with respect to an identical object; 
     subjecting a pixel value of a first image and a pixel value of a second image to a normalization process, the first image being an image in which the first object image is captured, and the second image being an image in which the second object image is captured; 
     calculating an average value of the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process; 
     calculating a correlation coefficient based on a value obtained by adding up values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to a subtraction process within a fall interval in which the average value decreases, and a value obtained by adding up values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to the subtraction process within a rise interval in which the average value increases; and 
     detecting a phase difference between the first image and the second image based on the correlation coefficient. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a view illustrating the effects of a degradation factor on a matching evaluation process. 
         FIG. 2  illustrates a configuration example of an imaging device. 
         FIG. 3  is a view illustrating an improved SAD matching evaluation process. 
         FIG. 4  illustrates simulation results for the statistical variance of a phase difference detection value with respect to the SN ratio of waveforms. 
         FIG. 5  illustrates a configuration example of an imaging device according to a second embodiment. 
         FIG. 6  is a view illustrating the basic principle of a stereo image measurement method that utilizes a pupil division technique. 
         FIG. 7  illustrates a configuration example of an imaging device according to a third embodiment. 
         FIG. 8  illustrates an example of the spectral characteristics of a pupil division filter and an image sensor. 
         FIG. 9  is a view illustrating a densification process. 
         FIGS. 10A to 10D  are views illustrating a densification process. 
         FIG. 11  is a view illustrating a densification process. 
         FIGS. 12A to 12C  illustrate simulation results for a phase difference detection process using a densification process. 
         FIG. 13  is a view illustrating a densification process. 
         FIG. 14  illustrates sampling data similarity simulation results. 
         FIG. 15  is a view illustrating a modification of a normalization method. 
     
    
    
     DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     Several aspects of the invention may provide an imaging device, a phase difference detection method, and the like that can reduce a variation in error with regard to phase difference detection. 
     Exemplary embodiments of the invention are described in detail below. Note that the following exemplary embodiments do not in any way limit the scope of the invention defined by the claims laid out herein. Note also that all of the elements described below in connection with the exemplary embodiments should not necessarily be taken as essential elements of the invention. 
     1. Outline 
     A degradation factor may be added to the comparison target waveforms that are subjected to the matching evaluation process. Examples of the degradation factor include random noise, quantization noise, deterioration in similarity between the comparison target waveforms (that occurs when the point spread functions of two pupils are asymmetric), crosstalk between the comparison target waveforms, and the like. An example in which random noise is added as the degradation factor is discussed below. Although an example in which the comparison target waveforms are those of a stereo image is described below, the matching evaluation method according to several embodiments of the invention can be applied to the case where two signal waveforms having a phase difference are compared. 
     When using a known matching evaluation method (e.g., ZNCC or SAD), a matching position detection error may occur due to the degradation factor, and it may be difficult to obtain high detection resolution or detection accuracy. 
     As illustrated in  FIG. 1 , the effects of noise more significantly occur as the frequency component of the comparison target waveforms decreases in frequency. Specifically, since the correlation coefficient of the comparison target waveforms of images that have low optical resolution corresponds to the waveform of a low-frequency component, the correlation coefficient shows a small difference between the peak and the values around the peak. Therefore, a variation in peak significantly increases when noise is added, and it becomes difficult to determine the true peak value. Therefore, noise is significantly disadvantageous when it is desired to calculate the phase difference with high accuracy. 
     When calculating the phase difference, the comparison target waveforms do not necessarily include a high-frequency component. For example, when performing a stereo image 3D measurement process, the comparison target waveforms include a large amount of high-frequency component at the in-focus position, but include only a low-frequency component at a relatively defocus position (i.e., a position at which a defocus state occurs). When implementing a 3D measurement process, since it is necessary to calculate the phase difference within a given measurement range in the depth direction, it is necessary to also use a defocused image. Therefore, it is necessary to calculate the accurate phase difference even when the comparison target waveforms include only a low-frequency component. It is important to eliminate the effects of noise as much as possible in order to implement a highly accurate phase difference detection process. 
     The method disclosed in JP-A-2001-22941 removes the estimated noise from the comparison target waveforms. However, since noise is estimated from the comparison target waveforms to which a signal component and noise are added, it may be difficult to achieve high estimation accuracy. The estimation process disclosed in JP-A-2001-22941 is performed on the assumption that a signal component has an almost constant value in a micro-area within an image. However, such a condition may not be satisfied in an area in which a change in contrast occurs to a large extent. Therefore, part of a signal component may be erroneously estimated to be a noise component, and it may be difficult to implement a highly accurate matching detection process. 
     As described above, it is important to detect the correct matching position while eliminating the effects of noise when implementing the phase difference detection process. 
     As illustrated in  FIG. 2 , an imaging device according to several embodiments of the invention includes an imager  10  that captures a first object image and a second object image that have parallax with respect to an identical object, and a phase difference detection section  30  that calculates a correlation coefficient between a first image in which the first object image is captured, and a second image in which the second object image is captured, and detects the phase difference between the first image and the second image based on the correlation coefficient. 
     The phase difference detection section  30  subjects the pixel value of the first image and the pixel value of the second image to a normalization process, calculates the average value of the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process, and calculates the correlation coefficient based on a value obtained by adding up the values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to a subtraction process within a fall interval in which the average value decreases, and a value obtained by adding up the values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to a subtraction process within a rise interval in which the average value increases. 
     In the first embodiment described later, a left-pupil image I L  and a right-pupil image I R  (see  FIG. 3 ) respectively correspond to the first image and the second image, and are subjected to the normalization process to calculate a left-pupil image nI L  and a right-pupil image nI R . The left-pupil image nI L  and the right-pupil image nI R  (obtained by the normalization process) are subjected to the subtraction process within each of a fall interval Fa in which the average value nI decreases and a rise interval Ra in which the average value nI increases (see the expressions (4) and (6)). The values obtained by the subtraction process are added up corresponding to each interval, and the resulting values are added up to calculate a correlation coefficient ISAD. 
     It is possible to change the order of subtraction so that a signal subtractive value D obtained by subjecting the comparison target waveforms to the subtraction process has a positive value corresponding to each interval (see the expressions (3) and (5)) by adding up the values obtained by subjecting the left-pupil image nI L  and the right-pupil image nI R  to the subtraction process corresponding to each of the fall interval Fa and the rise interval Ra. Note that the values obtained by subjecting the left-pupil image nI L  and the right-pupil image nI R  to the subtraction process may be added up corresponding to each interval, and the absolute value of the resulting value may be calculated so that the subtractive value D has a positive value (see the expression (13)). 
     When the correlation coefficient ISAD is calculated as described above, the signal component included in the comparison target waveforms is represented by the sum of absolute differences (i.e., the sum of |I R −I L |), and the noise component included in the comparison target waveforms is represented by the sum of differences (i.e., the sum of n R −n L ) (see the expression (9)). Therefore, the noise component is reduced by the effect of addition. It is possible to implement a phase difference detection process that reduces a variation in error and achieves high resolution or high accuracy by evaluating a position at which the correlation coefficient ISAD becomes a minimum as the matching position. Note that the above method is hereinafter referred to as “improved SAD”. 
     Note that the imaging device according to the embodiments of the invention may be configured as described below. Specifically, the imaging device includes the imager  10 , a memory that stores information (e.g., a program and various types of data), and a processor (i.e., a processor including hardware) that operates based on the information stored in the memory. The processor is configured to implement a phase difference detection process that calculates the correlation coefficient between the first image in which the first object image is captured, and the second image in which the second object image is captured, and detects the phase difference between the first image and the second image based on the correlation coefficient. The processor is configured to implement the phase difference detection process that subjects the pixel value of the first image and the pixel value of the second image to the normalization process, calculates the average value of the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process, and calculates the correlation coefficient based on a value obtained by adding up the values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to the subtraction process within the fall interval in which the average value decreases, and a value obtained by adding up the values obtained by subjecting the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to the subtraction process within the rise interval in which the average value increases. 
     The processor may implement the function of each section by individual hardware, or may implement the function of each section by integrated hardware, for example. The processor may be a central processing unit (CPU), for example. Note that the processor is not limited to a CPU. Various other processors such as a graphics processing unit (GPU) or a digital signal processor (DSP) may also be used. The processor may be a hardware circuit that includes an ASIC. The memory may be a semiconductor memory (e.g., SRAM or DRAM), a register, a magnetic storage device (e.g., hard disk drive), or an optical storage device (e.g., optical disk device). For example, the memory stores a computer-readable instruction, and each section (e.g., phase difference detection section  30  illustrated in  FIG. 2 ; or densification processing section  20  and phase difference detection section  30  illustrated in  FIG. 5 ; or densification processing section  20 , phase difference detection section  30 , ranging calculation section  80 , and three-dimensional shape output processing section  90  illustrated in  FIG. 7 ) of the imaging device is implemented by causing the processor to execute the instruction. The instruction may be an instruction included in an instruction set that is included in a program, or may be an instruction that causes a hardware circuit included in the processor to operate. 
     The operation according to the embodiments of the invention is implemented as described below, for example. The first image and the second image captured by the imager  10  are stored in the memory. The processor reads the first image and the second image from the memory, subjects the pixel value of the first image and the pixel value of the second image to the normalization process, and stores the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process in the memory. The processor reads the first image and the second image that have been subjected to the normalization process from the memory, calculates the average value of the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process, subjects the pixel value of the first image and the pixel value of the second image that have been subjected to the normalization process to the subtraction process, and stores the average value and the value obtained by the subtraction process in the memory. The processor reads the average value and the value obtained by the subtraction process from the memory, calculates a value obtained by adding up the values obtained by the subtraction process within the fall interval in which the average value decreases, and a value obtained by adding up the values obtained by the subtraction process within the rise interval in which the average value increases, and stores the values obtained by the addition process in the memory. The processor reads the values obtained by the addition process from the memory, calculates the correlation coefficient based on the values read from the memory, and stores the correlation coefficient in the memory. The processor reads the correlation coefficient from the memory, detects the phase difference based on the correlation coefficient, and stores the phase difference in the memory. 
     Each section of the imaging device according to the embodiments of the invention is implemented as a module of a program that operates on the processor. For example, the phase difference detection section  30  is implemented as a phase difference detection module that calculates the correlation coefficient between the first image in which the first object image is captured, and the second image in which the second object image is captured, and detects the phase difference between the first image and the second image based on the correlation coefficient. 
     2. First Embodiment 
     2.1. Improved SAD Matching Method 
     The details of the improved SAD are described below. The imaging device is configured in the same manner as illustrated in  FIG. 2 . An example in which the pupil of a monocular imaging optical system is divided into a left pupil and a right pupil to generate a left-pupil image (that is used as the first image) and a right-pupil image (that is used as the second image) that have parallax is described below. Note that the configuration according to the first embodiment is not limited thereto. For example, the first embodiment may also be applied to the case where a binocular imager is used. 
       FIG. 3  is a view illustrating an improved SAD (improved sum of absolute differences) matching evaluation process (i.e., a matching evaluation process that utilizes the improved SAD). 
     I L  indicates the partial profile (waveform pattern) of the captured left-pupil image, and I R  indicates the partial profile (waveform pattern) of the captured right-pupil image. Specifically, I L  and I R  indicate the pixel value patterns of the parallax images (formed on the image sensor by light that has passed through the left pupil and light that has passed through the right pupil) in the horizontal direction×(parallax direction). The pupil image I L  and the pupil image I R  have a phase difference δ. 
     Since the pupil image I L  and the pupil image I R  differ in amplitude gain, a normalization process is performed using a value within a given calculation interval w (i.e., an interval used to calculate the correlation coefficient) to adjust the amplitude gain. The normalized pupil image nI L  and the normalized pupil image nI R  are calculated by the following expression (1). Note that “w” attached to the sigma notation represents that the sum is calculated within the range of the given calculation interval w. 
     
       
         
           
             
               
                 
                   
                     
                       nI 
                       R 
                     
                     = 
                     
                       
                         I 
                         R 
                       
                       
                         
                           
                             Σ 
                             w 
                           
                            
                           
                             I 
                             R 
                             2 
                           
                         
                       
                     
                   
                   , 
                   
                       
                   
                    
                   
                     
                       nI 
                       L 
                     
                     = 
                     
                       
                         I 
                         L 
                       
                       
                         
                           
                             Σ 
                             w 
                           
                            
                           
                             I 
                             L 
                             2 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     The normalized pupil image nI L  and the normalized pupil image nI R  are added up to generate a composite waveform nI (see the following expression (2)). 
         nI=nI   R   +nI   L   (2)
 
     The cross points of the pupil image nI L  and the pupil image nI R  are detected within the given calculation interval w, and the interval between the adjacent cross points is calculated. An interval in which the composite waveform nI has a tendency to rise is referred to as “rise interval Ra”, and an interval in which the composite waveform nI has a tendency to fall is referred to as “fall interval Fa” For example, the differential value between the adjacent pixels of the composite waveform nI within the interval defined by the adjacent cross points is integrated, and the interval is determined to be the rise interval when the integral value is positive, and determined to be the fall interval when the integral value is negative. 
     A subtractive value D is calculated corresponding to the rise interval Ra and the fall interval Fa while changing the order of subtraction of the pupil image I L  and the pupil image I R  (see the following expression (3)). Specifically, the order of subtraction is determined so that “subtractive value D&gt;0” in each interval. 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             
                               
                                 Ra 
                                  
                                 
                                   : 
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                   nI 
                                   R 
                                 
                               
                               &gt; 
                               
                                 nI 
                                 L 
                               
                             
                             ∴ 
                             D 
                           
                           = 
                           
                             
                               
                                 nI 
                                 R 
                               
                               - 
                               
                                 nI 
                                 L 
                               
                             
                             &gt; 
                             0 
                           
                         
                       
                     
                     
                       
                         
                           
                             
                               
                                 Fa 
                                  
                                 
                                   : 
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                   nI 
                                   R 
                                 
                               
                               &lt; 
                               
                                 nI 
                                 L 
                               
                             
                             ∴ 
                             D 
                           
                           = 
                           
                             
                               
                                 nI 
                                 L 
                               
                               - 
                               
                                 nI 
                                 R 
                               
                             
                             &gt; 
                             0 
                           
                         
                       
                     
                   
                   } 
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     The calculated subtractive values D are added up within the given calculation interval w (see the following expression (4)) to calculate an ISAD evaluation value (matching evaluation coefficient). Note that “Ra” and “Fa” attached to the sigma notation represent that the sum is calculated corresponding to each of the ranges Ra and Fa within the given calculation interval w. When no cross point is present within the given calculation interval w, whether the given calculation interval w is the rise interval or the fall interval is determined, and the ISAD evaluation value is calculated corresponding to the given calculation interval w. 
     
       
         
           
             
               
                 
                   ISAD 
                   = 
                   
                     
                       
                         ∑ 
                         Ra 
                       
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           
                             nI 
                             R 
                           
                           - 
                           
                             nI 
                             L 
                           
                         
                         ) 
                       
                     
                     + 
                     
                       
                         ∑ 
                         Fa 
                       
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           
                             nI 
                             L 
                           
                           - 
                           
                             nI 
                             R 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
     The positions of the pupil image I L  and the pupil image I R  in the rightward-leftward direction differ between a front-focus state and a rear-focus state. When the left-pupil image I L  is shifted to the left relative to the right-pupil image I R , differing from the example illustrated in  FIG. 3 , the subtractive value D is calculated using the following expression (5) since the magnitude relationship between the normalized pupil image nI L  and the normalized pupil image nI R  differs from that illustrated in  FIG. 3 . The ISAD evaluation value is calculated using the following expression (6). 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             
                               
                                 Ra 
                                  
                                 
                                   : 
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                   nI 
                                   R 
                                 
                               
                               &lt; 
                               
                                 nI 
                                 L 
                               
                             
                             ∴ 
                             D 
                           
                           = 
                           
                             
                               
                                 nI 
                                 L 
                               
                               - 
                               
                                 nI 
                                 R 
                               
                             
                             &gt; 
                             0 
                           
                         
                       
                     
                     
                       
                         
                           
                             
                               
                                 Fa 
                                  
                                 
                                   : 
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                   nI 
                                   R 
                                 
                               
                               &gt; 
                               
                                 nI 
                                 L 
                               
                             
                             ∴ 
                             D 
                           
                           = 
                           
                             
                               
                                 nI 
                                 R 
                               
                               - 
                               
                                 nI 
                                 L 
                               
                             
                             &gt; 
                             0 
                           
                         
                       
                     
                   
                   } 
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
             
               
                 
                   ISAD 
                   = 
                   
                     
                       
                         ∑ 
                         Ra 
                       
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           
                             nI 
                             L 
                           
                           - 
                           
                             nI 
                             R 
                           
                         
                         ) 
                       
                     
                     + 
                     
                       
                         ∑ 
                         Fa 
                       
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           
                             nI 
                             R 
                           
                           - 
                           
                             nI 
                             L 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
     The magnitude relationship between the pupil image nI L  and the pupil image nI R  is determined by comparing the pixel value of the pupil image nI L  and the pixel value of the pupil image nI R  within each of the interval Ra and the interval Fa, for example. The expression (4) or the expression (6) is selected based on the determination result, and the ISAD evaluation value is calculated. 
     In the first embodiment, whether each interval is the rise interval or the fall interval is determined, and the sum of differences is calculated for the pupil image I L  and the pupil image I R  for the following reasons instead of calculating the sum of absolute differences for the pupil image I L  and the pupil image I R  without determining whether each interval is the rise interval or the fall interval (i.e., known SAD method). Note that the normalized waveform is also hereinafter referred to as “I L ”, “I R ”, or the like. 
     Suppose that the waveform patterns I L  and I R  are waveform patterns having very high similarity. A waveform obtained by adding a noise component n L  to the waveform pattern I L  is referred to as I L ′, and a waveform obtained by adding a noise component n R  to the waveform pattern I R  is referred to as I R ′ (see the following expression (7)). 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             I 
                             L 
                             ′ 
                           
                           = 
                           
                             
                               I 
                               L 
                             
                             + 
                             
                               n 
                               L 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             I 
                             R 
                             ′ 
                           
                           + 
                           
                             I 
                             R 
                           
                           + 
                           
                             n 
                             R 
                           
                         
                       
                     
                   
                   } 
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     The following expression (8) represents the case where a known SAD matching evaluation process is applied to the waveform I L ′ and the waveform I R ′. 
     
       
         
           
             
               
                 
                   SAD 
                   = 
                   
                     
                       
                         ∑ 
                         w 
                       
                        
                       
                           
                       
                        
                       
                          
                         
                           
                             I 
                             R 
                             ′ 
                           
                           - 
                           
                             I 
                             L 
                             ′ 
                           
                         
                          
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           w 
                         
                          
                         
                             
                         
                          
                         
                            
                           
                             
                               ( 
                               
                                 
                                   I 
                                   R 
                                 
                                 - 
                                 
                                   I 
                                   L 
                                 
                               
                               ) 
                             
                             + 
                             
                               ( 
                               
                                 
                                   n 
                                   R 
                                 
                                 - 
                                 
                                   n 
                                   L 
                                 
                               
                               ) 
                             
                           
                            
                         
                       
                       ≤ 
                       
                         
                           ∑ 
                           w 
                         
                          
                         
                             
                         
                          
                         
                           { 
                           
                             
                                
                               
                                 
                                   I 
                                   R 
                                 
                                 - 
                                 
                                   I 
                                   L 
                                 
                               
                                
                             
                             + 
                             
                                
                               
                                 
                                   n 
                                   R 
                                 
                                 - 
                                 
                                   n 
                                   L 
                                 
                               
                                
                             
                           
                           } 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
     The SAD evaluation value becomes 0 when the comparison target waveforms coincide with each other. However, the maximum value of the SAD evaluation value is obtained by calculating the sum of the sum of absolute differences between the waveform I L  and the waveform I R  and the sum of absolute differences between the noise component n R  and the noise component n L  (see the expression (8)). The noise component n R  and the noise component n L  may be random noise. Since the absolute value is used, the noise component n R  and the noise component n L  do not counterbalance each other even when added up. This means that the SAD evaluation value includes a large amount of noise component even when the waveform I L  and the waveform I R  coincide with each other (i.e., |I L −I R |=0). Specifically, since the SAD evaluation value does not necessarily become a minimum even when |I L −I R |=0, it is impossible to determine the correct matching position. Specifically, the SAD evaluation value is very easily affected by noise. 
     On the other hand, the relationship represented by the following expression (9) is obtained by applying the expression (7) to the ISAD evaluation value defined by the expression (4). 
     
       
         
           
             
               
                 
                   
                     
                       
                         ISAD 
                         = 
                           
                          
                         
                           
                             
                               ∑ 
                               Ra 
                             
                              
                             
                                 
                             
                              
                             
                               { 
                               
                                 
                                   ( 
                                   
                                     
                                       I 
                                       R 
                                     
                                     + 
                                     
                                       n 
                                       R 
                                     
                                   
                                   ) 
                                 
                                 - 
                                 
                                   ( 
                                   
                                     
                                       I 
                                       L 
                                     
                                     + 
                                     
                                       n 
                                       L 
                                     
                                   
                                   ) 
                                 
                               
                               } 
                             
                           
                           + 
                           
                             
                               ∑ 
                               Fa 
                             
                              
                             
                                 
                             
                              
                             
                               { 
                               
                                 
                                   ( 
                                   
                                     
                                       I 
                                       L 
                                     
                                     + 
                                     
                                       n 
                                       L 
                                     
                                   
                                   ) 
                                 
                                 - 
                                 
                                   ( 
                                   
                                     
                                       I 
                                       R 
                                     
                                     + 
                                     
                                       n 
                                       R 
                                     
                                   
                                   ) 
                                 
                               
                               } 
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                               ∑ 
                               w 
                             
                              
                             
                                 
                             
                              
                             
                                
                               
                                 
                                   I 
                                   R 
                                 
                                 - 
                                 
                                   I 
                                   L 
                                 
                               
                                
                             
                           
                           + 
                           
                             
                               ∑ 
                               w 
                             
                              
                             
                                 
                             
                              
                             
                               ( 
                               
                                 
                                   n 
                                   R 
                                 
                                 - 
                                 
                                   n 
                                   L 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     The ISAD evaluation value is calculated by calculating the sum of the sum of absolute differences between the waveform I L  and the waveform I R  and the sum of differences between the noise component n R  and the noise component n L . The sum of absolute differences between the waveform I L  and the waveform I R  becomes 0 (|I L −I R |=0) when the waveform I L  and the waveform I R  coincide with each other. The sum of differences between the noise component n R  and the noise component n L  decreases due to the effect of addition of random noise since the absolute value is not used. The sign of the difference between the noise components differs between the interval Ra and the interval Fa, but does not affect the effect of addition since the noise component is random noise. Therefore, the matching position of the waveform I L  and the waveform I R  can be evaluated using the ISAD evaluation value in a state in which noise is significantly reduced. Specifically, the ISAD evaluation value makes it possible to implement a matching evaluation process that is not easily affected by noise, and the ISAD evaluation method is superior to the SAD evaluation method. 
       FIG. 4  illustrates the simulation results for the statistical variance a of the phase difference detection value with respect to the SN ratio (SNR) of the waveform I L ′ and the waveform I R ′. 
     An edge waveform is used as the waveform I L ′ and the waveform I R ′. The phase difference when the matching evaluation value becomes a maximum (peak value) is used as the phase difference detection value. The variance a is calculated as described below. Specifically, the waveform I L ′ and the waveform I R ′ are generated while randomly changing the appearance pattern of noise having an identical power. The matching process is performed a plurality of times using the waveform I L ′ and the waveform I R ′ to calculate the phase difference. An error between the phase difference and the true value of the phase difference between the waveform I L  and the waveform I R  is calculated, and the variance a is calculated from the distribution of the occurrence of the error. 
       FIG. 4  also illustrates the variance a when using a correlation coefficient calculated using the zero-mean normalized cross-correlation (ZNCC) method. It is obvious that the error variation a of the phase difference detection value when using the ISAD evaluation value is smaller than the error variation a of the phase difference detection value when using the ZNCC evaluation value with respect to the same SN ratio. Specifically, the ISAD evaluation value is not easily affected by noise, and achieves high phase difference detection resolution. 
     For example, when the correlation calculation process is performed on low-frequency images, a variation in correlation peak increases if a degradation factor such as noise is applied, and the phase difference detection accuracy deteriorates. According to the first embodiment, however, since the variation σ in correlation peak is small as compared with the case of using a known method even when noise is applied, it is possible to implement a highly accurate phase difference detection process. 
     Note that it is possible to effectively implement a more accurate phase difference detection process by combining the improved SAD with the second embodiment or the third embodiment described later. In such a case, the imaging device is configured in the same manner as in the second embodiment or the third embodiment. In the third embodiment, the phase difference fine detection section  70  performs the phase difference detection process according to the first embodiment. 
     According to the first embodiment, the phase difference detection section (processor)  30  calculates the intersections of the pixel value of the first image nI L  and the pixel value of the second image nI R  (obtained by the normalization process) within a given interval w (given calculation interval) along the epipolar line of the first image I L  (left-pupil image) and the second image (right-pupil image) to determine a plurality of intervals that are included in the given interval w and defined by the intersections, sets an interval among the plurality of intervals in which the average value nI increases to be the rise interval Ra, and sets an interval among the plurality of intervals in which the average value nI decreases to be the fall interval Fa. 
     According to this configuration, the rise interval Ra in which the average value nI increases and the fall interval Fa in which the average value nI decreases can be determined from the average value nI of the pixel value of the first image nI L  and the pixel value of the second image nI R  (obtained by the normalization process). It is possible to add up the subtractive values corresponding to each interval (see the expression (4), (6), and (13)) by setting the rise interval Ra and the fall interval Fa, and calculate the improved SAD evaluation value. 
     The term “epipolar line” used herein refers to a straight line that is used to search two stereo images for the corresponding points. Specifically, the epipolar line is obtained by projecting a line of sight that corresponds to a point within one image onto the other image. When searching the other image for a point that corresponds to a point within the one image, the search range is limited to a range situated on the epipolar line. When the pupil of a monocular imaging optical system is divided in the horizontal scan direction (parallel stereo) (see  FIG. 7 ), the epipolar line corresponds to the horizontal scan line. 
     According to the first embodiment, the phase difference detection section (processor)  30  determines the magnitude relationship between the pixel value of the first image nI L  and the pixel value of the second image nI R  that have been subjected to the normalization process corresponding to each of the fall interval Fa and the rise interval Ra (see the expressions (3) and (5)). The phase difference detection section  30  subjects the pixel value of the first image nI L  and the pixel value of the second image nI R  that have been subjected to the normalization process to the subtraction process corresponding to each interval based on the determined magnitude relationship so that the values D obtained by the subtraction process are positive values, and adds up the values obtained by the subtraction process to calculate the correlation coefficient ISAD (see the expressions (3) to (6)). 
     According to this configuration, it is possible to determine the magnitude relationship between the waveform of the first image nI L  and the waveform of the second image nI R  corresponding to each of the rise interval Ra and the fall interval Fa (see the expressions (3) and (5)), and determine the order of waveform subtraction using the quantitative relationship so that the subtractive value D has a positive value corresponding to each interval. This makes it possible to allow the signal component |I R −I L | of the correlation coefficient ISAD to remain as the sum of absolute differences, and reduce the noise component (n R -n L ) as the sum of differences by utilizing the effect of addition (see the expression (9)). 
     2.2. Modification of Normalization Process 
     The normalization process (method) used for the pupil image I L  and the pupil image I R  is not limited to the normalization process represented by the expression (1). For example, the normalization process may be performed as described below. 
     The left side in  FIG. 15  illustrates the partial profile (waveform pattern) of the left-pupil image I L  and the partial profile (waveform pattern) of the right-pupil image I R . When implementing the matching evaluation method (ISAD) according to the first embodiment, it is necessary to normalize the level of the left-pupil image I L  and the level of the right-pupil image I R  that differ from each other to obtain the waveform patterns illustrated on the right side in  FIG. 15 . 
     For example, a gain correction process is performed on the right-pupil image I R  within an interval w R . For example, the left-pupil image I L  and the right-pupil image I R  have a similar waveform within the interval w, and the shift amounts δ U  and δ D  are very small. In this case, it is obvious that the pixel value of the right-pupil image I R  at the x-position U R  (upper peak) corresponds to the pixel value of the left-pupil image I L  at the x-position U L  (upper peak). It is also obvious that the pixel value of the right-pupil image I R  at the x-position D R  (lower peak) corresponds to the pixel value of the left-pupil image I L  at the x-position D L  (lower peak). 
     Therefore, the peak positions U R , D R , U L , and D L  are calculated. The average value Av(R) is calculated within the range of the peak positions U R  and D R , and the average value Av(L) is calculated within the range of the peak positions U L  and D L  (see the following expression (10)). 
     
       
         
           
             
               
                 
                   
                     
                       Av 
                        
                       
                         ( 
                         R 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             x 
                             = 
                             
                               U 
                               R 
                             
                           
                           
                             D 
                             R 
                           
                         
                          
                         
                             
                         
                          
                         
                           I 
                           R 
                         
                       
                       
                         w 
                         R 
                       
                     
                   
                   , 
                   
                       
                   
                    
                   
                     
                       Av 
                        
                       
                         ( 
                         L 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             x 
                             = 
                             
                               U 
                               L 
                             
                           
                           
                             D 
                             L 
                           
                         
                          
                         
                             
                         
                          
                         
                           I 
                           L 
                         
                       
                       
                         w 
                         L 
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     A correction gain is calculated from the average value Av(R) and the average value Av(L), and the normalization process is performed within the interval w R  using the following expression (11). The left-pupil image I L  and the right-pupil image I R  thus have waveform patterns having the same level (see the right side in  FIG. 15 ). 
     
       
         
           
             
               
                 
                   
                     
                       nI 
                       R 
                     
                     = 
                     
                       
                         
                           Av 
                            
                           
                             ( 
                             L 
                             ) 
                           
                         
                         
                           Av 
                            
                           
                             ( 
                             R 
                             ) 
                           
                         
                       
                       × 
                       
                         I 
                         R 
                       
                     
                   
                   , 
                   
                       
                   
                    
                   
                     
                       nI 
                       L 
                     
                     = 
                     
                       I 
                       L 
                     
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     Note that the normalization expression is not limited to the expression (11). For example, the normalization process may be performed on both the left-pupil image I L  and the right-pupil image I R  (see the following expression (12)). 
     
       
         
           
             
               
                 
                   
                     
                       nI 
                       R 
                     
                     = 
                     
                       
                         I 
                         R 
                       
                       
                         Av 
                          
                         
                           ( 
                           R 
                           ) 
                         
                       
                     
                   
                   , 
                   
                       
                   
                    
                   
                     
                       nI 
                       L 
                     
                     = 
                     
                       
                         I 
                         L 
                       
                       
                         Av 
                          
                         
                           ( 
                           L 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   12 
                   ) 
                 
               
             
           
         
       
     
     In either case, since the left-pupil image I L  and the right-pupil image I R  are normalized by performing the gain correction process on the intervals (intervals w R  and w L ) in which matching occurs, the matching evaluation value can be obtained by comparing the waveforms having the same level. Moreover, it is possible to utilize the relationship between the adjacent peaks when calculating the phase difference in a state in which the left-pupil image I L  and the right-pupil image I R  are close to each other, and the shift amount is very small 
     2.3. Second ISAD Evaluation Value Calculation Method 
     The ISAD evaluation value calculation method is not limited to the expressions (4) and (6). For example, the following expression (13) may be used. 
     
       
         
           
             
               
                 
                   ISAD 
                   = 
                   
                     
                        
                       
                         
                           ∑ 
                           Ra 
                         
                          
                         
                             
                         
                          
                         
                           ( 
                           
                             
                               nI 
                               R 
                             
                             - 
                             
                               nI 
                               L 
                             
                           
                           ) 
                         
                       
                        
                     
                     + 
                     
                        
                       
                         
                           ∑ 
                           Fa 
                         
                          
                         
                             
                         
                          
                         
                           ( 
                           
                             
                               nI 
                               L 
                             
                             - 
                             
                               nI 
                               R 
                             
                           
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
     According to the expression (13), the phase difference detection section  30  adds up the values obtained by subjecting the pixel value of the first image nI L  (left-pupil image) and the pixel value of the second image nI R  (right-pupil image) that have been subjected to the normalization process to the subtraction process within the fall interval Fa, and calculates the absolute value of the resulting value. The phase difference detection section  30  adds up the values obtained by subjecting the pixel value of the first image nI L  and the pixel value of the second image nI R  that have been subjected to the normalization process to the subtraction process within the rise interval Ra, and calculates the absolute value of the resulting value. The phase difference detection section  30  adds up the absolute value that corresponds to the fall interval Fa and the absolute value that corresponds to the rise interval Ra to calculate the correlation coefficient ISAD. 
     The value obtained by subjecting the pupil images nI R  and nI L  is either a positive value or a negative value within the rise interval Ra or the fall interval Fa. A positive value or a negative value is obtained by integrating the value obtained by subjecting the pupil images nI R  and nI L  within each interval. The ISAD evaluation value similar to that calculated using the expressions (4) and (6) can be obtained by calculating the absolute value of the resulting value. According to this method, it is unnecessary to change the order of subtraction when subjecting the pupil images nI R  and nI L  to the subtraction process since the resulting value may be a negative value. 
     For example, the magnitude relationship between the pupil images nI R  and nI L  differs between the case where the left-pupil image nI L  is shifted to the right with respect to the right-pupil image nI R  and the case where the left-pupil image nI L  is shifted to the left with respect to the right-pupil image nI R . Therefore, it is necessary to determine the magnitude relationship between the pupil images nI R  and nI L  within each interval, and adaptively change the order of subtraction. However, it is unnecessary to change the order of subtraction by utilizing the expression (13). 
     3. Second Embodiment 
     A second embodiment of the invention is described below. In the second embodiment, a densification process with regard to the sampling pitch is performed by image processing, and an accurate phase difference detection process is performed using high-density parallax images. It is possible to implement a more accurate phase difference detection process by applying the improved SAD (see above). 
     According to a known phase difference detection process, the phase difference detection resolution is determined by the density of the sampling pixels that correspond to each parallax image (i.e., each of two parallax images) captured using the pupil division technique. Specifically, the waveform pattern of each parallax image is handled as data sampled corresponding to each sampling pixel (see the left side in  FIG. 13 ). When calculating the correlation coefficient while shifting the relative position of two waveform patterns from the initial position, the correlation coefficient is obtained at a relative position at which the sampling positions of the two waveform patterns coincide with each other. Therefore, the matching position detection resolution is determined by the sampling density, and the resolution of the phase difference that is the difference between the initial position and the matching position is also determined by the sampling density. 
     For example, a case where the phase difference detection process is applied to a ranging process is discussed below. The range resolution Δz is determined by the phase difference detection resolution Δs (as described later with reference to the expression (15)). Specifically, it is necessary to increase the phase difference detection resolution in order to implement a high-resolution ranging process. However, the pixel density of an image sensor has approached the upper limit of the optical resolution, and it is not considered that a significant improvement in pixel density will be achieved in the future. Therefore, it is a great challenge to implement high-density sampling at a sampling density equal to or higher than the pixel density of an image sensor. 
       FIG. 5  illustrates a configuration example of an imaging device according to the second embodiment that can solve the above problem. The imaging device includes an imager  10  that includes an optical low-pass filter  11 , a densification processing section  20 , and a phase difference detection section  30 . 
     The densification processing section (processor)  20  performs a densification process that increases the number of pixels of the first image and the second image to virtually decrease the sampling pitch of the first object image and the second object image. The phase difference detection section  30  detects the phase difference between the first image and the second image that have been subjected to the densification process. 
     For example, a monocular imaging optical system is subjected to pupil division, and parallax images are acquired using an image sensor having a Bayer array (see the third embodiment described later). The first object image that has passed through the first pupil is captured using the red pixels, and the second object image that has passed through the second pupil is captured using the blue pixels. The first image and the second image that have a sampling density (pixel pitch p/N) that is higher than the pixel density (pixel pitch p) of the image sensor by a factor of N are thus generated. 
     According to this configuration, it is possible to generate parallax images of which the apparent sampling density is significantly higher than the pixel density of the image sensor. It is possible to implement a phase difference detection process with a significantly improved detection resolution by detecting the phase difference using the resulting parallax images. According to the above example, since the correlation coefficient can be calculated at an N-fold density, it is possible to detect the phase difference at an N-fold resolution. 
     4. Third Embodiment 
     4.1. Stereo Image Measurement Method 
     The details of the densification process are described below in connection with a third embodiment of the invention. In the third embodiment, a monocular imager is subjected to pupil division, and different colors are respectively assigned to the two pupils to acquire parallax images, which are subjected to the densification process. 
     The basic principle of the stereo image measurement method that utilizes the pupil division technique is described below with reference to  FIG. 6 . Although an example in which the pupil is divided in the rightward-leftward direction (horizontal scan direction) is described below, the pupil need not necessarily be divided in the rightward-leftward direction. It suffices that the pupil be divided in an arbitrary direction that is orthogonal to the optical axis. 
     Reflected light from the surface of the object passes through an imaging lens  12  (imaging optical system), forms an image in the image sensor plane, and is acquired by the image sensor as an image signal. The coordinate axes when a reference position RP of the object is set to be the origin are referred to as (x, y, z), and the coordinate axes when an in-focus position RP′ in the image sensor plane is set to be the origin are referred to as (x′, y′). For example, the x′-axis corresponds to the horizontal scan direction of the image sensor, and the y′-axis corresponds to the vertical scan direction of the image sensor. The z-axis corresponds to the direction along the optical axis of the imaging lens  12  (i.e., depth distance direction). 
     The distance from the reference position RP of the object to the center of the imaging lens  12  is referred to as a 0 , and the distance from the center of the imaging lens  12  to the image sensor plane is referred to as b 0 . The distance a 0  and the distance b 0  are determined by the design of the imager. 
     The left half of the imaging lens  12  is referred to as a left pupil, and the right half of the imaging lens  12  is referred to as a right pupil. GP L  is the center-of-gravity position (center of gravity) of the left pupil, and GP R  is the center-of-gravity position (center of gravity) of the right pupil. An image obtained in the image sensor plane is defocused as the surface of the object moves away from the reference position in the z-direction, and an image I L  that has passed through the left pupil and an image I R  that has passed through the right pupil (hereinafter referred to as “left-pupil image” and “right-pupil image”, respectively) are shifted from each other (i.e., have a phase difference s). Although  FIG. 6  illustrates an example in which the pupil position is situated at the center of the lens for convenience of explanation, the pupil position is present at a position (e.g., aperture) outside the lens in the actual situation. 
     The relationship between the phase difference s and the position z of the surface of the object is calculated. The relationship between the phase difference s between the left-pupil image I L  and the right-pupil image I R  obtained in the image sensor plane and the position z of the surface of the object is determined by the following expression (14). 
     
       
         
           
             
               
                 
                   s 
                   = 
                   
                     
                       
                         - 
                         Ml 
                       
                       · 
                       z 
                     
                     
                       z 
                       + 
                       
                         a 
                         0 
                       
                     
                   
                 
               
               
                 
                   ( 
                   14 
                   ) 
                 
               
             
           
         
       
     
     Note that M is the total optical magnification at a reference in-focus position. When the imaging field-of-view circle diameter is φIC, and the field-of-view circle diameter in the imaging range is φOC, M=φIC/φOC=b 0 /a 0 . 1 is the distance between the center of gravity GP L  of the left pupil and the center of gravity GP R  of the right pupil. Note that the expression (14) is satisfied with respect to the axis of the optical system. A relational expression with respect to the outside of the axis is omitted for convenience of explanation. 
     It is necessary to separately acquire the left-pupil image I L  and the right-pupil image I R  in order to calculate the phase difference s. The left-pupil image I L  and the right-pupil image I R  may be separately acquired (separated) in various ways. For example, a red-pass optical filter is provided at the left pupil position, and a blue-pass optical filter is provided at the right pupil position. A red image obtained by the image sensor is separated as the left-pupil image, and a blue image obtained by the image sensor is separated as the right-pupil image. Alternatively, the left-pupil image and the right-pupil image are separately acquired using the angle of light that enters the image sensor plane (see JP-A-2009-145401). Alternatively, parallax stereo images that correspond to the left-pupil image and the right-pupil image are separately acquired using a binocular camera. These methods may be selectively used corresponding to the intended use (objective) and the application. 
     It is important to increase the z resolution in order to implement an accurate 3D measurement process. The following expression (15) is obtained by transforming the expression (14) so that the z resolution Δz is represented using the phase difference resolution Δs. 
     
       
         
           
             
               
                 
                   
                     Δ 
                      
                     
                         
                     
                      
                     z 
                   
                   = 
                   
                     
                       
                         - 
                         Δ 
                       
                        
                       
                           
                       
                        
                       s 
                     
                     
                       Ml 
                       + 
                       
                         Δ 
                          
                         
                             
                         
                          
                         s 
                       
                     
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
           
         
       
     
     As is clear from the expression (15), it is necessary to decrease the z resolution Δz by decreasing the phase difference resolution Δs in order to improve the measurement resolution in the z-direction. Specifically, it is necessary to more finely detect the phase difference between the left-pupil image and the right-pupil image in order to increase the z resolution Δz. It is necessary to increase the sampling density of the left-pupil image and the right-pupil image using the image sensor in order to more finely detect the phase difference between the left-pupil image and the right-pupil image. However, the sampling density is limited by the pixel pitch of an image sensor, and the pixel pitch of an image sensor has approached the limit. It is difficult to further reduce the pixel pitch of an image sensor. 
     4.2. Imaging Device 
       FIG. 7  illustrates a configuration example of an imaging device according to the third embodiment. The imaging device includes an imager  10 , a densification processing section  20  (densification measurement development section), a phase difference detection section  30 , an optical characteristic memory  40 , a ranging calculation section  80 , and a three-dimensional shape output processing section  90 . Note that the same elements as those described above are indicated by the same reference signs (symbols), and description thereof is appropriately omitted. 
     The imager  10  includes an optical low-pass filter  11 , an imaging lens  12  (imaging optical system), a pupil division filter  13 , an image sensor  14 , and an imaging processing section  15 . 
     An R (red) filter is provided to the pupil division filter  13  corresponding to the left pupil, and a B (blue) filter is provided to the pupil division filter  13  corresponding to the right pupil. The image sensor  14  is an RGB color image sensor having a Bayer pixel array.  FIG. 8  illustrates the spectral characteristics of the pupil division filter  13  and the image sensor  14 . F L  indicates the spectral characteristics of the left-pupil filter (R filter), and F R  indicates the spectral characteristics of the right-pupil filter (B filter). T B , T G , and T R  indicate the spectral characteristics of the B pixel, the G (green) pixel, and the R pixel, respectively. The pupil spectral characteristics F L  and F R  are divided at the cross point (wavelength λc) of the spectral characteristics T B  of the B pixel and the spectral characteristics T R  of the R pixel, and cover the entire RGB band. The spectral characteristics F L  and F R  are designed to allow the G component (part of the G component) to pass through. 
     Note that the spectral characteristics {T B , T G , T R } are defined as composite spectral characteristics of the characteristics of the color filters provided to the image sensor  14  on a pixel basis, the spectral characteristics of external light or illumination light applied to the object, and the spectral characteristics of each pixel. The parameters regarding the spectral characteristics are setting values (corresponding values) with respect to the wavelength λ. Note that the notation of the wavelength λ used as a dependent variable is omitted. 
     Reflected light from the object passes through the imaging lens  12 , the pupil division filter  13 , and the optical low-pass filter  11 , and forms an image on the image sensor  14 . In this case, a component value calculated by multiplying the spectral characteristics of the reflected light from the object by the left-pupil spectral characteristics F L  and the spectral characteristics T R  of the R pixel is obtained as the pixel value of the R pixel. Likewise, a component value calculated by multiplying the spectral characteristics of the reflected light from the object by the right-pupil spectral characteristics F R  and the spectral characteristics T B  of the B pixel is obtained as the pixel value of the B pixel. Specifically, the left-pupil image is obtained by the R image included in the Bayer image, and the right-pupil image is obtained by the B image included in the Bayer image. 
     The imaging processing section  15  controls the imaging operation, and processes an imaging signal. For example, the imaging processing section  15  converts the pixel signal from the image sensor  14  into digital data, and outputs Bayer-array image data (RAW image data). 
     The densification processing section  20  performs the sampling density densification process for detecting the phase difference between the R image and the B image at a resolution smaller (lower) than the sampling pixel pitch. The densification process increases the sampling density by a factor of N×N. Note that N is 100 to 10,000, for example. The details of the densification process are described later. 
     Note that the densification processing section  20  may perform a high-accuracy separation process on the R image and the B image based on the spectral characteristics F R , F L , T B , T G , and T R  stored in the optical characteristic memory  40 . For example, the spectral characteristics T B  of the R pixel also have a component within the band of the left-pupil spectral characteristics F L . Therefore, the R image (right-pupil image) includes the left-pupil component mixed therein. The densification processing section  20  may perform a process that reduces such a right pupil-left pupil mixed state based on the spectral characteristics F R , F L , T B , T G , and T R . 
     The phase difference detection section  30  includes a phase difference rough detection section  50 , a detectable area extraction section  60  (detectable feature part extraction section), and a phase difference fine detection section  70 . 
     The phase difference rough detection section  50  performs the phase difference detection process that is lower in density than the phase difference detection process performed by the phase difference fine detection section  70 . For example, the phase difference rough detection section  50  performs a correlation calculation process on the image that has been subjected to the densification process or the Bayer image that has not been subjected to the densification process in a state in which the pixels are thinned out. 
     The detectable area extraction section  60  determines whether or not a phase difference can be detected based on the correlation coefficient from the phase difference rough detection section  50 , determines whether or not the distance information in the z-direction can be acquired based on the determination result, and outputs an image of the detectable area to the phase difference fine detection section  70 . For example, the detectable area extraction section  60  determines whether or not a phase difference can be detected by determining whether or not a correlation peak is present. 
     The phase difference fine detection section  70  performs the phase difference detection process on the image that has been subjected to the densification process to finely detect the phase difference at a resolution smaller than the sampling pixel pitch. The phase difference fine detection section  70  performs the phase difference detection process on the area for which it has been determined by the detectable area extraction section  60  that a phase difference can be detected. 
     The ranging calculation section  80  calculates the distance in the z-direction at a high resolution based on the phase difference detected by the phase difference fine detection section  70 . The three-dimensional shape output processing section  90  generates three-dimensional shape data based on the distance information in the z-direction, and outputs the generated three-dimensional shape data. 
     4.3. Densification Process 
     The sampling density densification process is described in detail below. 
     The right-pupil image and the left-pupil image (R pupil image and B pupil image) that have passed through the optical low-pass filter  11  are sampled by the color image sensor  14 . The R pixels and the B pixels are arranged in the image sensor  14  as illustrated in  FIG. 9 . The optical low-pass filter  11  is an anti-aliasing filter, and is provided so that folding noise does not occur in the R pupil image and the B pupil image. Since the sampling pitch of each pupil image is 2 p, the sampling frequency is 1/(2 p), and the cut-off frequency is set to be equal to or lower than the Nyquist frequency (1/(4 p)) determined corresponding to the sampling frequency. 
       FIG. 10A  illustrates the frequency characteristics of the R image and the B image. Specifically, when the frequency characteristics of the optical LPF are represented by 1/(4 p), for example, the R image and the B image have a band within the range from −1/(4 p) to +1/(4 p). The repetition cycle is 1/(2 p) (not illustrated in the drawings). Note that the dotted line represents the frequency characteristics of the pixel aperture. The pixel aperture has a band within the range from −1/p to +1/p corresponding to the aperture width p. 
     Next, data is generated so that each sampling pixel of the R pupil image and the B pupil image obtained by the image sensor  14  includes micro-pixels (apparent pixels) that have a size equal to or smaller than that of one pixel. For example, when generating the pixel values sampled using pixels having a size 1/10th of that of one pixel in the vertical direction and the horizontal direction, one pixel is equally divided into ten areas (N=10) in the vertical direction and the horizontal direction so that one pixel includes one hundred (N×N=100) micro-pixels. The pixel value of the original pixel is used as the pixel value of each micro-pixel. The above upsampling process is performed on each R pixel and each B pixel. 
       FIG. 10B  illustrates the frequency characteristics of the resulting R image and the resulting B image. Since each pixel is merely divided, and the data is merely duplicated, the frequency characteristics are the same as those before the upsampling process is performed. Specifically, the R image and the B image have a band within the range from −1/(4 p) to +1/(4 p), and the repetition cycle is 1/(2 p), for example. 
     The sampling data formed by the micro-pixels is filtered using a two-dimensional low-pass filter, and the micro-pixels (including pixels in an undetected area) over the entire captured image are reconstructed. Specifically, image data having a pixel pitch of p/N (p=pixel pitch of image sensor  14 ) is generated to obtain an N-fold sampling density (apparent sampling density). The cut-off frequency of the two-dimensional low-pass filter is set to be equal to or lower than the Nyquist frequency (1/(4 p)) that is determined by the R or B sampling pitch 2 p in the same manner as the optical low-pass filter. The two-dimensional low-pass filter is a Gaussian filter, for example. 
     The two-dimensional low-pass filter has the frequency characteristics illustrated in  FIG. 10C , for example. The R image and the B image that have been subjected to the two-dimensional low-pass filtering process have the frequency characteristics illustrated in  FIG. 10D , for example. The repetition frequency changes to N/p since the pixel pitch has changed to p/N. The band of the R image and the B image corresponds to a band calculated by multiplying the frequency characteristics of the optical low-pass filter by the frequency characteristics of the two-dimensional low-pass filter. 
     The left-pupil image (R pupil image) I L  and the right-pupil image (B pupil image) I R  before being subjected to the densification process are images sampled at a pitch of 2 p (see the left side in  FIG. 11 ). The left-pupil image (R pupil image) I L  and the right-pupil image (B pupil image) I R  that have been subjected to the densification process are obtained as image data sampled at a density (pitch: p/N) that is significantly higher than the sampling density of the image sensor  14  (see the right side in  FIG. 11 ). 
       FIGS. 12A to 12C  illustrate the simulation results for the phase difference detection process using the densification process. The horizontal axis indicates the shift amount (pixels) from the initial position “0” used for the correlation calculation process. 
       FIG. 12A  illustrates a waveform for calculating the phase difference. When the pixel pitch of the image sensor is p, the waveform I(x) and the waveform I(x-δ) have a phase difference δ of 0.2 p (waveforms sampled at a pixel pitch of p). 
       FIG. 12B  illustrates the simulation results when the sampling waveform I(x) and the sampling waveform I(x-δ) are merely upsampled (0.1 p) (i.e., one pixel is divided on a 0.1 p basis, and the pixel value of the pixel is duplicated), and the cross-correlation coefficient is calculated (shift: 0.1 p). The correlation peak should be detected at δ=0.2 p. However, since the sampling waveform has only a value on a pixel basis, the correlation peak is observed at δ=0 (i.e., detection at a resolution equal to or lower than one pixel is not achieved). 
       FIG. 12C  illustrates the simulation results when the densification process according to the third embodiment is performed on the sampling waveform I(x) and the sampling waveform I(x-δ), and the cross-correlation coefficient is calculated. Specifically,  FIG. 12C  illustrates the simulation results when the low-pass filter having a cut-off frequency equal to or lower than 1/(4 p) is applied after the upsampling process, and the cross-correlation coefficient is calculated (shift: 0.1 p). A clear correlation peak is observed at 6=0.2 p (i.e., a resolution of 0.1 p is achieved). 
     Specifically, it is possible to achieve a phase difference detection resolution equal to or smaller than the pixel pitch of the image sensor by performing the upsampling process and the two-dimensional low-pass filtering process according to the third embodiment. 
     When using a normal phase difference detection process, the similarity between the left-pupil image (R pupil image) sampling data and the right-pupil image (B pupil image) sampling data deteriorates due to the difference in sampling position. The method according to the third embodiment can solve this problem. This feature is described below with reference to  FIG. 13 . 
     As illustrated in  FIG. 13  (see the left side), the left-pupil image (R pupil image) I L  and the right-pupil image (B pupil image) I R  have an approximately identical waveform, and have a phase difference S. The right side in  FIG. 13  illustrates a state in which the waveform of the pupil image I L  and the waveform of the pupil image I R  are caused to overlap each other. In this case, the pupil image I L  and the pupil image I R  are matched, and it is desirable that the correlation coefficient at a position at which the waveforms have the highest similarity be obtained. 
     However, since the parallax δ is arbitrary, the R pixel sampling position and the B pixel sampling position normally differ from each other with respect to the pupil image I L  and the pupil image I R  that have an approximately identical waveform. Therefore, even when the left-pupil image (R pupil image) I L  and the right-pupil image (B pupil image) I R  are optically identical, different sampling data is obtained (i.e., the similarity is lost), for example. This means that it is impossible to calculate the correct position when calculating the matching position of the pupil image I L  and the pupil image I R  from the correlation coefficient. 
     For example, when the correlation coefficient is calculated while shifting the pupil image I L  and the pupil image I R  by one sampling pixel (i.e., at a pitch of 2 p), the correlation coefficient is obtained at each position at which the pixel of the pupil image I L  and the pixel of the pupil image I R  (i.e., solid arrow and dotted arrow) coincide with each other. Specifically, the correlation coefficient when the waveforms coincide with each other is not obtained when the pupil image I L  and the pupil image I R  differ in sampling position, and a phase difference detection error occurs. 
     According to the third embodiment, since the high-density sampling data of the pupil image I L  and the pupil image I R  can be obtained (see the right side in  FIG. 11 ), it is possible to ensure that the sampling data have similarity, and improve the phase difference detection accuracy. Moreover, since the noise component superimposed on the R pupil image I L  and the B pupil image I R  is reduced by applying the two-dimensional low-pass filtering process, it is possible to suppress or reduce a variation in matching position detection error due to noise. 
       FIG. 14  illustrates the sampling data similarity simulation results. The upper part in  FIG. 14  illustrates the sampling position. The sampling positions B 2 , B 4 , B 6 , and B 8  are positions that are sequentially shifted from the sampling position A by 0.2 p. For example, when the phase difference is 0.6 p, the left-pupil image is sampled at the sampling position A, and the right-pupil image is sampled at the sampling position B 6 . 
     The middle part in  FIG. 14  illustrates the sampling data. The sampling data represents data obtained by sampling the sensor input waveform at the sampling positions A, B 2 , B 4 , B 6 , and B 8 . The sensor input waveform is the waveform of the object image formed in the sensor plane. In this case, the similarity between the sampling data is low due to the difference in sampling position. 
     The lower part in  FIG. 14  illustrates the results obtained by subjecting the sampling data to the densification process according to the third embodiment. The waveform data As, Bs 2 , Bs 4 , Bs 6 , and Bs 8  correspond to the sampling positions A, B 2 , B 4 , B 6 , and B 8 . The waveform data As, Bs 2 , Bs 4 , Bs 6 , and Bs 8  coincide with each other, and cannot be distinguished from each other (i.e., the similarity between the sampling data is high). It is possible to implement a highly accurate phase difference detection process by utilizing the sampling data having high similarity. 
     According to the third embodiment, the imager  10  includes the optical low-pass filter  11  that has a cut-off frequency equal to or lower than 1/(2 P) when the pitch of the pixels used to capture the first object image and the pitch of the pixels used to capture the second object image are P. The densification processing section (processor)  20  performs the densification process that includes performing the upsampling process on the first image I L  (left-pupil image, R image) and the second image I R  (right-pupil image, B image), and performing the two-dimensional low-pass filtering process on the first image I L  and the second image I R  that have been subjected to the upsampling process. 
     In the third embodiment, the sampling pitch of the first image I L  and the sampling pitch of the second image I R  are P=2 p (see  FIG. 9 ). The upsampling process is performed on the first image I L  and the second image I R  to increase the number of pixels of the first image I L  and the number of pixels of the second image I R  by a factor of N×N, and the two-dimensional low-pass filtering process is performed on the first image I L  and the second image I R . 
     This makes it possible to obtain parallax images having an apparent sampling density (pixel pitch p/N) that is higher than the pixel density (pixel pitch p) of the image sensor by a factor of N. It is possible to implement a phase difference detection process with a significantly improved detection resolution by detecting the phase difference using the resulting parallax images (as described above with reference to  FIG. 12  and the like). 
     It is possible to implement a more accurate phase difference detection process by applying the improved SAD (see above). Specifically, since a high-frequency component is cut by the two-dimensional LPF during the densification process, the phase difference detection process may be affected by noise since the phase difference between the waveforms of the low-frequency components are detected. However, since the effects of noise can be reduced by applying the improved SAD, it is possible to maximize the detection resolution achieved by the densification process. 
     When the third embodiment is applied to a binocular imager, image sensors are respectively provided to binocular imaging optical systems, for example. In this case, the sampling pitch P of each parallax image is the same as the pixel pitch p of the image sensor (i.e., P=p). 
     According to the third embodiment, the imager  10  includes the imaging optical system (imaging lens  12 ), the pupil division filter  13  that divides the pupil of the imaging optical system into a first pupil (left pupil) that allows the first object image to pass through, and a second pupil (right pupil) that allows the second object image to pass through, and the image sensor  14  that captures the first object image and the second object image formed by the imaging optical system. 
     According to this configuration, it is possible to capture parallax images using the monocular imager  10 . It is possible to implement a high-resolution ranging process using a monocular system by subjecting the parallax images to the densification process. Specifically, it is necessary to increase the pupil-to-pupil center-of-gravity distance  1  in order to increase the resolution Δz of the ranging process (see the expression (15)). However, it is difficult to increase the pupil-to-pupil center-of-gravity distance  1  when using a monocular system as compared with the case of using a binocular system. According to the third embodiment, however, since the phase difference detection resolution Δs can be increased by utilizing the densification process, it is possible to implement a high-resolution ranging process even when the pupil-to-pupil center-of-gravity distance  1  is short (see the expression (15)). For example, a reduction in the diameter of a scope is desired for an endoscope (e.g., industrial endoscope and medical endoscope). It is possible to easily implement a reduction in the diameter of a scope when using a monocular system, and it is possible to implement a highly accurate ranging process by utilizing the densification process even when the pupil-to-pupil center-of-gravity distance  1  has decreased due to a reduction in the diameter of the scope. 
     According to the third embodiment, the image sensor  14  is an image sensor having a primary-color Bayer array. The pupil division filter  13  includes a filter that corresponds to the first pupil and allows light within a wavelength band that corresponds to red to pass through (spectral characteristics F L  illustrated in  FIG. 8 ), and a filter that corresponds to the second pupil and allows light within a wavelength band that corresponds to blue to pass through (spectral characteristics F R  illustrated in  FIG. 8 ). The densification processing section (processor) 20  performs the densification process on a red image and a blue image included in a Bayer-array image captured by the image sensor  14 , the red image corresponding to the first image (left-pupil image), and the blue image corresponding to the second image (right-pupil image). 
     This makes it possible to implement a high-resolution phase difference detection process using a color image sensor having a primary-color Bayer array that is widely used. Since the parallax images can be formed by merely inserting the pupil division filter  13 , and extracting the R image and the B image, it is possible to implement a high-resolution phase difference detection process without changing a known imager to a large extent. Since only the pupil division filter  13  is additionally provided to the optical system, it is possible to use the imager  10  having a compact configuration, and implement an endoscope having a small diameter (see above), for example. 
     According to the third embodiment, when the pixel pitch of the image sensor  14  is referred to as p, the pitch of the red pixels used to capture the first object image and the pitch of the blue pixels used to capture the second object image are P=2 p. The cut-off frequency of the optical low-pass filter  11  is equal to or lower than 1/(2 P)=1/(4 p). 
     When implementing a normal capture operation without using the pupil division technique, the Nyquist frequency that corresponds to the pixel pitch p of the image sensor is 1/(2 p), and the cut-off frequency of the optical low-pass filter  11  is set to be equal to or lower than 1/(2 p). According to the third embodiment, since the sampling process is performed on each parallax image, the cut-off frequency of the optical low-pass filter  11  is set to be equal to or lower than the Nyquist frequency 1/(4 p) that corresponds to the sampling pitch 2 p. This makes it possible to suppress or reduce the occurrence of folding noise in the parallax images. 
     According to the third embodiment, the densification processing section (processor) 20  performs the upsampling process that divides each pixel of the first image and the second image into N×N pixels, and duplicates the pixel value of the original pixel to the N×N pixels. 
     According to the third embodiment, the cut-off frequency of the two-dimensional low-pass filtering process is equal to or lower than 1/(2 P). It is possible to provide data that includes micro-pixels by dividing each pixel of the parallax images into N×N pixels, and duplicating the pixel value of the original pixel. It is possible to generate the parallax images (as if the sampling process were performed using the micro-pixels) by subjecting the data to the two-dimensional low-pass filtering process using a cut-off frequency equal to or lower than 1/(2 P). Since the frequency band of the parallax image is limited to be equal to or lower than 1/(2 P) due to the optical low-pass filter  11 , it is possible to reduce noise outside the band while allowing the component of the parallax image to remain by setting the cut-off frequency of the two-dimensional low-pass filter to be equal to or lower than 1/(2 P). 
     Although an example that utilizes a color image sensor having a primary-color Bayer array has been described above, the configuration is not limited thereto. For example, a complementary-color image sensor may also be used. In this case, an R image and a B image are generated from a YCrCb image captured by the complementary-color image sensor, and used as the parallax images. 
     The embodiments to which the invention is applied and the modifications thereof have been described above. Note that the invention is not limited to the above embodiments and the modifications thereof. Various modifications and variations may be made without departing from the scope of the invention. A plurality of elements described in connection with the above embodiments and the modifications thereof may be appropriately combined to implement various configurations. For example, some elements may be omitted from the elements described in connection with the above embodiments and the modifications thereof. The elements described above in connection with different embodiments or modifications thereof may be appropriately combined. Specifically, various modifications and applications are possible without materially departing from the novel teachings and advantages of the invention. Any term cited with a different term having a broader meaning or the same meaning at least once in the specification and the drawings can be replaced by the different term in any place in the specification and the drawings.