Patent Publication Number: US-11029149-B2

Title: Multipath mitigation for time of flight system

Description:
BACKGROUND 
     Time-of-flight (ToF) systems produce a depth image of an object, each pixel of such image encoding the distance to the corresponding point in the object. In recent years, time-of-flight depth-imaging technology has become more accurate and more affordable. These advances are a consequence of improved imaging-array fabrication and intelligent post-processing, which garners improved signal-to-noise levels from the raw output of the array. 
     SUMMARY 
     A time-of-flight (ToF) system disclosed herein provides a method of a method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector, the method comprising acquiring three or more images represented by three or more matrices in response to illuminating a target with a light source using a first spatial pattern at three or more different modulation frequencies, acquiring an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, and determining one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix. 
     This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. 
     Other implementations are also described and recited herein. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A further understanding of the nature and advantages of the present technology may be realized by reference to the figures, which are described in the remaining portion of the specification. In the figures, like reference numerals are used throughout several figures to refer to similar components. 
         FIG. 1  illustrates an example implementation of the time-of-flight multipath mitigation technology disclosed herein. 
         FIG. 2  illustrates example operations for multipath mitigation according to the implementation disclosed in  FIG. 1 . 
         FIG. 3  illustrates an example schematics of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 1 . 
         FIG. 4  illustrates alternative example operations for multipath mitigation according to the implementation disclosed in  FIG. 1 . 
         FIG. 5  illustrates alternative example schematics of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 1 . 
         FIG. 6  illustrates an alternative example implementation of the time-of-flight multipath mitigation technology disclosed herein. 
         FIG. 7  illustrates example operations for multipath mitigation according to the implementation disclosed in  FIG. 6 . 
         FIG. 8  illustrates an example schematics of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 6 . 
         FIG. 9  illustrates an alternative example implementation of the time-of-flight multipath mitigation technology disclosed herein. 
         FIG. 10  illustrates example operations for multipath mitigation according to the implementation disclosed in  FIG. 9 . 
         FIG. 11  illustrates an example computing system that may be used to implement the technology disclosed herein. 
     
    
    
     DETAILED DESCRIPTION 
     Time-of-flight (ToF) systems produce a depth image of an object or a scene, each pixel of such image encoding the distance to the corresponding point in the object or the scent. When a scene is illuminated by a source of light, the radiance of each point in the scene can be viewed as having two components—a direct component and a global component. The term “radiance” and “light,” of a point on a surface are used herein interchangeably to refer to the amount of electromagnetic radiation leaving or arriving at that point on the surface. Here the direct component is due to the direct illumination of a point in the scene (referred to as the “scene point”) by the source of light and the global component is due to the indirect illumination of the scene point. Such indirect illumination of the scene point may be due to interreflection of light between scene points, subsurface scattering of light within a medium beneath the surface of the scene, volumetric scattering of light, diffusion of light by translucent surface, etc. 
       FIG. 1  illustrates a ToF system  100  including a ToF module  102  for separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector. Specifically, a ToF module  102  may include a computing module  104  that includes a processor  106  and a spatial pattern data store  108 . The spatial pattern data store  108  may store one or more spatial patterns, such as spatial patterns  132 ,  134  that may be used to spatially modulate illumination signal from a light source  110 . The light source  110  may be a two-dimensional grid of light source elements, referred to as light source pixels. For example, the light source  110  may be an array of m×n light source elements, each light source element being a laser diode. 
     In one implementation, the processor  106  may communicate with the light source  110  to cause the illumination signal generated by the light source to be modulated by the spatial patterns  132   a ,  132   b ,  132   c  (collectively “spatial patterns  132 ”), and  134 . For example, the spatial patterns  132  may be uniform patterns and the spatial pattern  134  is a pattern of horizontal lines. The spatial patterns  132  and  134  are two dimensional patterns that map to the two-dimensional grid of light source elements  110   a ,  110   b . Additionally, other patterns such as a dot-pattern may also be used to modulate the illumination signal generated by the light source. Specifically, each of the spatial patterns  132   a ,  132   b ,  132   c  are the same but they are temporally apart from each other and for each of the spatial patterns  132   a ,  132   b ,  132   c , the light source is modulated at different frequencies k (k∈K, K−1, . . . , 1). Moreover, when the light source is modulated by the spatial pattern  134 , the light source is also frequency modulated at the higher of the k frequencies such as K. In one implementation of the ToF system  100 , K=3. Alternatively, K may be any number greater than or equal to three. 
     The light source  110  may include a large number of light source pixels. The illustrated implementation shows one such light source pixel  110   a  generating a light signal that is projected on a surface  152  of an object  150 . The light signal generated by the light source pixel  110   a  may include a direct light component  112  and a global light component  112   a . Specifically, the direct light component  112  is directly incident on a point  154  on the surface  152  whereas the global light component  112   a  is incident on the point  154  indirectly as a result of an interreflection. The scattering of each of the direct light component  112  and the global light component  112   a  at the point  154  results in a total light signal  114  that is captured by a camera  120  of the ToF module  102 . 
     An implementation of the camera  120  includes a plurality of camera pixels including a camera pixel  120   a , wherein each camera pixel is assumed to be able to observe at most one significant scattering event from the surface  152 . As a result, is assumed that two different light source pixels cannot produce a direct light component along a given camera pixels line of sight.  FIG. 1  illustrates one such camera pixel including its imaging lens  116 , a light sampling array  124 , and a sampler  124 . For example, the sampling array  122  may be a charge coupled device (CCD) array that converts light energy into electrical charge in analog form. The sampler  124  may include an analog-digital converter (ADC) that converts the electrical charge into a digital signal. The combination of the imaging lens  116 , the light sampling array  124 , and the sampler  124  receives and converts the total light signal  114  into a sampled signal  126  that is communicated to a multipath mitigation module  128 . The multipath mitigation module  128  includes an image analyzer  130  that analyzes images captured by the camera  120  to determine various parameters of a direct component and a global component of the total light signal  114 . 
     In one implementation, the light source  110  and the camera  120  may have same number of pixels represented by a matrix with a resolution of m×n such that m is number of rows and n is the number of columns. Each pixel of the camera  120  may be represented by (i,j), with i=1, 2, . . . , m and j=1, 2, . . . n. In such an implementation, the light L collected by each pixel of a continuous wave time-of-flight (CW ToF) camera can be represented by L(i,j) Specifically, as the total light  114  is a combination of the scattering of the direct light component  112  and a global light component  112   a , L(i,j) is a linear combination of a direction component of light L d (i,j) and a global component of light L g (i,j), as disclosed below in equation 1.
 
 L   i,j   =L   d ( i,j )+ L   g ( i,j )  (1)
 
     Here the direct component of light L d (i,j) can be defined as the light for which the origin is the source pixel  110   a  and it is backreflected directly by the point  154  of the surface  152 . The global component of the light L g (i,j) includes feasible contribution of various global components, including the back-reflected light collected by the camera pixel (i,j) other than the light for which the origin is the source pixel  110   a  and it is backreflected directly by the point  154  of the surface  152 . Thus the global component of the light L g (i,j) includes global light component  112   a  reflected at point  154  or light coming from other light sources such as light  112   b  from a source pixel  110   b  backreflected directly by the another point  154   b , and other illumination sources or the contribution of the ambient light. 
     The signal L(i,j) received by the camera pixel  120   a  is a complex number that can be written in phasor notation as the multiplication of the amplitude A and the complex exponential of the associated angle ϕ. Specifically, as the signal L(i,j) is a linear combination of the direct and global component of the light, Denominating the direct component of the light with the subscript d and the global component with the subscript g, the signal L(i,j) at the camera pixel  120   a  may be disclosed as per the equation 2 below:
 
 L ( i,j )= A ( i,j ) e   iϕ(i,j)   =A   d ( i,j ) e   iϕ     d     (i,j)   +A   g ( i,j ) e   iϕ     g     (i,j)   (2)
 
     When the amplitude and phase of the global light component  112   a  are non-zero, the time of flight distance calculated for the total light signal  114  as well the magnitude of the total light signal  114  are modified compared to when the amplitude and phase of the global light component  112   a  are zero. 
     The solution disclosed in  FIG. 1  collects images with two different spatial patterns. Specifically, in the ToF system  100 , the processor  106  temporally modulates the source light generated by the light source  110  using the spatial patterns  132  and  134 . In one implementation, the processor  106  temporally modulates the source light generated by the light source  110  using the spatial patterns such that k source light images are modulated using one of the spatial patterns  132  and frequency modulated at each of k frequencies and a subsequent image is modulated with spatial pattern  134  and at one of the k frequencies. In one implementation, the subsequent image is modulated with spatial pattern  134  preferably at the highest of the k frequencies. For example, in one such implementation, at time interval t 0, . . . k-1 , the light source  110  is modulated using spatial pattern  132  and at time source t k , the light source  110  is modulated by the spatial pattern  134 . 
     The modulated source light illuminates the object  150 . The camera  120  acquires the total light  114  scattered by the object  150  in response to the illumination by the modulated source light. Specifically, the camera  120  acquires k+1 images (k images with spatial pattern  132  and k frequencies and k+1th image with spatial pattern  134  and modulation frequency K) of the object  150  at the camera  120  to separate the direction component of light P d (i,j) and the global component of light P g (i,j). If the signal collected when a uniform pattern  132  illuminates the light source can be represented by U(i,j) and the signal collected when the spatial pattern  134  illuminates the light source can be represented by U(i,j), U(i,j) and P(i,j) can be represented by P(i,j) as provided below in equation 3.
 
 U=A   d   e   iϕ     d     +A   g   e   iϕ     g    
 
 P=γ·A   d   e   iϕ     d     +C·A   g   e   iϕ     g     (3)
 
     Note that each of the U and P represent matrices. Thus, U represents a matrix U(i,j), with (i,j), with i=1, 2, . . . , m and j=1, 2, . . . n. As seen from equation 3, each of the U and P has a direct component and a global component. The amplitudes of the direct component and the global components for each of the U and P are represented by, respectively, A d , and A g . While the phases of the direct component and the global components for each of the U and P are represented by, respectively, ϕ d , and ϕ g ·γ is the spatial pattern contrast regarding the uniform illumination, and C the global contrast, i.e., the light loss induced by using a pattern. 
     Rewriting equations 3 in real and complex parts using Euler notation, represented by underscripts r and c being real and complex part of the signal, and D being the direct component of the signal and G being the global component of the signal, equations 3 may be rewritten as follows, not including (i,j) for clarity:
 
 U   r   =D   r   +G   r  
 
 U=D   c   +G   c  
 
 P   r   =γ·D   r   +C·G   r  
 
 P   c   =γ·D   c   +C·G   c   (4)
 
     In equation 7 above, the subscripts r and c denote the real and the complex parts of the phasors. D is the direct component of the total light  114  captured by a pixel of the camera  120  and G is the global component of the total light  114  captured by a pixel of the camera  120 . Solving these equations, the values of the variables {D r , D c , G r , G c } may be provided by equation 8 below: 
     
       
         
           
             
               
                 
                   
                     
                       D 
                       r 
                     
                     = 
                     
                       
                         
                           - 
                           
                             P 
                             r 
                           
                         
                         + 
                         
                           C 
                           ⁢ 
                           
                             U 
                             r 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       D 
                       c 
                     
                     = 
                     
                       
                         
                           - 
                           
                             P 
                             c 
                           
                         
                         + 
                         
                           C 
                           ⁢ 
                           
                             U 
                             c 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       G 
                       r 
                     
                     = 
                     
                       
                         
                           - 
                           
                             P 
                             r 
                           
                         
                         + 
                         
                           γ 
                           ⁢ 
                           
                             U 
                             r 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       G 
                       c 
                     
                     = 
                     
                       
                         
                           - 
                           
                             P 
                             c 
                           
                         
                         + 
                         
                           γ 
                           ⁢ 
                           
                             U 
                             c 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     The amplitudes of the direct component and the global {A d , A g } and the phases of the direct and global components {ϕ d ,ϕ g } can calculated as per Equation 6 below: 
     
       
         
           
             
               
                 
                   
                     
                       A 
                       d 
                     
                     = 
                     
                       
                         
                           
                             
                               ( 
                               
                                 
                                   - 
                                   
                                     P 
                                     r 
                                   
                                 
                                 + 
                                 
                                   C 
                                   ⁢ 
                                   
                                     U 
                                     r 
                                   
                                 
                               
                               ) 
                             
                             2 
                           
                           + 
                           
                             
                               ( 
                               
                                 
                                   - 
                                   
                                     P 
                                     c 
                                   
                                 
                                 + 
                                 
                                   C 
                                   ⁢ 
                                   
                                     U 
                                     c 
                                   
                                 
                               
                               ) 
                             
                             2 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       A 
                       g 
                     
                     = 
                     
                       
                         
                           
                             
                               ( 
                               
                                 
                                   - 
                                   
                                     P 
                                     r 
                                   
                                 
                                 + 
                                 
                                   γ 
                                   ⁢ 
                                   
                                     U 
                                     r 
                                   
                                 
                               
                               ) 
                             
                             2 
                           
                           + 
                           
                             
                               ( 
                               
                                 
                                   - 
                                   
                                     P 
                                     c 
                                   
                                 
                                 + 
                                 
                                   γ 
                                   ⁢ 
                                   
                                     U 
                                     c 
                                   
                                 
                               
                               ) 
                             
                             2 
                           
                         
                       
                       
                         C 
                         - 
                         γ 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       ϕ 
                       d 
                     
                     = 
                     
                       atan 
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               - 
                               
                                 P 
                                 c 
                               
                             
                             + 
                             
                               C 
                               ⁢ 
                               
                                 U 
                                 c 
                               
                             
                           
                           
                             
                               - 
                               
                                 P 
                                 r 
                               
                             
                             + 
                             
                               C 
                               ⁢ 
                               
                                 U 
                                 r 
                               
                             
                           
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       ϕ 
                       g 
                     
                     = 
                     
                       atan 
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               - 
                               
                                 P 
                                 c 
                               
                             
                             + 
                             
                               γ 
                               ⁢ 
                               
                                 U 
                                 c 
                               
                             
                           
                           
                             
                               - 
                               
                                 P 
                                 r 
                               
                             
                             + 
                             
                               γ 
                               ⁢ 
                               
                                 U 
                                 r 
                               
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
     Here average contrast C and immediate contrast γ are considered as constant for materials with diffuse backscattering and they can be evaluated from a previous calibration pattern or calculated from an estimation of the immediate contrast γ The immediate contrast γ may be obtained from calibration of the patterns  132  or  134 . C can be calculated as the mean or median of all the values of the scene provided by the immediate contrast. Where A d &gt;A g , the γ may be calculated as 
     
       
         
           
             γ 
             = 
             
               
                  
                 P 
                  
               
               
                  
                 U 
                  
               
             
           
         
       
     
     The above solution allows to mitigate the multipath, however, as it can be seen from equation 6, when γ→C, this solution results in a residue that produces a structure that depends on the pattern  134 —for example, when the spatial pattern  134  has vertical bars, the residue is a vertical structure. To reduce the vertical residue, the multipath mitigation module  128  includes a denoising module  140  that performs a denoising operation as further disclosed below in  FIG. 2  below. 
       FIG. 2  illustrates example operations for multipath mitigation according to the implementation disclosed in  FIG. 1 . Specifically,  FIG. 2  illustrates example operations  200  for separating the direct component of the light from the global component that contains the multipath. One or more operations  200  may be performed by the image analyzer  130  disclosed in  FIG. 1 . Specifically, an operation  204  modulates a light source (such as the light source  110  of  FIG. 1 ) using a uniform spatial pattern (such as the uniform spatial patterns  132  of  FIG. 1 ) from a pattern data store. In the illustrated implementation, the operation  204  modulates the light source with the uniform spatial pattern and temporally modulates the light signal with k different frequencies. For example, k may be greater than or equal to three. An operation  206  illuminates an object (such as the object  150  of  FIG. 1 ) using the modulated light source. The modulated light is scattered from the object towards a camera (such as the camera  150  of  FIG. 1 ). An operation  208  acquires k images where the object is illuminated with the light source being modulated with the uniform spatial pattern and k frequencies. 
     Similarly, an operation  214  acquires an additional k+1 th  image where the object is illuminated with the light source being modulated with the non-uniform spatial pattern and frequency modulated with one of the k frequencies. In one implementation, operation  214  acquires an additional k+1 th  image where the object is illuminated with the light source being modulated with the non-uniform spatial pattern and frequency modulated with the highest of the k frequencies. An operation  216  illuminates the object with the modulated light signal and an operation  218  acquires an additional, k+1 th , image when the light source is modulated with a spatial pattern and frequency modulated with one of the k frequencies. In one implementation, operation  216  illuminates the object with the modulated light signal and an operation  218  acquires an additional, k+1 th , image when the light source is modulated with a spatial pattern and frequency modulated with the highest of the k frequencies. Each of the k+1 images may be in the form of a matrix with each observation of the matrix corresponding to individual pixels of the camera. 
     An operation  230  generate an example of denoising coefficient (m,n) to increase the signal to noise ratio (SNR). In one implementation, the denoising operation generates the denoising or weighing coefficient (m,n) using the equation below: 
     
       
         
           
             
               ξ 
               ⁡ 
               
                 ( 
                 
                   m 
                   , 
                   n 
                 
                 ) 
               
             
             = 
             
               
                 
                   [ 
                   
                     C 
                     - 
                     
                       γ 
                       ⁡ 
                       
                         ( 
                         
                           m 
                           , 
                           n 
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               m 
                               - 
                               
                                 m 
                                 _ 
                               
                             
                             
                               σ 
                               m 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               n 
                               - 
                               
                                 n 
                                 _ 
                               
                             
                             
                               σ 
                               n 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ∑ 
                   
                     m 
                     = 
                     
                       - 
                       M 
                     
                   
                   M 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       n 
                       = 
                       
                         - 
                         N 
                       
                     
                     N 
                   
                   ⁢ 
                   
                     
                       [ 
                       
                         C 
                         - 
                         
                           γ 
                           ⁡ 
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                         
                       
                       ] 
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   m 
                                   - 
                                   
                                     m 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   m 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   n 
                                   - 
                                   
                                     n 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   n 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                   
                 
               
             
           
         
       
     
     Here m (−M, . . . , M) and n (−N, . . . , N) denotes a size of a kernel matrix applied to each pixel locations in each of the k+1 images acquired at operations  208  and  218 . Note that m and n do not have to be equal, therefore, the kernel may be rectangular, which allows using patterns with predominant direction. Depending on the size of the kernel and the values of the standard deviations, the denoising coefficient ξ(m,n) can produce a reduction in the spatial frequencies of the image. An example kernel may be a kernel to apply immediate contrast. A property o such an immediate contrast kernel is that if the kernel is large, the average contrast C is calculated as the global change of optical power between the uniform and the pattern image as provided below: 
     
       
         
           
             
               
                 ∑ 
                 
                   m 
                   = 
                   
                     - 
                     M 
                   
                 
                 M 
               
               ⁢ 
               
                 
                   ∑ 
                   
                     n 
                     = 
                     
                       - 
                       N 
                     
                   
                   N 
                 
                 ⁢ 
                 
                   γ 
                   ⁡ 
                   
                     ( 
                     
                       m 
                       , 
                       n 
                     
                     ) 
                   
                 
               
             
             -&gt; 
             C 
           
         
       
     
     Subsequently an operation  232  determines the real and complex parameters of the direct and global components of a signal received at the camera using the following equation: 
     
       
         
           
             Direct 
             ⁢ 
             
                 
             
             ⁢ 
             
               { 
               
                 
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             r 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               K 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               ξ 
                               ⁢ 
                               
                                 { 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               r 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 K 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             U 
                                             r 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               K 
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             c 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               K 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               
                                 ξ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               c 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 K 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             U 
                                             c 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               K 
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
                 ⁢ 
                 
                   
 
                 
                 ⁢ 
                 Global 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   { 
                   
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               r 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 K 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 ξ 
                                 ⁢ 
                                 
                                   { 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 r 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   K 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               U 
                                               r 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 K 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               c 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 K 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 
                                   ξ 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       m 
                                       , 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 c 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   K 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               U 
                                               c 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 K 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     An operation  234  performs range calculation. 
       FIG. 3  illustrates an example schematics  300  of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 1 . Specifically,  FIG. 3  illustrates three uniform patterns  310   a ,  310   b ,  310   c  that may be used to modulate a light source applied to an object  315 . Specifically, each of the uniform patterns  310   a ,  310   b ,  310   c  are same, however, temporally the light source is modulated at different frequency when applying these uniform patterns. Thus, for example, the modulation frequency of the light source when uniform pattern  310   a  is applied maybe f K , the modulation frequency of the light source when uniform pattern  310   b  is applied maybe f K-1 , and the modulation frequency of the light source when uniform pattern  310   c  is applied maybe f K-2 . Subsequently, a non-uniform spatial pattern  320  is applied to the light source with modulation frequency f K . 
       FIG. 4  illustrates alternative example operations  400  for multipath mitigation according to the implementation disclosed in  FIG. 1 . One or more of the operations  400  may be substantially similar to the operations  200  disclosed in  FIG. 2 . However, the operations  400  differ from the operations  200  in that while operation  204  modulates the light source with uniform patterned spatial modulation, operation  404  modulates the light source with first patterned spatial modulation. Specifically, the first spatial pattern used by operation  404  is different from the second spatial patter used by an operation  414 . 
       FIG. 5  illustrates alternative example schematics  500  of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 1 . Specifically,  FIG. 5  illustrates three horizontal patterns  510   a ,  510   b ,  510   c  that may be used to modulate a light source applied to an object  515 . Specifically, each of the uniform patterns  510   a ,  510   b ,  510   c  are same, however, temporally the light source is modulated at different frequency when applying these uniform patterns. Thus, for example, the modulation frequency of the light source when uniform pattern  510   a  is applied maybe f K , the modulation frequency of the light source when uniform pattern  510   b  is applied maybe f K-1 , and the modulation frequency of the light source when uniform pattern  510   c  is applied maybe f K-2 . Subsequently, another non-uniform spatial pattern  320 , specifically a vertical spatial pattern, is applied to the light source with modulation frequency f K . 
       FIG. 6  illustrates an alternative example implementation  600  of the time-of-flight multipath mitigation technology disclosed herein. One or more of the elements disclosed in  FIG. 6  are substantially similar to those recited in  FIG. 1 . However, the multipath mitigation module  628  has one or more functions different than the multipath mitigation module  128  disclosed in  FIG. 1 , the denoising module  640  performs one or more functions compared to the denoising module  140  disclosed in  FIG. 1 , and the spatial pattern data store  608  may store one or more spatial patterns different than the spatial pattern data store  108  disclosed in  FIG. 1 . These elements are further disclosed below with respect to  FIG. 6 . 
     A ToF module  602  may include a computing module  604  that includes a processor  606  and a spatial pattern data store  608 . The spatial pattern data store  608  may store one or more spatial patterns, such as spatial patterns  632 ,  634  that may be used to spatially modulate illumination signal from a light source  610 . The light source  610  may be a two-dimensional grid of light source elements, referred to as light source pixels. For example, the light source  610  may be an array of m×n light source elements, each light source element being a laser diode. 
     In one implementation, the processor  606  may communicate with the light source  610  to cause the illumination signal generated by the light source to be modulated by the spatial patterns  632   a ,  632   b  (collectively “spatial patterns  632 ”), and  634 . For example, the spatial patterns  632  may be uniform patterns and the spatial pattern  634  is a pattern of horizontal lines. The spatial patterns  632  and  634  are two dimensional patterns that map to the two-dimensional grid of light source elements  610   a ,  610   b . Additionally, other patterns such as a dot-pattern may also be used to modulate the illumination signal generated by the light source. Specifically, all of the spatial patterns  632   a ,  632   b  are the same but they are temporally apart from each other and for each of the spatial patterns  632   a  and  632   b  the light source is modulated at different frequencies k (k∈K, K−1, . . . , 1). Moreover, when the light source is modulated by the spatial pattern  134 , the light source is also frequency modulated at the higher of the k frequencies such as K. In one implementation of the ToF system  600 , K=2. Alternatively, K may be any number greater than or equal to two. 
     The light source  610  may include a large number of light source pixels. The illustrated implementation shows one such light source pixel  610   a  generating a light signal that is projected on a surface  652  of an object  650 . The light signal generated by the light source pixel  610   a  may include a direct light component  612  and a global light component  612   a . Specifically, the direct light component  612  is directly incident on a point  654  on the surface  652  whereas the global light component  612   a  is incident on the point  654  indirectly as a result of an interreflection. The scattering of each of the direct light component  612  and the global light component  612   a  at the point  654  results in a total light signal  614  that is captured by a camera  620  of the ToF module  602 . 
     An implementation of the camera  620  includes a plurality of camera pixels including a camera pixel  620   a , wherein each camera pixel is assumed to be able to observe at most one significant scattering event from the surface  652 . As a result, is assumed that two different light source pixels cannot produce a direct light component along a given camera pixels line of sight.  FIG. 6  illustrates one such camera pixel including its imaging lens  616 , a light sampling array  624 , and a sampler  624 . For example, the sampling array  622  may be a charge coupled device (CCD) array that converts light energy into electrical charge in analog form. The sampler  624  may include an analog-digital converter (ADC) that converts the electrical charge into a digital signal. The combination of the imaging lens  616 , the light sampling array  624 , and the sampler  624  receives and converts the total light signal  614  into a sampled signal  626  that is communicated to a multipath mitigation module  628 . The multipath mitigation module  628  includes an image analyzer  630  that analyzes images captured by the camera  620  to determine various parameters of a direct component and a global component of the total light signal  614 . 
     The modulated source light illuminates the object  650 . The camera  620  acquires the total light  614  scattered by the object  650  in response to the illumination by the modulated source light. Specifically, the camera  620  acquires k+1 images (k images with spatial pattern  632  and k frequencies and k+1 th  image with spatial pattern  634  and modulation frequency K) of the object  650  at the camera  620  to separate the direction component of light P d (i,j) and the global component of light P g (i,j). 
     The light source  610  may include a large number of light source pixels. The illustrated implementation shows one such light source pixel  610   a  generating a light signal that is projected on a surface  652  of an object  650 . The light signal generated by the light source pixel  610   a  may include a direct light component  612  and a global light component  612   a . Specifically, the direct light component  612  is directly incident on a point  654  on the surface  652  whereas the global light component  612   a  is incident on the point  654  indirectly as a result of an interreflection. The scattering of each of the direct light component  612  and the global light component  612   a  at the point  654  results in a total light signal  614  that is captured by a camera  620  of the ToF module  602 . 
     In the present implementation, the multipath mitigation module  628  includes a denoising module  640  and a frequency conversion module  644 . The denoising module  640  generates a denoising coefficient (m,n) to increase the signal to noise ratio (SNR). An example of the denoising module  640  generates the denoising coefficient (m,n) using the equation below: 
     
       
         
           
             
               ξ 
               ⁡ 
               
                 ( 
                 
                   m 
                   , 
                   n 
                 
                 ) 
               
             
             = 
             
               
                 
                   [ 
                   
                     C 
                     - 
                     
                       γ 
                       ⁡ 
                       
                         ( 
                         
                           m 
                           , 
                           n 
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               m 
                               - 
                               
                                 m 
                                 _ 
                               
                             
                             
                               σ 
                               m 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               n 
                               - 
                               
                                 n 
                                 _ 
                               
                             
                             
                               σ 
                               n 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ∑ 
                   
                     m 
                     = 
                     
                       - 
                       M 
                     
                   
                   M 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       n 
                       = 
                       
                         - 
                         N 
                       
                     
                     N 
                   
                   ⁢ 
                   
                     
                       [ 
                       
                         C 
                         - 
                         
                           γ 
                           ⁡ 
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                         
                       
                       ] 
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   m 
                                   - 
                                   
                                     m 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   m 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   n 
                                   - 
                                   
                                     n 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   n 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                   
                 
               
             
           
         
       
     
     Here m (−M, . . . , M) and n (−N, . . . , N) denotes a size of a kernel matrix applied to each pixel locations in each of the k+1 images acquired at operations  708  and  718  disclosed below in  FIG. 7 . Note that m and n do not have to be equal, therefore, the kernel may be rectangular, which allows using patterns with predominant direction. Depending on the size of the kernel and the values of the standard deviations, the denoising coefficient (m,n) can produce a reduction in the spatial frequencies of the image captured by a camera  620  of the ToF module  602 . 
     In an example implementation, the frequency conversion module  644  is based on constant Time of Flight for the total of frequencies. Specifically, time t(i,j) can be converted into total phase ϕ(i,j,k) of a specific frequency f(k) using the following equation
 
ϕ( i,j,k )=4π t ( i,j ) f ( k )
 
This total phase is related to the measured phase through the following equation:
 
ϕ( i,j,k )={tilde over (ϕ)}( i,j,k )+2π PO ( i,j,k )
 
     The module  644  performs conversion of frequencies keeping the active brightness of the frequencies to be converted f(k→K) with k&lt; &gt;K and transforming the phase into the frequency K using the following equations: 
     
       
         
           
             
               ϕ 
               ⁡ 
               
                 ( 
                 
                   i 
                   , 
                   j 
                   , 
                   
                     k 
                     → 
                     K 
                   
                 
                 ) 
               
             
             = 
             
               
                 [ 
                 
                   
                     
                       
                         
                           ϕ 
                           ~ 
                         
                         k 
                       
                       ⁡ 
                       
                         ( 
                         
                           i 
                           , 
                           j 
                           , 
                           k 
                         
                         ) 
                       
                     
                     + 
                     
                       2 
                       ⁢ 
                       π 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         PO 
                         ⁡ 
                         
                           ( 
                           
                             i 
                             , 
                             j 
                             , 
                             k 
                           
                           ) 
                         
                       
                     
                   
                   
                     f 
                     ⁡ 
                     
                       ( 
                       k 
                       ) 
                     
                   
                 
                 ] 
               
               ⁢ 
               
                 f 
                 ⁡ 
                 
                   ( 
                   K 
                   ) 
                 
               
             
           
         
       
     
     Where {tilde over (ϕ)}(i,j,k→K) is the total converted phase from frequency f(k) into f(K), {tilde over (ϕ)} k (i,j,k) is the measured phase for the specific frequency, and PO(i,j,k)∈ ≡{0, 1, 2, . . . } is the number of phase orders (wraps) required for producing a constant Timeof Flight for the total number of frequencies. Assuming that the casted frequencies correspond to the uniform acquisitions, the converted signal can be described as below:
 
 Ũ ( i,j,k→K )= Ã ( i,j,k ) e   iϕ(i,j,k→K)  
 
     After this transformation, the multipath mitigation module  628  separates the global and direct components of the light using the following equation. 
     
       
         
           
             Direct 
             ⁢ 
             
                 
             
             ⁢ 
             
               { 
               
                 
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             r 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               K 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               ξ 
                               ⁢ 
                               
                                 { 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               r 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             U 
                                             r 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               
                                                 k 
                                                 -&gt; 
                                                 K 
                                               
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             c 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               K 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               
                                 ξ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               c 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             U 
                                             c 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               
                                                 k 
                                                 -&gt; 
                                                 K 
                                               
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
                 ⁢ 
                 
                   
 
                 
                 ⁢ 
                 Global 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   { 
                   
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               r 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 K 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 ξ 
                                 ⁢ 
                                 
                                   { 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 r 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   k 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               U 
                                               r 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 
                                                   k 
                                                   -&gt; 
                                                   K 
                                                 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               c 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 k 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 
                                   ξ 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       m 
                                       , 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 c 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   k 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               U 
                                               c 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 
                                                   k 
                                                   -&gt; 
                                                   K 
                                                 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
       FIG. 7  illustrates example operations  700  for multipath mitigation according to the implementation disclosed in  FIG. 6 . Specifically,  FIG. 7  illustrates example operations  700  for separating the direct component of the light from the global component that contains the multipath. One or more operations  700  may be performed by the image analyzer  630  disclosed in  FIG. 6 . Specifically, an operation  704  modulates a light source (such as the light source  610  of  FIG. 6 ) using a uniform spatial pattern (such as the uniform spatial patterns  632  of  FIG. 6 ) from a pattern data store. In the illustrated implementation, the operation  704  modulates the light source with the uniform spatial pattern and temporally modulates the light signal with k different frequencies. For example, k may be equal to two. An operation  706  illuminates an object (such as the object  650  of  FIG. 6 ) using the modulated light source. The modulated light is scattered from the object towards a camera (such as the camera  620  of  FIG. 6 ). An operation  708  acquires k images where the object is illuminated with the light source being modulated with the uniform spatial pattern and k frequencies. 
     Similarly, an operation  714  acquires an additional k+1 th  image where the object is illuminated with the light source being modulated with the non-uniform spatial pattern and frequency modulated with one of the k frequencies. In one implementation, operation  714  acquires an additional k+1 th  image where the object is illuminated with the light source being modulated with the non-uniform spatial pattern and frequency modulated with, preferably, the highest of the k frequencies. An operation  716  illuminates the object with the modulated light signal and an operation  718  acquires an additional, k+1 th , image when the light source is modulated with a spatial pattern and frequency modulated with one of the k frequencies. Each of the k+1 images may be in the form of a matrix with each observation of the matrix corresponding to individual pixels of the camera  620 . 
     An operation  730  generates a denoising coefficient (m,n) to increase the signal to noise ratio (SNR) as discussed above with respect to the denoising module  640 . An operation  732  performs one or more range calculation operations. Subsequently an operation  734  performs frequency conversion as discussed above with respect to the frequency conversion module  644 . Finally, an operation  736  determines the real and the complex parameters of the direct and global components of the light using the calculation discussed above with respect to the multipath mitigation module  628 . 
       FIG. 8  illustrates an example schematics  800  of the implementation of the time-of-flight multipath mitigation technology disclosed in  FIG. 6 . Specifically,  FIG. 8  illustrates three uniform patterns  810   a  and  810   b  that may be used to modulate a light source directed to an object  825 . Specifically, each of the uniform patterns  810   a  and  810   b  are same, however, temporally the light source is modulated at different frequency when applying these uniform patterns. Thus, for example, the modulation frequency of the light source when uniform pattern  810   a  is applied maybe f K  and the modulation frequency of the light source when uniform pattern  810   b  is applied maybe f K-1 . Subsequently, a non-uniform spatial pattern  820  is applied to the light source directed to an object  825  with modulation frequency f K . 
       FIG. 9  illustrates an alternative example implementation  900  of the ToF multipath mitigation technology disclosed herein. One or more of the elements disclosed in  FIG. 9  are substantially similar to those recited in  FIG. 1 . However, the multipath mitigation module  928  has one or more functions different than the multipath mitigation module  128  disclosed in  FIG. 1 , the denoising module  940  performs one or more functions compared to the denoising module  140  disclosed in  FIG. 1 , and the spatial pattern data store  908  may store one or more spatial patterns different than the spatial pattern data store  108  disclosed in  FIG. 1 . These elements are further disclosed below with respect to  FIG. 9 . 
     A ToF module  902  may include a computing module  904  that includes a processor  906  and a spatial pattern data store  908 . The spatial pattern data store  908  may store one or more spatial patterns, such as spatial patterns  932 ,  934  that may be used to spatially modulate illumination signal from a light source  910 . The light source  910  may be a two-dimensional grid of light source elements, referred to as light source pixels. For example, the light source  910  may be an array of m×n light source elements, each light source element being a laser diode. 
     In one implementation, the processor  906  may communicate with the light source  910  to cause the illumination signal generated by the light source to be modulated by the spatial patterns  934   a ,  934   b ,  934   c  (collectively “spatial patterns  934 ”). For example, the spatial patterns  934  may be a pattern of horizontal lines. The spatial patterns  934  are two dimensional patterns that map to the two-dimensional grid of light source elements  910   a ,  910   b . Additionally, other patterns such as a dot-pattern may also be used to modulate the illumination signal generated by the light source. Specifically, all of the spatial patterns  934  are the same but they are temporally apart from each other and for each of the spatial patterns  934  the light source is modulated at different frequencies k (k∈K, K−1, . . . , 1). In one implementation of the ToF system  900 , K=3. Alternatively, K may be any number greater than or equal to three. 
     The light source  910  may include a large number of light source pixels. The illustrated implementation shows one such light source pixel  910   a  generating a light signal that is projected on a surface  952  of an object  950 . The light signal generated by the light source pixel  910   a  may include a direct light component  912  and a global light component  912   a . Specifically, the direct light component  912  is directly incident on a point  954  on the surface  952  whereas the global light component  912   a  is incident on the point  954  indirectly as a result of an interreflection. The scattering of each of the direct light component  912  and the global light component  912   a  at the point  6954  results in a total light signal  914  that is captured by a camera  920  of the ToF module  902 . 
     An implementation of the camera  920  includes a plurality of camera pixels including a camera pixel  920   a , wherein each camera pixel is assumed to be able to observe at most one significant scattering event from the surface  952 . As a result, is assumed that two different light source pixels cannot produce a direct light component along a given camera pixels line of sight.  FIG. 9  illustrates one such camera pixel including its imaging lens  916 , a light sampling array  924 , and a sampler  924 . For example, the sampling array  922  may be a charge coupled device (CCD) array that converts light energy into electrical charge in analog form. The sampler  924  may include an analog-digital converter (ADC) that converts the electrical charge into a digital signal. The combination of the imaging lens  916 , the light sampling array  924 , and the sampler  924  receives and converts the total light signal  914  into a sampled signal  926  that is communicated to a multipath mitigation module  928 . The multipath mitigation module  928  includes an image analyzer  930  that analyzes images captured by the camera  920  to determine various parameters of a direct component and a global component of the total light signal  914 . 
     The modulated source light illuminates the object  950 . The camera  920  acquires the total light  914  scattered by the object  950  in response to the illumination by the modulated source light. Specifically, the camera  920  acquires k+1 images with spatial pattern  934  and modulation frequency K) of the object  650  at the camera  620  to separate the direction component of light P d (i,j) and the global component of light P g (i,j). 
     The light source  910  may include a large number of light source pixels. The illustrated implementation shows one such light source pixel  910   a  generating a light signal that is projected on a surface  952  of an object  950 . The light signal generated by the light source pixel  610   a  may include a direct light component  912  and a global light component  912   a . Specifically, the direct light component  912  is directly incident on a point  954  on the surface  952  whereas the global light component  612   a  is incident on the point  954  indirectly as a result of an interreflection. The scattering of each of the direct light component  912  and the global light component  912   a  at the point  954  results in a total light signal  914  that is captured by a camera  920  of the ToF module  902 . 
     In the present implementation, the multipath mitigation module  928  includes a denoising module  940  and a law-pass filter  944 . The denoising module  940  generates a denoising coefficient ξ(m,n) to increase the signal to noise ratio (SNR). An example, of the denoising module  940  generates the denoising coefficient ξ(m,n) using the equation below: 
     
       
         
           
             
               ξ 
               ⁡ 
               
                 ( 
                 
                   m 
                   , 
                   n 
                 
                 ) 
               
             
             = 
             
               
                 
                   [ 
                   
                     C 
                     - 
                     
                       γ 
                       ⁡ 
                       
                         ( 
                         
                           m 
                           , 
                           n 
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               m 
                               - 
                               
                                 m 
                                 _ 
                               
                             
                             
                               σ 
                               m 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   
                     [ 
                     
                       - 
                       
                         
                           ( 
                           
                             
                               n 
                               - 
                               
                                 n 
                                 _ 
                               
                             
                             
                               σ 
                               n 
                             
                           
                           ) 
                         
                         2 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ∑ 
                   
                     m 
                     = 
                     
                       - 
                       M 
                     
                   
                   M 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       n 
                       = 
                       
                         - 
                         N 
                       
                     
                     N 
                   
                   ⁢ 
                   
                     
                       [ 
                       
                         C 
                         - 
                         
                           γ 
                           ⁡ 
                           
                             ( 
                             
                               m 
                               , 
                               n 
                             
                             ) 
                           
                         
                       
                       ] 
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   m 
                                   - 
                                   
                                     m 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   m 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                     ⁢ 
                     
                       exp 
                       ⁡ 
                       
                         [ 
                         
                           - 
                           
                             
                               ( 
                               
                                 
                                   n 
                                   - 
                                   
                                     n 
                                     _ 
                                   
                                 
                                 
                                   σ 
                                   n 
                                 
                               
                               ) 
                             
                             2 
                           
                         
                         ] 
                       
                     
                   
                 
               
             
           
         
       
     
     Here m (−M, . . . , M) and n (−N, . . . , N) denotes a size of a kernel matrix applied to each pixel locations in each of the k+1 images acquired at operation  1008  disclosed below in  FIG. 10 . Note that m and n do not have to be equal, therefore, the kernel may be rectangular, which allows using patterns with predominant direction. Depending on the size of the kernel and the values of the standard deviations, the denoising coefficient (m,n) can produce a reduction in the spatial frequencies of the image captured by a camera  920  of the ToF module  902 . One assumption behind the implementation disclosed in  FIG. 9  is that the direct component of the light has higher frequencies than the global component because the multipath tends to wash out the high frequencies for the global component. Therefore, the implementation  900  provides the low-pass filter (LPF)  944  that is applied to the image generated by the camera  920 . The LPF  944  may be a gaussian filter, a weighted filter, a standard mean filter, a Gabor wavelengths filter, etc. 
     The effect of the application of the LPF  944  is illustrated by the graph  946 , where the signal output from the denoising module  940  is represented by the green signal whereas he signal after application of the LPF  944  is illustrated by black. 
       FIG. 10  illustrates example operations  1000  for multipath mitigation according to the implementation disclosed in  FIG. 9 . Specifically,  FIG. 10  illustrates example operations  1000  for separating the direct component of the light from the global component that contains the multipath. One or more operations  1000  may be performed by the image analyzer  930  disclosed in  FIG. 9 . Specifically, an operation  1004  modulates a light source (such as the light source  910  of  FIG. 9 ) using the spatial patterns (such as the uniform spatial patterns  934  of  FIG. 9 ) from a pattern data store. In the illustrated implementation, the operation  1004  modulates the light source with the spatial patterns  934  and temporally modulates the light signal with k different frequencies. For example, k may be equal to three. An operation  1006  illuminates an object (such as the object  950  of  FIG. 9 ) using the modulated light source. The modulated light is scattered from the object towards a camera (such as the camera  920  of  FIG. 9 ). An operation  1008  acquires k images where the object is illuminated with the light source being modulated with the spatial pattern  934  and k modulation frequencies. Each of the k images may be in the form of a matrix with each observation of the matrix corresponding to individual pixels of the camera  920 . 
     An operation  1030  generates a denoising coefficient (m,n) to increase the signal to noise ratio (SNR) as discussed above with respect to the denoising module  640 . An LPF, such as the LPF  944 , is applied to the output by an operation  1032 . Subsequently, an operation  1034  determines the real and the complex parameters of the direct and global components of the light using the calculation discussed below: 
     
       
         
           
             Direct 
             ⁢ 
             
                 
             
             ⁢ 
             
               { 
               
                 
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             r 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               K 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               ξ 
                               ⁢ 
                               
                                 { 
                                 
                                   m 
                                   , 
                                   n 
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               r 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             P 
                                             r 
                                             LP 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               k 
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           
                             
                               D 
                               _ 
                             
                             c 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                               , 
                               k 
                             
                             ) 
                           
                         
                         = 
                         
                           
                             ∑ 
                             
                               m 
                               = 
                               
                                 - 
                                 M 
                               
                             
                             M 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 
                                   - 
                                   N 
                                 
                               
                               N 
                             
                             ⁢ 
                             
                               
                                 ξ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 
                                   
                                     
                                       
                                         
                                           - 
                                           
                                             
                                               P 
                                               c 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                         + 
                                       
                                     
                                   
                                   
                                     
                                       
                                         C 
                                         · 
                                         
                                           
                                             P 
                                             c 
                                             LP 
                                           
                                           ⁡ 
                                           
                                             ( 
                                             
                                               m 
                                               , 
                                               n 
                                               , 
                                               k 
                                             
                                             ) 
                                           
                                         
                                       
                                     
                                   
                                 
                                 
                                   C 
                                   - 
                                   
                                     γ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         m 
                                         , 
                                         n 
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
                 ⁢ 
                 
                   
 
                 
                 ⁢ 
                 Global 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   { 
                   
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               r 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 K 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 ξ 
                                 ⁢ 
                                 
                                   { 
                                   
                                     m 
                                     , 
                                     n 
                                   
                                   ) 
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 r 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   K 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               P 
                                               r 
                                               LP 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             
                               
                                 G 
                                 _ 
                               
                               c 
                             
                             ⁡ 
                             
                               ( 
                               
                                 i 
                                 , 
                                 j 
                                 , 
                                 k 
                               
                               ) 
                             
                           
                           = 
                           
                             
                               ∑ 
                               
                                 m 
                                 = 
                                 
                                   - 
                                   M 
                                 
                               
                               M 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   n 
                                   = 
                                   
                                     - 
                                     N 
                                   
                                 
                                 N 
                               
                               ⁢ 
                               
                                 
                                   ξ 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       m 
                                       , 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                   
                                     
                                       
                                         
                                           
                                             - 
                                             
                                               
                                                 P 
                                                 c 
                                               
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   m 
                                                   , 
                                                   n 
                                                   , 
                                                   K 
                                                 
                                                 ) 
                                               
                                             
                                           
                                           + 
                                         
                                       
                                     
                                     
                                       
                                         
                                           
                                             γ 
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                               
                                               ) 
                                             
                                           
                                           · 
                                           
                                             
                                               P 
                                               c 
                                               LP 
                                             
                                             ⁡ 
                                             
                                               ( 
                                               
                                                 m 
                                                 , 
                                                 n 
                                                 , 
                                                 k 
                                               
                                               ) 
                                             
                                           
                                         
                                       
                                     
                                   
                                   
                                     C 
                                     - 
                                     
                                       γ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           m 
                                           , 
                                           n 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     Finally an operation  1036  performs one or more range calculation operations. 
       FIG. 11  illustrates an example system  1200  that may be useful in implementing the described ToF system disclosed herein. The example hardware and operating environment of  FIG. 11  for implementing the described technology includes a computing device, such as a general-purpose computing device in the form of a computer  20 , a mobile telephone, a personal data assistant (PDA), a tablet, smart watch, gaming remote, or other type of computing device. In the implementation of  FIG. 11 , for example, the computer  20  includes a processing unit  21 , a system memory  22 , and a system bus  23  that operatively couples various system components including the system memory to the processing unit  21 . There may be only one or there may be more than one processing unit  21 , such that the processor of a computer  20  comprises a single central-processing unit (CPU), or a plurality of processing units, commonly referred to as a parallel processing environment. The computer  20  may be a conventional computer, a distributed computer, or any other type of computer; the implementations are not so limited. 
     The system bus  23  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a switched fabric, point-to-point connections, and a local bus using any of a variety of bus architectures. The system memory may also be referred to as simply the memory, and includes read-only memory (ROM)  24  and random access memory (RAM)  25 . A basic input/output system (BIOS)  26 , containing the basic routines that help to transfer information between elements within the computer  20 , such as during start-up, is stored in ROM  24 . The computer  20  further includes a hard disk drive  27  for reading from and writing to a hard disk, not shown, a magnetic disk drive  28  for reading from or writing to a removable magnetic disk  29 , and an optical disk drive  30  for reading from or writing to a removable optical disk  31  such as a CD ROM, DVD, or other optical media. 
     The hard disk drive  27 , magnetic disk drive  28 , and optical disk drive  30  are connected to the system bus  23  by a hard disk drive interface  32 , a magnetic disk drive interface  33 , and an optical disk drive interface  34 , respectively. The drives and their associated tangible computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer  20 . It should be appreciated by those skilled in the art that any type of tangible computer-readable media may be used in the example operating environment. 
     A number of program modules may be stored on the hard disk drive  27 , magnetic disk drive  29 , optical disk drive  31 , ROM  24 , or RAM  25 , including an operating system  35 , one or more application programs  36 , other program modules  37 , and program data  38 . A user may generate reminders on the computer  20  through input devices such as a keyboard  40  and pointing device  42 . Other input devices (not shown) may include a microphone (e.g., for voice input), a camera (e.g., for a natural user interface (NUI)), a joystick, a game pad, a satellite dish, a scanner, or the like. These and other input devices are often connected to the processing unit  21  through a serial port interface  46  that is coupled to the system bus  23 , but may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB). A monitor  47  or other type of display device is also connected to the system bus  23  via an interface, such as a video adapter  48 . In addition to the monitor, computers typically include other peripheral output devices (not shown), such as speakers and printers. 
     The computer  20  may operate in a networked environment using logical connections to one or more remote computers, such as remote computer  49 . These logical connections are achieved by a communication device coupled to or a part of the computer  20 ; the implementations are not limited to a particular type of communications device. The remote computer  49  may be another computer, a server, a router, a network PC, a client, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer  20 . The logical connections depicted in  FIG. 6  include a local-area network (LAN)  51  and a wide-area network (WAN)  52 . Such networking environments are commonplace in office networks, enterprise-wide computer networks, intranets and the Internet, which are all types of networks. 
     When used in a LAN-network environment, the computer  20  is connected to the local area network  51  through a network interface or adapter  53 , which is one type of communications device. When used in a WAN-networking environment, the computer  20  typically includes a modem  54 , a network adapter, a type of communications device, or any other type of communications device for establishing communications over the wide area network  52 . The modem  54 , which may be internal or external, is connected to the system bus  23  via the serial port interface  46 . In a networked environment, program engines depicted relative to the personal computer  20 , or portions thereof, may be stored in the remote memory storage device. It is appreciated that the network connections shown are for example and other means of communications devices for establishing a communications link between the computers may be used. 
     In an example implementation, software or firmware instructions for doing multipath mitigation as disclosed in  FIGS. 2 and 4  may be stored in system memory  22  and/or storage devices  29  or  31  and processed by the processing unit  21 . One of more of the patterns  132  and  134  disclosed in  FIG. 1  may be stored in system memory  22  and/or storage devices  29  or  31  as persistent datastores. The computer  20  also includes a mitigation module  1110  that may be used to receive sampled light signals from a camera (such as the camera  120  of  FIG. 1 ) and process the sampled signals to determine the direct component and the global component of the light signal. The mitigation module  1110  may include an image analyzer such as the image analyzer  130  disclosed in  FIG. 1 . 
     A physical hardware system to provide multipath mitigation in a time-of-flight (ToF) system includes a camera configured to acquire two or more images represented by two or more matrices in response to illuminating a target with a light source using a first spatial pattern at two or more different light frequencies and to acquire an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, a denoising module configured to generate a denoising coefficient based on a kernel matrix, and an image analyzer to apply the denoising coefficient to the two or more matrices and the additional matrix and to determine one or more parameters of a direct component of light and a global component of light received at the camera from the target. 
     In one implementation, the first spatial pattern is a uniform spatial pattern. In an alternative implementation, the second spatial pattern is a non-uniform spatial pattern. Yet alternatively, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. In one implementation, the additional image is acquired in response to illuminating the target with the light source using the second spatial pattern at a light frequency similar to one of the two or more different light frequencies used when illuminating the target with the first spatial pattern. 
     In another implementation, the camera is further configured to acquire three or more images represented by three or more matrices in response to illuminating the target with the light source using the first spatial pattern at three or more different light frequencies. Alternatively, the non-uniform spatial pattern is at least one of a dot-pattern, a vertical-line pattern, and a horizontal line pattern. Yet alternatively, the kernel matrix is an m×n matrix with m being different than n. 
     A method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector includes acquiring three or more images represented by three or more matrices in response to illuminating a target with a light source using a first spatial pattern at three or more different modulation frequencies, acquiring an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, generate a denoising coefficient based on a kernel matrix, and applying the denoising coefficient to the two or more matrices and the additional matrix and to determine one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix. 
     In one implementation, the first spatial pattern is a uniform spatial pattern. Alternatively, the second spatial pattern is a non-uniform spatial pattern. Yet alternatively, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. In another implementation, the additional image is acquired in response to illuminating the target with the light source using the second spatial pattern at a light frequency similar to a one of the three or more different light frequencies used when illuminating the target with the first spatial pattern. Alternatively, the non-uniform spatial pattern is at least one of a dot-pattern, a vertical-line pattern, and a horizontal line pattern. Yet alternatively, the kernel matrix is an m×n matrix with m being different than n. 
     A physical article of manufacture including one or more tangible computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process including acquiring two or more images represented by three or more matrices in response to illuminating a target with a light source using a first spatial pattern at two or more different modulation frequencies, acquiring an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, generating a denoising coefficient based on a kernel matrix, and applying the denoising coefficient to the two or more matrices and the additional matrix and to determine one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix. 
     In an alternative implementation, the first spatial pattern is a uniform spatial pattern and the second spatial pattern is a non-uniform spatial pattern. Yet alternatively, the process further includes performing a frequency conversion operation to transform a phase into a frequency K. Yet alternatively, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. Alternatively, the light source illuminates a target using N temporal modulation frequencies with the uniform spatial pattern and N temporal modulation frequencies with the non-uniform spatial pattern. 
     In contrast to tangible computer-readable storage media, intangible computer-readable communication signals may embody computer readable instructions, data structures, program modules or other data resident in a modulated data signal, such as a carrier wave or other signal transport mechanism. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. 
     Some embodiments may comprise an article of manufacture. An article of manufacture may comprise a tangible storage medium to store logic. Examples of a storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of the logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. In one embodiment, for example, an article of manufacture may store executable computer program instructions that, when executed by a computer, cause the computer to perform methods and/or operations in accordance with the described embodiments. The executable computer program instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The executable computer program instructions may be implemented according to a predefined computer language, manner or syntax, for instructing a computer to perform a certain function. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language. 
     The system for secure data onboarding may include a variety of tangible computer-readable storage media and intangible computer-readable communication signals. Tangible computer-readable storage can be embodied by any available media that can be accessed by the ToF system disclosed herein and includes both volatile and non-volatile storage media, removable and non-removable storage media. Tangible computer-readable storage media excludes intangible and transitory communications signals and includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Tangible computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium which can be used to store the desired information and which can be accessed by the ToF system disclosed herein. In contrast to tangible computer-readable storage media, intangible computer-readable communication signals may embody computer readable instructions, data structures, program modules or other data resident in a modulated data signal, such as a carrier wave or other signal transport mechanism. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include signals moving through wired media such as a wired network or direct-wired connection, and signals moving through wireless media such as acoustic, RF, infrared and other wireless media. 
     The ToF system including multipath mitigation, as disclosed herein provides solution to a technological problem necessitated by the requirements of highly accurate depth perception information. Specifically, the multipath mitigation system disclosed herein provides an unconventional technical solution to this technological problem by providing a system and method that allows for efficient determination of parameters of direct component in a light signal received at a pixel of a camera and parameters of global component in the light signal received at a pixel of a camera by modulating a light source with pattern to generate patterned source light. 
     A physical hardware system to provide multipath mitigation in a time-of-flight (ToF) system includes a camera configured to acquire a first image represented by a first matrix in response to illuminating a target with a light source using a first spatial pattern and to acquire a second image represented by a second matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern and an image analyzer to analyze the first matrix and the second matrix to determine one or more parameters of a direct component of light and a global component of light received at the camera from the target. 
     In one implementation of the physical hardware system the first spatial pattern is a uniform spatial pattern. In an alternative implementation of the physical hardware system, second spatial pattern is a non-uniform spatial pattern. In yet another implementation, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. In one implementation, the light source illuminates a target using N temporal modulation frequencies with the first spatial pattern and one temporal modulation frequency with the second spatial pattern. Alternatively, the light source illuminates a target using N temporal modulation frequencies with the first spatial pattern and N temporal modulation frequency with the second spatial pattern. Yet alternatively, the non-uniform spatial pattern is at least one of a dot-pattern, a vertical-line pattern, and a horizontal line pattern. 
     A method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector, includes acquiring a first image represented by a first matrix in response to illuminating a target with a light source using a first spatial pattern, acquiring a second image represented by a second matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, and determining one or more parameters of the direct component of light and the global component of light based on analysis of the first matrix and the second matrix. In one implementation of the method, the first spatial pattern is a uniform spatial pattern. In another implementation, the second spatial pattern is a non-uniform spatial pattern. 
     Alternatively, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. Yet alternatively, the light source illuminates a target using N temporal modulation frequencies with the first spatial pattern and one temporal modulation frequency with the second spatial pattern. In an alternative implementation, the non-uniform spatial pattern is one of a dot-pattern, a vertical line pattern, and a horizontal line pattern. 
     A physical article of manufacture including one or more tangible computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process includes acquiring a first image represented by a first matrix in response to illuminating a target with a light source using a first spatial pattern, acquiring a second image represented by a second matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, and determining one or more parameters of the direct component of light and the global component of light based on analysis of the first matrix and the second matrix 
     In one implementation, the first spatial pattern is a uniform spatial pattern and the second spatial pattern is a non-uniform spatial pattern. Alternatively, the light source illuminates a target using N temporal modulation frequencies with the uniform spatial pattern and one temporal modulation frequency with the non-uniform spatial pattern. Yet alternatively, each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern. In one implementation, the light source illuminates a target using N temporal modulation frequencies with the uniform spatial pattern and N temporal modulation frequencies with the non-uniform spatial pattern. 
     The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the invention. Since many implementations of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended. Furthermore, structural features of the different embodiments may be combined in yet another implementation without departing from the recited claims.