Abstract:
A rangefinder has a depth camera and a computer. The depth camera is positioned to receive electromagnetic energy from a scene. Imaging optics and a sensor in the camera generate an input image based on electromagnetic energy. The computer receives the input image, calculates intermediate images by applying differentiation operators along different axes to the input image, creates a merit function map, and computes a depth map from the merit function map.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit of the Apr. 18, 2013 priority date of U.S. Provisional Application 61/854,149, the contents of which are herein incorporated by reference. 
     
    
     FIELD OF INVENTION 
       [0002]    The present invention relates to three-dimensional imaging systems, and particularly to depth-from-defocus optical apparatus. 
       BACKGROUND OF THE INVENTION 
       [0003]    Various optical systems have been developed for estimating the distance to and shape of objects. These are sometimes referred to as rangefinders, range sensors, and depth map cameras. Here, they will be referred to as “rangefinders.” A rangefinder estimates distances to one or more regions within a scene. The scene can contain one or more objects. 
         [0004]    Some rangefinders estimate depth by analyzing the blur that is present in at least one image of the scene. Examples of such rangefinders are described in S. K. Nayar, M. Watanabe, and M. Noguchi, “Real-Time Focus Range Sensor,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 12, 1186-1198 (December 1996), S. Bae and F. Durand, “Defocus Magnification,” Computer Graphics Forum, 26, 571-579 (2007), G. C. Birch, J. S. Tyo, and J. Schwiegerling, “3D Astigmatic Depth Sensing Camera,” Proc. SPIE 8129, Novel Optical Systems Design and Optimization XIV, 812903 (Sep. 9, 2011), and in J. H. Elder and S. W. Zucker, “Local Scale Control for Edge Detection and Blur Estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 7, 699-716 (July 1998) 
         [0005]    Existing rangefinders have various disadvantages. For example, time-of-flight, structured illumination, or existing depth-from-defocus devices, are typically undesirably sensitive to ambient light and object color or reflectivity, provide infrequent depth data, are too expensive for certain applications, or have insufficient depth accuracy. 
         [0006]    Thus, to obtain range information of the scene in a manner that cures at least some of these disadvantages, another approach to obtaining depth information must be used. 
       SUMMARY 
       [0007]    The invention is based on the recognition that astigmatic optical imaging and certain software algorithms are adaptable to measurement of distances to regions within a scene. 
         [0008]    In one aspect, the invention features an apparatus comprising a rangefinder. The rangefinder has a depth camera and a computer. The depth camera is positioned to receive electromagnetic energy from a scene. The depth camera comprises imaging optics and a sensor configured to generate an input image based on the electromagnetic energy. The computer is computer programmed to receive the input image, to calculate intermediate images H(x,y) and V(x,y) by applying the partial differentiation operators along 2 orthogonal directions (for example, X direction and Y direction) x- and y-differentiation operators to the input image, to create a merit function map; and to compute a depth map from the merit function map. 
         [0009]    In some embodiments, the imaging optics are astigmatic. 
         [0010]    In another aspect, the apparatus features a rangefinder and a computer. The rangefinder has a depth camera that is positioned to receive electromagnetic energy from a scene. The depth camera includes imaging optics and a sensor configured to generate an input image based on the electromagnetic energy. The computer is programmed to receive the input image, to calculate first and second intermediate images along corresponding first and second orthogonal axes that differ, to create a merit function map based on the intermediate images, and to compute a depth map from the merit function map. 
         [0011]    In some embodiments, the first and second axes may be orthogonal. Among these are embodiments in which the first and second axes are horizontal and vertical axes. 
         [0012]    In some embodiments, the imaging optics are astigmatic. In others, the imaging optics are stigmatic. 
         [0013]    Among the embodiments are those in which the computer is programmed to calculate the intermediate images by applying, to the input image, corresponding differentiation operators along the first and second axes that differ, and those in which the computer is programmed to calculate the intermediate images by anisotropic filtering of the input image in the frequency domain. 
         [0014]    In some embodiments, the computer is programmed to create the merit function map in a way that includes blurring the intermediate images. 
         [0015]    In other embodiments, the computer is programmed to create the merit function in a way that includes evaluation of a function of the intermediate images on a pixel-by-pixel basis. 
         [0016]    Embodiments also include those in which the computer is programmed to select the first and second axes to be horizontal and vertical axes, those in which the computer is programmed to select the first and second axes to be orthogonal axes, and those in which the computer is programmed to select the first and second axes to be non-orthogonal, but nevertheless different, axes. 
         [0017]    In another aspect, the invention features an apparatus having a computer and means for relaying information indicative of a scene to the computer. The computer is programmed to receive an input image from the means for relaying information, to calculate, from the input image, first and second intermediate images along first and second axes that differ from each other, to create a merit function map based on the images, and to compute a depth map from the merit function map. 
         [0018]    In yet another aspect, the invention features a method of generating a depth map. Such a method includes receiving an input image indicative of a scene, calculating a first intermediate image from the input image, calculating a second intermediate image from the input image, creating a merit function map based on the intermediate horizontal and vertical images, and computing the depth map from the merit function map. In this embodiment, the first and second intermediate images are along first and second axes that differ from each other. 
         [0019]    In some practices, receiving the image comprises receiving an astigmatic image. In others, it includes receiving a stigmatic image. 
         [0020]    Among the practices of the invention are those in which calculating an intermediate horizontal image from the input image comprises applying a horizontal differentiation operator to the input image, and calculating an intermediate vertical image from the input image comprises applying a vertical differentiation operator to the input image. 
         [0021]    In other practices, calculating first and second intermediate images from the input image includes single-coordinate differentiation of the input image in the spatial domain. 
         [0022]    In some practices, calculating a first intermediate image from the input image comprises anisotropic filtering of the input image in the frequency domain, and calculating a second intermediate image from the input image comprises anisotropic filtering in the frequency domain 
         [0023]    Among the practices of the invention are those that also include creating the merit function map comprises blurring the intermediate images. 
         [0024]    In yet other practices of the invention, creating the merit function comprises evaluating a function of the intermediate images on a pixel-by-pixel basis. 
         [0025]    Also among the practices of the invention are those that include selected the first and second axes to be horizontal and vertical axes, selecting them to be orthogonal axes, and selecting them to be non-orthogonal axes. 
         [0026]    These and other features of the invention will be apparent from the following detailed description and the accompanying drawings, in which: 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0027]      FIG. 1  shows a typical embodiment of an astigmatic rangefinder; 
           [0028]      FIG. 2  shows scenes and depth maps; 
           [0029]      FIG. 3  shows imaging systems with stigmatic and astigmatic focus; 
           [0030]      FIG. 4  shows the imaging optics; 
           [0031]      FIG. 5  shows a first example scene; 
           [0032]      FIG. 6  shows the scene as imaged by an astigmatic depth camera; 
           [0033]      FIG. 7  shows the first example depth map; 
           [0034]      FIG. 8  shows a second example scene as imaged by the astigmatic depth camera; 
           [0035]      FIG. 9  shows accentuated vertical and horizontal image components; 
           [0036]      FIG. 10  shows the second example depth map; 
           [0037]      FIG. 11  shows “bublik,” an isotropic pre-filtering kernel; 
           [0038]      FIG. 12  shows the isotropic filters “gsf” and “gtf”; 
           [0039]      FIG. 13  shows combined filters; and 
           [0040]      FIGS. 14A-E  show MATLAB code for implementing the method described herein. 
       
    
    
     DETAILED DESCRIPTION 
       [0041]    Referring to  FIG. 1 , in a typical embodiment of the astigmatic rangefinder, a rangefinder  10  images a scene  5  to produce at least one depth map  15 . The rangefinder  10  has a depth camera  20  and a processing system  25 . The depth camera  20  has imaging optics  30  and a sensor  35 . 
         [0042]    The depth camera  20  produces image data  24  (alternatively, an “input image”) that is made available to the processing system  25 . 
         [0043]      FIG. 2  illustrates scenes and depth maps as generated by an existing rangefinder. The top row shows an example of a scene  5   a  of liquid being poured out of a container and the corresponding depth map  15   a.  Usually, a depth maps is a two-dimensional scalar array in which each scalar corresponds to a distance. For example, a value near 0 indicates “near” and a value near 255 indicates “far.” It is convenient to depict depth maps graphically, as shown by the exemplary depth map  15   a.  In this case, “near” is rendered in white, intermediate distances are rendered in gray, and “far” is rendered in black. 
         [0044]    An alternative representation of a depth map  15  is illustrated in the bottom row of  FIG. 2 . In this representation, a depth map  15   b  is rendered as a three-dimensional surface plot that can be computationally rotated to inspect the estimated depths of a corresponding scene  5   b.    
         [0045]    There are many ways to decouple the time and space aspects of depth map estimation.  FIG. 2  shows two-dimensional depth maps in which the input “snapshot” was taken approximately instantaneously. Other types of rangefinders create a one-dimensional estimate of distances, or assemble a two-dimensional depth map from a series of one-dimensional range estimates that are scanned in time and space. Alternatively, a point-scanning rangefinder estimates depth measurements along a vector directed at a single point, which amounts to a zero-dimensional measurement. 
         [0046]    Imaging systems with stigmatic and astigmatic focusing are illustrated in  FIG. 3 , with the optical axis of the system being the z-axis  40  and with an x-axis and a y-axis orthogonal to this optical axis. 
         [0047]    A typical imaging system, for example a camera, is designed to be as stigmatic as practical. In such a camera, there is ideally one location or surface of best focus  41 . In such a case, the image of a single point is approximately a disc. The point-spread function (PSF) of such a single-focus imaging system has an aspect ratio of 1. For completeness, it is noted that the radius of the PSF in such a well-corrected system is primarily a function of the distance from the system to the object point, and to a first approximation the x, y, and z coordinates of the point do not result in a significant deviation from a 1:1 aspect ratio of the object point&#39;s PSF. To more complete approximations, the PSF is indeed a function of these variables, but the functional dependence is difficult to discern. 
         [0048]    An important aspect of this invention is that the depth camera  20  is astigmatic, and therefore has multiple foci. 
         [0049]    Referring now to the bottom of  FIG. 3 , the rangefinder  10  has a horizontal focus (or so-called XZ focus) at a first location  45  and a vertical (YZ) focus at a second location  50 . Another way to say this is that the depth camera produces imagery with an elliptical point spread function (PSF) whose eccentricity ε(z)=a(z)/b(z) varies with distance to the point being imaged. A point situated on a surface with a particular distance from the system will thus produce an image with an eccentricity that is different from that of the image formed of a point situated on a surface that is at a different distance from the system. In addition, as is the case in in the stigmatic system, the scale of those images will vary with the point&#39;s distance from the system. 
         [0050]    In an astigmatic system, information about the distance to the object is contained in the eccentricity of the PSF rather than primarily from its size, as is the case in a stigmatic system. This property of an astigmatic system provides improved robustness in situations relating to object brightness, illuminance, color, and texture. 
         [0051]    In other versions of the system, the foci can have orientations other than horizontal and vertical. It is also possible to use optical components that employ nonstandard manipulations of the light to achieve any of a wide range of phase modulations, e.g. vortices, non-perpendicular orientations, etc. 
         [0052]      FIG. 4  illustrates the imaging optics  30 . The sensor  35 , imaging optics  30 , and other elements are selected using well-known optical systems engineering methods. 
         [0053]    In a particular embodiment, the depth camera  20  is based on a Sony DSLR-A580. The depth camera  20  has imaging optics  30  that include a combination of an objective  55  (Sony SAL 50M28: macro objective, F=50 mm, f/#=2.8) and an astigmatic attachment  60 . The astigmatic attachment  60  includes a step-up ring  65  (Marumi Stepping ring, 55 mm-72 mm), an opto-mechanical mount  75 , and a cylindrical lens  70  (KangMing eyeglass lens, +0.00 S, +4.00 C, diameter 65 mm). The variable S is the spherical optical power of the cylindrical lens  70  in diopters, and C is the cylindrical optical power of the cylindrical lens  70  in diopters. Therefore the cylindrical lens  70  used in this embodiment is a purely cylindrical lens with a focal distance of 250 mm. 
         [0054]    The optical components are held in position using standard techniques. In this case, the objective  55  is mounted to the step-up ring  65 , which is in turn attached to the cylindrical lens  70  with the opto-mechanical mount  75 . In the experimental system, the distance between the face of the objective  55  and the cylindrical lens  70  is approximately 10 mm and may be adjusted ±1.5 mm. The particular choice of these distances and tolerances is not limiting. The values are provided for reference. 
         [0055]    The depth camera  20  begins generating the estimated depth map  15  by acquiring an input image  80  of the scene  5 . This input image is denoted Io(x,y) and has horizontal and vertical dimensions of M by N pixels. Typically M=640 and N=480. In a more spatially-detailed analysis, (M, N)=(1200, 960). Other values of M and N can be used depending on the application. 
         [0056]    After acquiring the input image  80 , there is an optional step of pre-processing the image. This is carried out by computing a two-dimensional Fourier transfer of the image, Io(x,y). The result of this is a Fourier spectrum of the input image, S Io (ξ, η). 
         [0057]    This is followed by an optional step of choosing an isotropic filter for removing low-frequency components, with a cut-off frequency near zero, and high frequency components, which are caused by picture noise. A suitable isotropic filter, “bublik,” is described in more detail below. One purpose of the filter is to remove background illumination variations. Typically, the filter is cylindrically symmetric in a two-dimensional frequency domain, as depicted. Referring to  FIG. 12 , the low and high cut-off frequencies of the filter are tuned by varying filter constants 
         [0058]    After having chosen the filter, there is the optional step of generating a filtered two-dimensional image spatial Fourier spectrum S Ip (ξ, η). This can be carried out by applying “bublik” to the unfiltered spectrum S Io (ξ, η) by writing “bublik” and S Io (ξ, η) as M×N conformal matrices of scalar values and carrying out an element-by-element multiplication of these two matrices. Applying the filter “bublik” to S Io (ξ, η) thus amounts to the multiplication of each matrix element of the picture matrix at the corresponding matrix element of the filter. 
         [0059]    The next step is to extract separate images with vertical and horizontal spatial frequencies, V(x, y) and H(x, y) respectively. There are two ways to do this. One is by anisotropic filtering in the frequency domain. The other, which is preferred, is by single-coordinate differentiation in the spatial domain. 
         [0060]    Anisotropic filtering in the frequency domain is carried out by first filtering S Ip (ξ, η) in the frequency domain, generate intermediate images H(x,y) and V(x,y), and proceeding to distance map forming step described in more detail below. 
         [0061]    To filter S Ip (ξ, η) in the frequency domain, one begins by choosing the horizontal and proper vertical anisotropic filter. A suitable filter looks like a vertical or horizontal stripe, such as a Gauss Stripe Filter (“GSF”) or oblong ellipse, such as a Gauss Tenon Filter (“GTE”). Examples are shown in  FIG. 12 . 
         [0062]    Applying the filter to S Ip (ξ, η) results in two filtered two-dimensional spatial spectra: a horizontal spectrum S H (ξ, η) and a vertical spectrum S V (ν, η). The horizontal spectrum, S H (ξ, η), contains mostly horizontal spatial frequencies. The vertical spectrum, S V (ξ, η), contains mostly vertical frequencies. The inverse Fourier transforms of the horizontal and vertical spectra yield horizontal and vertical intermediate pictures H(x,y) and V(x,y) respectively. 
         [0063]    Once the intermediate pictures are available, it is possible to proceed to the distance map forming step. 
         [0064]    The alternative method for arriving at the distance map forming step is to carry out single-coordinate differentiation in the spatial domain. 
         [0065]    This procedure begins by applying a two-dimensional inverse Fourier transform to the pre-filtered spectrum, S Ip (ξ, η). This results in a pre-filtered image Ip(x,y). 
         [0066]    Next, a one-dimensional X differentiation operator is applied to the pre-filtered image Ip(x,y) or directly to the input image Io(x,y), if no image pre-processing was carried out. The X differentiation operator is the pixel-by-pixel difference of 2 adjacent columns. The result is H(x,y), the horizontal intermediate picture. 
         [0067]    An analogous procedure is carried out to obtain the vertical intermediate picture V(x, y). A one-dimensional Y differentiation operator is applied to the pre-filtered image Ip(x,y). The Y differentiation operator is the pixel-by-pixel difference of 2 adjacent rows. The result is the vertical intermediate picture V(x,y). 
         [0068]    Once the intermediate pictures are available, it is possible to proceed to the distance map forming step. 
         [0069]    Combined filters that overlap the isotropic pre-filtering and the extracting of intermediate pictures H(x,y) and V(x,y) may be used. One of the possible combined filters is presented in the  FIG. 13 . 
         [0070]    The next step in either case is to create a merit function map. This includes the optional step of blurring the horizontal and vertical intermediate images H(x,y) and V(x,y) to obtain blurred horizontal and vertical intermediate images H B (x,y) and V B (x,y). Image blurring is carried out by convolving the relevant function with a Gaussian or cylindrical kernel. 
         [0071]    A merit-function-forming algorithm is then selected to create the merit function map on a pixel-by-pixel basis. Examples include a differencing algorithm that evaluates a difference between the first and second intermediate images, a normalized difference algorithm in which the difference between the first and second intermediate images is normalized by their sum, a ratio algorithm in which the ratio of the norms of the first and second intermediate images is evaluated, a logarithmic ratio algorithm in which the logarithm of the ratio of the norms of the first and second intermediate images is evaluated, and a vector sum algorithm, which is analogous to a distance formula, which involves evaluating a square root of the sum of squares of the norms of the ratio of the norms of the first and second intermediate images. Other examples include a complex-valued merit function in which the first and second intermediate images are real and imaginary parts of a complex number, and a combined merit function created by a weighted ratio of first and second images. These exemplary merit function forming algorithms are summarized below, where “MFM” stands for “merit function map”: 
         [0000]      Difference: MFM= H   B ( x,y )− V   B ( x,y )
 
         [0000]      Normalized difference: MFM=( H   B ( x,y )− V   B ( x,y ))/( H   B ( x,y )+ V   B ( x,y ))
 
         [0000]      Ratio: MFM=| H   B ( x,y )|/| V   B ( x,y )| 
         [0000]      Logarithmic ratio: MFM=log n   [|H   B ( x,y )|/| V   B ( x,y )|] 
         [0000]      Vector sum: MFM=[|H B ( x,y )| 2   +|V   B ( x,y ) 2 |] 0.5    
         [0000]      Combined: MFM=[ xV   B ( x,y )/ yH   B ( x,y )] 0.5    
         [0000]      Complex: MFM= H   B ( x,y )+ iV   B ( x,y ), where  i =(−1) 0.5  
 
         [0072]    Once the merit function map is evaluated, it is useful to inspect it. This can be carried out by transforming the merit function map into a two-dimensional image using well-known gray scale or pseudo-color visualization methods. 
         [0073]    The final step is computation of the depth map  15 . This is carried out by converting the relative distance estimates of the preceding step into calibrated, absolute estimates, which can the be portrayed in the depth map  15 . Using well-known methods of calibration and for example look-up tables, the depth map  15  is based on the known dependence of merit function map values and distance values on the depth-map versus merit function map calibration table. 
         [0074]      FIG. 14A-E  shows these steps represented in the programming language MATLAB. 
         [0075]    A first example scene  5 , illustrated in  FIG. 5 , shows a group of test targets at various depths. The scene  5 , as imaged by the depth camera  20 , is an input image  24  illustrated in  FIG. 6 . The impact of the astigmatic optics is noticeable as pronounced horizontal or vertical image features depending on the target depth. The first example depth map  15  is shown in  FIG. 7 , in which the distance values correspond to different numbers, and plotted as an image with pseudocolor. 
         [0076]    A second example scene  5  is imaged by the depth camera  20  as a second input image  24  as depicted in  FIG. 8 . The intermediate VB(x,y) and HB(x,y) arrays are obtained by single-coordinate differentiation in spatial domain (Type B) and depicted in  FIG. 9 . The resulting estimated depth map  15  is illustrated in  FIG. 10 , using the merit function maps forming algorithm following Step  4 B(i) and pseudo-color visualization. 
         [0077]    The scene  5  has three grids: (left) 137 cm, (center) 237 cm, (right) 37 cm. 
         [0078]    The software can use calibration data in order to better associate depth estimates with true distances. For example, a lookup table can map estimated distance values to actual distance values. 
         [0079]    The imaging optics  30  can be altered in various ways to change the characteristics of the input image  24  or to change the cost of the system. 
         [0080]    For example, a fixed or adjustable iris can be added to modify the brightness and blur characteristics, a lens array or phase and/or amplitude-modulating optic (such as a transparency, spatial light modulator, or other well-known component) can be added to provide further depth-coding characteristics, or the relative angles and/or focusing power(s) of the imaging optics  30  can be adjusted to provide further depth-coding characteristics. 
         [0081]    An active or passive illumination system can be added to the system, such as a point-cloud projector, a time-varying line projector, or other structured light device. The wavelength of the illumination can be visible, infrared, or other wavelength. The projected light would provide additional analysis points that can be inspected by the algorithm to produce a better depth estimate. 
         [0082]    The merit function maps can be modified by various coefficients, such as a constant of proportionality, various mathematical functions, or other parameters well-known to those working in the field of machine vision. 
         [0083]    A calibration step then lets the software output absolute (rather than relative) distance estimates in the depth map. 
         [0084]    In the example described in detail above, the first and second intermediate images are in the horizontal and vertical directions. More generally, the intermediate images can be along any two different axes, whether orthogonal or otherwise.