Depth from defocus calibration

A device and methods are provided for calculating depth estimation for a digital imaging device are disclosed and claimed. In one embodiment, a method includes detecting a first image associated with a first focus parameter, detecting a second image associated with a second focus parameter, calculating a statistical representation of a region of interest in the first and second images, and determining a ratio for the region of interest based on the statistical representation. The method may further include determining one or more focus characteristics using a memory table based on the determined ratio for the region of interest, and calculating a focus depth for capture of image data based on the determined one or more focus characteristics associated with the memory table.

FIELD OF THE INVENTION

The present invention relates in general to methods for focusing an imaging device and in particular to calibration of an imaging device for estimating focus depth for a scene based on depth from defocus.

BACKGROUND

Many imaging devices employ automatic focus (AF) to allow for automatic adjustment of an imaging device to capture image data. For example, one conventional type of AF is based on depth from focus (DFF). Conventional methods of DFF typically capture image data for each focal position of a scene. The sharpness of each focal position may be then analyzed to determine the sharpest focal position, that is the focal position with the highest sharpness metric. As a result of the sharpness determination, the imaging device may automatically set the focus. This approach, however, requires capturing a plurality of images and results in a period of delay during automatic focus. In many instances, the processing delays of DFF do not suit many users of imaging devices, particularly portable imaging device such as digital cameras.

Another approach to automatic focus involves depth from defocus (DFD). Conventional methods and devices employing DFD are typically limited to stationary imaging devices. Further, these methods generally require modeling of blur information in order to determine automatic focus. As a result, parameters of the imaging device must be determined for each zoom and aperture state to allow for AF. These determinations are complicated and result in processing delays which do not suit many users of imaging devices. Accordingly, there is a need in the art for improved automatic focus which overcomes one or more of the aforementioned drawbacks.

BRIEF SUMMARY OF THE INVENTION

Disclosed and claimed herein are a device and methods for calculating depth estimation for a digital imaging device. In one embodiment, a method includes detecting a first image, by the digital imaging device, associated with a first focus parameter, detecting a second image, by the digital imaging device, associated with a second focus parameter, calculating a statistical representation of a region of interest in the first and second images, and determining a ratio for the region of interest based on the statistical representation. The method may further include determining one or more focus characteristics using a memory table based on the determined ratio for the region of interest, and calculating a focus depth, by the digital imaging device, for capture of image data based on the determined one or more focus characteristics associated with the memory table.

Other aspects, features, and techniques of the invention will be apparent to one skilled in the relevant art in view of the following detailed description of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Overview and Terminology

One aspect of the present invention relates to determining focus parameters for capturing image data. In one embodiment, a process is provided for estimating focus depth based on depth from de-focus (DFD). DFD employing statistical analysis and look-up-tables (LUT) as described herein provides advantages over conventional methods of focusing imaging devices, including faster and improved automatic focusing. In one embodiment, a process may include statistical calibration to provide a direct correlation between the amount of object blur and correct object position or object distance. This technique allows for faster automatic focus using DFD. Another advantage of the processes described herein, may be that determination of some complex, and possibly inaccurate, camera parameters to model each zoom and aperture state may be avoided. Further, the direct use of a look-up-table (LUT) may be employed in the spatial domain to greatly simplify the implementation of depth estimation procedure. Determination of depth estimation may further include a calibration procedure for estimating a LUT for all possible and/or required pairs of focus positions for a scene. In addition, a confidence LUT may be calculated to provide a confidence level for the estimated depth, such as a confidence table based on confidence levels and an estimated depth probability distribution for the confidence levels.

In another embodiment, an imaging device is provided to include automatic focusing via depth estimation for a scene according to one or more of the processes described herein. The imaging device, such as a digital camera, may be configured to determine one or more focus parameters during automatic focus.

Reference throughout this document to “one embodiment”, “certain embodiments”, “an embodiment” or similar term means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of such phrases in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner on one or more embodiments without limitation.

In accordance with the practices of persons skilled in the art of computer programming, the invention is described below with reference to operations that can be performed by a computer system or a like electronic system. Such operations are sometimes referred to as being computer-executed. It will be appreciated that operations that are symbolically represented include the manipulation by a processor, such as a central processing unit, of electrical signals representing data bits and the maintenance of data bits at memory locations, such as in system memory, as well as other processing of signals. The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties corresponding to the data bits

When implemented in software, the elements of the invention are essentially the code segments to perform the necessary tasks. The code segments can be stored in a “processor storage medium,” which includes any medium that can store information. Examples of the processor storage medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory or other non-volatile memory, a floppy diskette, a CD-ROM, an optical disk, a hard disk, etc.

Exemplary Embodiments

Referring now to the figures,FIG. 1depicts a graphical representation of depth from defocus (DFD) according to one embodiment of the invention. In one embodiment, DFD may be employed to estimate the depth of one or more objects of a scene by an imaging device. A scene, for example, may relate to the detection area of an imaging device. DFD may allow for the amount of blur of an object to be directly correlated to depth (e.g., distance from the camera). As depicted inFIG. 1, imaging device100and lens surface105may be focused on a depth, shown as110. DFD may be employed to estimate the depth of objects in the field of view of imaging device100based on object depth and focus blur. For example, as will be described herein DFD may be employed to estimate depths, shown as1151-nof objects1201-n.

Blur can be modeled as a convolution between an unblurred object and a point spread function. As discussed in U.S. patent application Ser. No. 12/409,416, which is hereby fully incorporated by reference, iterative automatic focusing may be provided for digital imaging devices.

Blur ratio may be directly connected to object distance. By way of example, blur width of a point source can be expressed as:

where f is the focal length, u is the object distance from the lens, v is the distance between the lens and image detector, d is the aperture diameter and k is a camera constant that depends on camera optics.

According to one embodiment of the invention, DFD may be provided which includes statistical methods for estimating the connection between object depth and blur ratio. In certain embodiments, DFD may be provided without explicit consideration of camera model parameters or blur formation parameters. As a result, since a single blurred image does not allow for a point of reference with respect to blur and blur estimation from a single image is complicated and typically inaccurate, two blur images may be detected with different focus parameters in order to estimate a generalized blur ratio of the object in the two images.

In accordance with the blur width formulation as described above, for two images (of the same object(s)) with different distances, v1 and v2) the ratio w1/w2 is a function of u, based on the assumption that k, d, and f are constant. However, in practical imaging systems, blur characteristics may vary from the formulation and input may not be a point source. Therefore, a statistical approach is provided for implementation of a DFD concept.

FIG. 2depicts a simplified block diagram of an imaging device, and more particularly a digital camera, according to one or more embodiments of the invention. Digital camera200includes image sensor210. Sensor210may relate to a combination of an optical lens and light detection circuitry (e.g., CMOS integrated circuit, etc.). Image data detected by sensor210may be provided to processor205, including image data detected for estimating depth based on DFD, such as preview images.

Processor205may be implemented using one of an integrated circuit microprocessor, microcontroller, digital signal processor and processor in general. Processor205may be configured to process received image data based on, for example, specific image processing algorithms stored in memory215in the form of processor-executable instructions. Processor205may further be configured to control exposure parameters, such as exposure periods, focus depth, etc. In certain embodiments, processor205may be configured to control one or more components of imaging device200including image sensor210to control AF. For example, processor205may be configured to adjust the focus depth of a lens base on an estimated depth determined using DFD.

Processed image data may be stored in memory215and/or provided to display220for viewing. It should be appreciated that memory215may relate to any combination of different memory storage devices, such as a hard drive, random access memory (RAM), read only memory (ROM), flash memory, or any other type of volatile and/or nonvolatile memory. It should further be appreciated that memory215may be implemented as multiple or discrete memories for storing processed image data, as well as the processor-executable instructions for processing the captured image data. Further, memory215may include removable memory, such as flash memory, for storage of image data.

Display220may relate to a liquid crystal display (LCD) incorporated into digital camera200for display of captured image data. Alternatively, it should be appreciated that an external display device may be coupled to digital camera200for display. AlthoughFIG. 2has been described above with respect to a digital camera, it should be appreciated that the device may relate to other devices, such as imaging devices in general. Further, it should also be appreciated that the device ofFIG. 2may relate to, or be included in, other devices, such as a mobile communication device and portable communication devices in general.

Referring now toFIG. 3a process is depicted for depth estimation by a digital imaging device according to one or more embodiments of the invention. Process300may be performed by one or more elements of the device ofFIG. 2(e.g., digital camera200). According to another embodiment, process300may employed for automatic focus of an imaging device.

Process300is depicted for determining a focus depth for a scene by an imaging device. Process300may be initiated by detecting a first image of a scene at block305and detecting a second image of the scene at block310. According to one embodiment, the image data detected at blocks305and310may be detected based on different focus parameters.

In one embodiment, image data of the first and second images may be aligned. Image alignment may be applied to one of the images. In one embodiment, image alignment may be based on a transformation between image data of the scene. For example, a transform may be determined for a first image I1(x,y) and I2(x,y) wherein the transform may be characterized as:
I1(x,y)≈I2(fx(x,y),fy(x,y))

Accordingly, a transformation of image data for the first image may be characterized as
I1(x,y)≈I2(x−tx,y−ty)

It should also be appreciated that other methods may be employed for image alignment (e.g., image registration). By way of example, process300may include determining motion vectors between blocks, or feature points, of the two images for calculation of a global transformation. Similarly, it may also be appreciated that a RANSAC method may be employed for image alignment in certain embodiments. Based on the global transformation employed, the transformation may be applied to one of the images for correction based on one or more interpolation schemes (e.g., bilinear, bicubic, etc.). Process300may include application of an alignment transformation to regions of image data as will be further discussed in more detail with reference toFIG. 7below.

Process300may then continue by calculating a statistical representation for one or more regions of interest in the image data, (e.g., statistics extraction) at block315. In one embodiment, statistical representation may be calculated for each region of interest. A statistical representation may relate to statistical analysis of one or more regions of the image data. By way of example, statistical representation of an image region f(x,y) may relate to:
S=RankFilter[|(f*h)(x,y)|,L],

where h(x,y) is a filter (e.g. h=[−1 1]), * denotes the convolution operation, and RankFilter[f(x,y),L] represents ordering the values of f(x,y) in a decreasing order and taking the Lth value.

According to another example, statistical representation of an image region may be determined by:
S=Σx,yx,y(|(f*h)(x,y)|^2)

wherein the values of a filtered image may be summed.

Calculating a statistical representation at block315may further include calculating generalized ratios for statistics which belong to the same region of interest. For example, a generalized ratio of two statistics S1, S2may be characterized as: R=tg−1(S1/S2). In another embodiment, the generalized ratio may be characterized as: R=S1/S2.

Process300may then determine parameters based on a memory table, such as look-up-table (LUT), at block320. In one embodiment, determining parameters may include retrieving LUT values which correspond to different focus parameters. In one embodiment, generalized ratios may be employed for LUT input, wherein the LUT output may be employed to determine one or more depths for each region of interest. Based on values of the LUT and statistical representation, a focus depth may be calculated for automatic focus of an imaging device for a scene at block325.

In certain embodiments, as will be described in more detail with respect toFIG. 10, more than a single depth may be estimated for a region of interest. Further, process300may include determining a confidence level for each estimated depth. For example, depth estimates provided at block325may be associated with output of an additional confidence LUT. The confidence LUT may be created during a calibration procedure. In one embodiment, confidence levels may be calculated based on the LUT values for depth.

In certain embodiments, process300may further provide a validity indication to determine whether the calculated statistics are above image noise. If the calculated statistics in one of the images (in some specified region of interests) is below the noise level then, this region of interest may be declared as non-valid. The validity indication may be test-image content dependent, and further may be an optional output of process300. According to another embodiment, the DFD depth estimation process as described above inFIG. 3may be based on a predetermined LUT for a scene. Similarly, calibration of the DFD may be determined for all possible and/or required pairs of focus positions. According to another embodiment, it should be appreciated that depth estimation as calculated inFIG. 3may be applied to still and video imaging devices.

Referring now toFIG. 4, a process is depicted for calibration of a LUT of an imaging device according to one embodiment. Process400may be initiated by acquiring calibration images at block405. For example, in one embodiment an array of calibration chart images (e.g., V×U) may be acquired for V camera focus positions and U distances in the calibration chart. Each of the V camera positions and U distances may be employed to effectively span the focus range of an imaging device for a particular working point (e.g., zoom position). Calibration images may be denoted as (v,u), wherein 1≦v≦V, and 1≦u≦U.

At block410, regions of interest in the images may be defined. For example, as will be discussed in more detail below with respect toFIGS. 8A-8B, one or more regions of the image window may be defined. A statistical representation for each region of interest may be calculated at block415. The statistics may be denoted as S(v,u,i) where v and u represent focus positions and distance, respectively, and i represents the region of interest. At block420, process400may continue by calculating a generalized ratio of corresponding statistics for the images taken from the same region of interest. For example, the generalized ratio may be characterized as R(v1,v2,u,i)=tg−1(S(v1,u,i)/S(v2,u,i)).

At block425, a histogram may be calculated for each pair of focus positions (v1, v2) and for each distance u. In one embodiment the histogram may be created using all ratio functions calculated for all regions of interest. The histogram may be provided to describe the generalized ratio probability distribution for a specific focus position (e.g., v1, v2) and distance (e.g., u). As will be discussed in more detail below with respect toFIG. 9, histograms may be calculated for a plurality of distances. Calculation of the histogram at block425may further include determining one or more histogram characteristics, such as the major mode of the histogram, and mean or median of the histogram.

Process400may further include determining LUT values based on a determined histogram at block430. The major mode may be denoted by R0(v1,v2,u). According to one embodiment of the invention, for employed pair (v1, v2), the major mode may be employed for determining object distance relative to the generalized ratio. For example, the following represents the relation according to one embodiment:
r=R0(v1,v2,u)

wherein the function can be interpolated, either linearly or otherwise, for values of u which are not measured. The inverse function may define the LUT which represents distance as a function of the generalized ratio which may be used for estimating focus distance. A graphical representation of an exemplary LUT values is depicted inFIG. 10.

In certain instances, the generalized ratio r may not provide monotonic values for u, rather there may be distances which correspond to a specific ratio value. In such a case, the LUT will provide the plurality of distance values. Further, different LUT's may be estimated for different focus, aperture, and/or zoom positions.

According to another embodiment of the invention, a confidence LUT may be determined. For example, the calibration LUT may be trained as part of a DFD calibration process (e.g., process400). Referring now toFIG. 5, a process is depicted for calculating one or more confidence values associated with depth estimation. Process500may determine the probability for estimating a given depth, u_est for given focus positions (v1, v2). Process500may be initiated by calculating a generalized ratio for focus positions (e.g., (v1, v2)) at block505. Calculations at block505may be based on data determined by a calibration process, such as one or more of blocks405-420ofFIG. 4. It may also be appreciated that the generalized ratio for focus positions may be determined based on a differing set of chart images than previously determined for LUT calibration.

At block510, generalized ratios, R(v1,v2,u,i), may be used for a given focus position (v1,v2) and given depth u, to estimate the depth for each region of interest, u_est(v1,v2,u,i). Depth may be calculated based on values determined by the LUT as discussed above with reference toFIG. 3. At block515, a histogram may be calculated based on all estimated depths for each region of interest.

Based on the calculated histogram, the respective probability function for estimating depth, P(u_est(v1,v2,u) may be calculated at block520. By selecting a subset Uc⊂U, wherein the probability for estimating u_est from Uc can be formulated as:
P—Uc(u_est|v1,v2)=ΣuεUcP(u_est|v1,v2,u)·P(u|v1,v2)

For example, assuming P(u|v1,v2) is uniformly distributed in a predefined region, P_Uc(u_est|v1,v2) and P_Uc(u_est|v1,v2, u) may be calculated.

At block525, the confidence level for each u_estεU may be calculated and stored in a confidence LUT. By defining Uc as a subset of U that includes only values that are close to u_est, thus representing the probability for a correct estimation, the confidence level can be devised from the formulation:
Confidence(u_est|v1,v2)=P—Uc(u_est|v1,v2)/P—U(u_est|v1,v2)

where P_U(u_est|v1,v2) is the probability to obtain u_est when using all the elements from U.

According to another embodiment, a confidence level calculated for the depth LUT may be based on a derivative of distance u of the function r=R0(v1,v2, u), calculated during the calibration procedure. Accordingly, the confidence level may be denoted as:

Conf⁡(u)=∂R0⁡(v⁢⁢1,v⁢⁢2,u)∂u
In this fashion, the function may provide values representative of zero to infinity, wherein the higher the value, the higher the confidence that the resulted depth estimation is correct. In certain embodiments, the confidence level function may be limited, or truncated, and can be implemented using a confidence LUT. In one embodiment, confidence levels calculated for the depth LUT may be employed to select one or more values for setting AF of the imaging device.

Referring now toFIG. 6, a graphical representation of a DFD process is depicted according to one embodiment of the invention. DFD process600may be performed by an imaging device to provide auto focus. As depicted inFIG. 6, a DFD process may be initiated by receiving image data, shown as605. In one embodiment, DFD process600may be performed for two images, however it may be appreciated that additional images may be previewed by an imaging device (e.g., imaging device200). Image data605may relate to two images of a scene captured with different focus parameters, such as preview images associated with the first and second images. In certain embodiments, image data605may be down sampled since an image, I1for example, may be blurred (due to out of focus) version of I2. Accordingly, image data605may thus be down sampled prior to alignment.

As depicted inFIG. 6, image data605may be aligned at block610so that image data of the frames may be compared. Based on the alignment, a statistical representation at blocks615and620may be calculated for aligned image data, that is one or more of aligned image Ia1and Ia2, for each region of interest in an image. Statistical representations, S1and S2, generated at blocks615and620for each image may then be employed for depth estimation at block625based on one or more values of LUT630. LUT630may store information associated with the focus parameters of the preview images I1and I2. Estimated depth determined by the imaging device may then be output for automatic focus of the imaging device for a scene associated with the preview images I1and I2as shown by635.

According to another embodiment, an alignment transformation may be determined for image data and applied to statistics. Referring now toFIG. 7a graphical representation of a DFD process is depicted according to another embodiment of the invention. DFD process700may be performed by an imaging device to provide automatic focus in a similar fashion to the DFD process ofFIG. 6. As depicted inFIG. 7, DFD process may be initiated by receiving image data, shown as705. DFD process700may be performed for two images, however it may be appreciated that additional images may be previewed by an imaging device (e.g., imaging device200). As depicted inFIG. 7, an alignment transformation, shown as710, may be performed for image data705. In addition, process700may determine statistics for each of nonaligned images I1and I2as shown by statistics extraction715and720. The determined alignment transformation710may then be applied to the statistical representations S1and S2associated with images I1and I2, respectively, as shown by725. The alignment may be performed directly on the statistics, not the entire images. Accordingly, depth estimation may be determined for statistical representations, Sa1and Sa2as shown by730, based on LUT735information associated with the focus parameters of the preview images I1and I2. The estimated depth determined by the imaging device may then be output for automatic focus of the imaging device as shown by740.

Referring now toFIGS. 8A-8B, a graphical representation is depicted for regions of statistics extraction by a DFD process according to one or more embodiments. Referring first toFIG. 8A, statistics extraction may be performed on image data (e.g., image data605or705) for one or more sub-windows.FIG. 8Adepicts a frame800divided into sub-windows805. According to one embodiment, division of image data into sub-windows may allow for unique DFD statistics per sub-windows. In certain embodiments, each sub-window may be analyzed for statistics extraction. In other embodiments only a portion of the sub-windows, such as sub-windows8101-n, are utilized for calculation of statistics extraction.

Referring now toFIG. 8B, sub-windows for statistics extraction are depicted according to another embodiment. As depicted inFIG. 8B, sub-windows may be determined based on objects of an image scene and accordingly may not be uniform. Accordingly one or more sub-windows shown as8601-nmay be analyzed for statistics extraction.

According to one embodiment, the same sub-windows may be employed for each image received. Further, selected windows may be employed for performing image alignment based on similarities between image content associated with a sub-window.

Referring now toFIG. 9, a graphical representation is depicted of a probability function which may be employed for estimation of a focus depth according to one embodiment of the invention. A probability function may be generated for different distances (u) of objects in an image frame for DFD calibration. Respective probability function900is depicted for one or more depths u, shown as9051-n.

Referring now toFIG. 10, a graphical representation is depicted of a LUT which may be employed for estimation of a focus depth according to one embodiment of the invention. LUT1000is depicted for distances (u) which correspond to generalize ratio (r). As discussed above the generalized ratio may calculated represent the connection between objects distance as a function of the generalized ratio. As further discussed above, there may be several instances where the mode of the histogram, corresponds to several distances which correspond to a specific ratio value. Measurement values for distances are shown by nodes1010. As depicted inFIG. 10, the major mode of a histogram, shown as1005, includes several distances which correspond to a particular ratio value. For example, distances1015and1020correspond to a generalized ratio of 0.5. According to one embodiment, the LUT may output each distance, such as distances1015and1020for determining depth estimation.

While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art. Trademarks and copyrights referred to herein are the property of their respective owners.