Depth map generation and post-capture focusing

Aspects of depth map generation and post capture focusing and re-focusing are described. According to one embodiment, a depth map is generated. The depth map may include a mapping among relative depth values in a field of view of an image based on a difference between pixels of a first image and pixels of a second image. An edge map may also be generated by identifying edges in at least one of the first image or the second image. Using the depth map and the edge map, the relative depth values in the depth map may be smoothed using the edge map. In this manner, certain discontinuities in depth values may be smoothed within edge-bounded regions defined by the edge map. The depth map may be used for focusing and re-focusing, for example, or for object extraction, scene understanding, or gesture recognition, among other imaging processes.

BACKGROUND

Certain cameras, such as light-field or plenoptic cameras, rely upon a lens array over an image sensor and/or an array of image sensors to capture directional projection of light. Among other drawbacks, these approaches use relatively large and specialized image sensors which are generally unsuitable for other applications (e.g., video capture, video conferencing, etc.), use only a fraction of the information captured, and rely upon high levels of processing to deliver even a viewfinder image, for example. Further, some of these light-field or plenoptic camera devices require a relatively large height for specialized lens and/or sensor arrays and, thus, do not present practical solutions for use in cellular telephones.

The drawings illustrate are provided by way of example and should not be considered limiting of the scope of the embodiments described herein, as other equally effective embodiments are within the scope and spirit of this disclosure. The elements and features shown in the drawings are not necessarily drawn to scale, emphasis instead being placed upon clearly illustrating the principles of the embodiments. Additionally, certain dimensions or positions of elements and features may be exaggerated to help visually convey certain principles. In the drawings, similar reference numerals among the figures generally designate like or corresponding, but not necessarily the same, elements.

DETAILED DESCRIPTION

In the following paragraphs, the embodiments are described in further detail by way of example with reference to the attached drawings. In the description, well known components, methods, and/or processing techniques are omitted or briefly described so as not to obscure the embodiments.

Certain cameras, such as light-field or plenoptic cameras, rely upon a lens array over an image sensor and/or an array of image sensors to capture directional projection of light. Among other drawbacks, these approaches use relatively large and specialized image sensors which are generally unsuitable for other applications (e.g., video capture, video conferencing, etc.), use only a fraction of the information captured, and rely upon high levels of processing to deliver even a viewfinder image, for example. Further, some of these light-field or plenoptic camera devices require a relatively large height for specialized lens and/or sensor arrays and, thus, do not present practical solutions for use in cellular telephones.

In this context, the embodiments described herein include a system and device for focusing and re-focusing. The embodiments may be relied upon to achieve, among other processing results, image processing results that are similar, at least in some aspects, to those achieved by light-field or plenoptic imaging devices. In certain embodiments, the system may omit some elements which are generally common in imaging systems, such as optical and/or mechanical focusing mechanisms and associated controls. By the opportunity to omit these system elements, costs of camera modules may be reduced. Further, instantaneous power draw during focusing and focus time may be reduced. Additionally, the overall size, weight, and footprint of camera modules may be reduced.

In one embodiment, the system includes a main color image sensor having a pixel density ranging from 3 to 20 Megapixels, for example, with color pixels arranged in a Bayer pattern, and a secondary luminance image sensor having a relatively lower pixel density. It should be appreciated, however, that the system is generally agnostic to the resolution and format of the main and secondary sensors, which may be embodied as sensors of any suitable type, pixel resolution, process, structure, or arrangement (e.g., infra-red, charge-coupled device (CCD), 3CCD, Foveon X3, complementary metal-oxide-semiconductor (CMOS), red-green-blue-clear (RGBC), etc.).

In one aspect, the system and device may be relied upon for focusing and re-focusing images after they are captured. For example, a luminance image provided by the secondary luminance sensor may be relied upon, in connection with the luminance component of an image from the main color image sensor, to generate a depth map representative of relative depth values. The depth map may be used for focusing and re-focusing, for example, or for object extraction, scene understanding, gesture recognition, etc. In other aspects, a mix of image sensors may be used for high dynamic range (HDR) image processing. Further, according to the embodiments described herein, the mix of image sensors may be calibrated for focusing and re-focusing, object extraction, scene understanding, gesture recognition, HDR image processing, etc.

Turning now to the drawings, a description of exemplary embodiments of a system and its components are provided, followed by a discussion of the operation of the same.

FIG. 1Aillustrates a system10for depth map generation and post-capture focusing according to an example embodiment. The system10includes a processing environment100, a memory110, and first and second sensors150and152, respectively. The processing environment100includes a scaler120, a calibrator122, a depth map generator124, an edge map generator126, a smoother128, a focuser130, and an image processor132. The memory110includes memory areas for image data112and calibration characteristic data114.

The processing environment100may be embodied as one or more processors, processing circuits, and/or combinations thereof. Generally, the processing environment100includes embedded (i.e., application-specific) and/or general purpose processing circuitry and/or software structures that process data, such as image data captured by the first and second sensors150and152, for example. Further structural aspects of the processing environment100are described below with reference toFIG. 13. In one embodiment, among others, the processing environment100may include the scaler120, calibrator122, depth map generator124, edge map generator126, smoother128, focuser130, and image processor132. Each of these elements of the processing environment100, and the respective operation of each, is described in further detail below with reference to the remaining figures.

The first and second sensors150and152may be embodied as any suitable types of sensors, depending upon the application for use of the system10. For example, in image processing applications, the first and second sensors150and152may be embodied as image sensors having the same or different pixel densities, ranging from a fraction of 1 to 20 Megapixels, for example. The first image sensor150may be embodied as a color image sensor having a first pixel density, and the second image sensor152may be embodied as a luminance image sensor having a relatively lower pixel density. It should be appreciated, however, that the system10is generally agnostic to the resolution and format of the first and second sensors150and152, which may be embodied as sensors of any suitable type, pixel resolution, process, structure, or arrangement (e.g., infra-red, charge-coupled device (CCD), 3CCD, Foveon X3, complementary metal-oxide-semiconductor (CMOS), red-green-blue-clear (RGBC), etc.).

The memory110may be embodied as any suitable memory that stores data provided by the first and second sensors150and152, among other data, for example. In this context, the memory110may store image and image-related data for manipulation and processing by the processing environment100. As noted above, the memory110includes memory areas for image data112and calibration characteristic data114. Various aspects of processing and/or manipulation of the image data112by the processing environment100are described in further detail below.

FIG. 1Billustrates a device160for depth map generation and post-capture focusing the system ofFIG. 1Aaccording to an example embodiment. The device160includes the processing environment100, the memory110, and the first and second sensors150and152ofFIG. 1A, among other elements. The device160may be embodied as a cellular telephone, tablet computing device, laptop computer, desktop computer, television, set-top box, personal media player, appliance, etc., without limitation. In other embodiments, the device160may be embodied as a pair of glasses, a watch, wristband, or other device which may be worn or attached to clothing. If embodied as a pair of glasses, then the sensors150and152of the device160may be positioned at opposite corners of rims or end-pieces of the pair of glasses.

As illustrated inFIG. 1B, the first and second sensors150and152are separated by a first distance X in a first dimension and by a second distance Y in a second dimension. The distances X and Y may vary among embodiments, for example, based on aesthetic and/or performance factors, depending upon the application or field of use for the device160. Further, the relative positions (e.g., right verses left, top verses bottom, etc.) of the first and second sensors150and152may vary among embodiments. In this context, it is also noted that a relative difference in rotational or angular displacement (i.e., R1−R2) may exist between the first and second sensors150and152. Although not explicitly illustrated, it should be appreciated that the device160may include one or more additional elements for image capture, such as lenses, flash devices, focusing mechanisms, etc., although these elements may not be relied upon in certain embodiments and may be omitted.

As described herein, the first and second sensors150and152may be embodied as sensors of similar or varied operating and structural characteristics. The differences in operating characteristics may be identified during manufacturing and/or assembly of the device160, for example, based on manufacturing and/or assembly calibration processes. Additionally or alternatively, the differences in operating characteristics may be identified during post-assembly calibration processes. These differences may be quantified as calibration data which is representative of the operating characteristics of the first and second sensors150and152, and stored in the memory110as the calibration characteristic data114.

Among other operational aspects, the device160is configured to capture images using the first and second sensors150and152. Based on the processing techniques and processes described herein, images captured by the first and second sensors150and152may be focused and re-focused after being captured. Generally, using images captured by the first and second sensors150and152, the processing environment100is configured to generate a depth map including a mapping among relative depth values within a field of view of at least one of the images, generate an edge map, and smooth the relative depth values of the depth map using the edge map. The relative depth values and/or the smoothed relative depth values may be used for focusing and re-focusing images after they are captured. Further, the processing techniques and processes described herein may be used for object extraction, scene understanding, gesture recognition, etc.

FIG. 2illustrates a process flow for depth map generation and post-capture focusing the system10ofFIG. 1Aaccording to an example embodiment. As illustrated inFIG. 2, the first sensor150generates a first image202, and the second sensor152generates a second image204. The first and second images202and204may be captured at a substantially same time. Alternatively, the first and second images202and204may be captured, respectively, by the first and second sensors150and152, at different times. Data associated with the first and second images202and204may be stored in the memory110(FIG. 1).

According to certain aspects of the embodiments described herein, the first image202provided by the first sensor150is compared with the second image204provided by the second sensor152, to determine a depth map. In this context, the first and second images202and204capture or are representative of substantially the same field of view. Generally, similar or corresponding image information (e.g., pixel data) among the first and second images202and204is shifted in pixel space between the first and second images202and204due to the relative difference in position (e.g., illustrated as X, Y, R1, and R2 inFIG. 1B) between the first and second sensors150and152on the device160. The amount of this shift, per pixel, is representative of depth, because it is dependent (i.e., changes) upon the relative depths of items within a field of view of the images202and204. Additionally, it is noted that the image information among the first and second images202and204is typically shifted in other aspects, such as luminance, color, color coding, pixel density, noise, etc., and these differences should be accounted for by the calibrator122of the system10before or while processing the images202and204.

In alternative embodiments, the device160may include the first sensor150, and the second sensor152may be omitted. In this case, the first sensor150may be relied upon to provide first and second images, sequentially in time, to determine a depth map. That is, two respective images may be captured in a relatively short period of time by the first image sensor150, and these two images may be used to generate a depth map. The two images may be of the same or different pixel resolutions. Because the images are captured sequentially in time, the images may include a shift between them due to movement of the device160while the images are being captured, for example.

According to various embodiments described herein, the first and second images202and204may have the same or different pixel densities, depending upon the respective types and characteristics of the first and second image sensors150and152, for example. Further, the first and second images202and204may be of the same or different image formats. For example, the first image202may include several color components of a color image encoded or defined according to a certain color space (e.g., red, green, blue (RGB); cyan, magenta, yellow, key (CMYK); phase alternating line (PAL); YUV or Y′UV; YCbCr; YPbPr, etc.), and the second image204may include a single component of another color space.

To the extent that the processes of depth map generation and post-capture focusing described herein rely upon one or more components of image data captured by the first and second sensors150and152, the image processor132may extract those components, as necessary, for further processing. Further, to the extent that the characteristics of the first and second sensors150and152vary, such that the first and second images202and204deviate along a corresponding unit of measure or other qualitative or quantitative aspect, for example, the calibrator122may adjust one or more of the operating parameters of the first and second sensors150and152(e.g., operating voltages, timings, temperatures, exposure timings, etc.) or adjust one or more of the first and second images202and204to address the difference or differences. In other words, the calibrator122may seek to align or normalize aspects of the operating characteristics of the first and second sensors150and152or the image data captured by the first and second sensors150and152. In this way, downstream operations performed by other elements in the system10may be aligned, as necessary, for suitable performance and results in image processing.

As further illustrated inFIG. 2, the first and second images202and204are provided to the scaler120. Generally, the scaler120downscales and/or upscales images or maps (e.g., depth and/or edge maps), as described herein, in pixel density. It is noted that, in certain embodiments, the scaler120may be omitted from the process flow ofFIG. 2, for one or more of the first and second images202and204, as described in connection with other embodiments. The scaler120is generally relied upon, for example, to reduce the pixel processing loads of the edge map generator126and the depth map generator124, to align pixel densities among the first and second images202and204(e.g., if the first and second sensors150and152vary in pixel density), and/or to reduce or compact image features for edge or depth detection. The downscaling and/or upscaling operations of the scaler120may be embodied according to nearest-neighbor interpolation, bi-linear interpolation, bi-cubic interpolation, supersampling, and/or other suitable interpolation techniques, or combinations thereof, without limitation.

After the scaler120downscales the first image202into the first downscaled image212and downscales the second image204into the second downscaled image214, the first downscaled image212is provided to the edge map generator126. The edge map generator126, generally, generates an edge map by identifying edges in at least one image. In other words, the edge map generator126generates an edge map by identifying edges in one or more of the first or second downscaled images212and214. In the embodiment illustrated inFIG. 2, the edge map generator126generates the edge map222by identifying edges in the first downscaled image212, although the edge map222may be generated by identifying edges in the second downscaled image214. It should be appreciated that the performance of the edge map generator126may be improved by identifying edges in downscaled, rather than higher pixel density, images. For example, edges in higher density images may span several (e.g., 5, 10, 15, or more) pixels. In contrast, such edges may span relatively fewer pixels in downscaled images. Thus, in certain embodiments, the scaler120may be configured to downscale one or more of the first or second images202or204so as to provide a suitable pixel density for accurate edge detection by the edge map generator126.

FIG. 3illustrates an example edge map222generated by the edge map generator126ofFIG. 1Aaccording to an example embodiment. As illustrated inFIG. 3, the edge map222is embodied by data representative of edges. In the context ofFIGS. 2 and 3, the edge map222is embodied by data representative of edges in the first image202. In one embodiment, the edge map generator126generates the edge map222by identifying pixels or pixel areas in the first image202where pixel or pixel area brightness quickly changes or encounters a discontinuity (i.e., at “step changes”). Points at which pixel brightness change quickly are organized into edge segments in the edge map222by the edge map generator126. The changes may be due to changes in surface or material orientation, changes in surface or material properties, or variations in illumination, for example. Data associated with the edge map222may be stored by the edge map generator126in the memory110(FIG. 1).

Referring again toFIG. 2, the first and second downscaled images212and214are also provided to the depth map generator124. The depth map generator124, generally, generates a depth map including a mapping among relative depth values in a field of view based on a difference between pixels of a first image and pixels of a second image. In the context ofFIG. 2, the depth map generator124generates a depth map224including a mapping of relative depth values based on differences between pixels of the first downscaled image212and pixels of the second downscaled image214.

FIG. 4illustrates an example depth map224generated by the depth map generator124ofFIG. 1Aaccording to an example embodiment. As illustrated inFIG. 4, the depth map224is embodied by data representative of relative depths in a field of view based on differences between pixels of the first downscaled image212and pixels of the second downscaled image214. InFIG. 4, relatively darker areas are closer in depth and relatively lighter areas are further in depth, from the point of view of the first and second image sensors150and152and/or the device160(FIG. 1B). It should be appreciated that the relatively darker and lighter areas inFIG. 4are representative of depth values. That is, relatively darker areas are representative of data values (e.g., per pixel data values) associated with less depth, and relatively lighter areas are representative of data values associated with more depth. In the context ofFIGS. 6 and 7, as further described below, the depth map224is referred to as a “raw” depth map, because it is representative of unsmoothed or unfiltered depth values. Data associated with the depth map224may be stored by the depth map generator124in the memory110(FIG. 1).

The depth map generator124may generate the depth map224, for example, by calculating a sum of absolute differences (SAD) between pixel values in a neighborhood of pixels in the downscaled image212and a corresponding neighborhood of pixels in the downscaled image214, for each pixel in the downscaled images212and214. Each SAD value may be representative of a relative depth value in a field of view of the downscaled images212and214and, by extension, the first and second images202and204. In alternative embodiments, rather than (or in addition to) calculating relative depth values of the depth map224by calculating a sum of absolute differences, other stereo algorithms, processes, or variations thereof may be relied upon by the depth map generator124. For example, the depth map generator124may rely upon squared intensity differences, absolute intensity differences, mean absolute difference measures, or other measures of difference between pixel values, for example, without limitation. Additionally, the depth map generator124may rely upon any suitable size, shape, or variation of pixel neighborhoods for comparisons between pixels among images. Among embodiments, any suitable stereo correspondence algorithm may be relied upon by the depth map generator124to generate a depth map including a mapping among relative depth values between images.

According to certain embodiments, the depth map generator124generates the depth map224by iteratively searching for and evaluating depth among images of various pixel densities. For example, the scaler120may downscale each of the first and second images202and204into several downscaled images (e.g., 5× downscale, 10× downscale, 15× downscale, etc.), and the depth map generator124may generate the depth map224by iteratively searching for and evaluating depth among pairs of the downscaled images which correspond to the first and second images202and204.

In the context of iteratively searching for and evaluating depth among images of various pixel density,FIG. 5illustrates graduated levels of pyramid disparity depth maps generated and relied upon by the depth map generator124ofFIG. 1Aaccording to an example embodiment. InFIG. 5, depth maps500,502,504, and506of graduated pixel density are illustrated. The depth map500is embodied as a depth map of relatively high pixel density, and the depth maps502,504, and506are embodied as depth maps of progressively lower pixel density, as illustrated inFIG. 5. As suggested above, each of the depth maps500,502,504, and506may be generated by the depth map generator124based on images of corresponding pixel density provided by the scaler120.

Generally, when generating a depth map using pyramid disparity, the depth map generator124first generates the depth map506by comparing relatively low pixel density images (i.e., low pixel density copies of the first and second images202and204). Afterwards, the depth map generator124generates the depth map504by comparing images of relatively higher pixel density (i.e., images of higher pixel density than those used to generate the depth map506) and with reference to the depth map506. That is, when generating the depth map504, the depth map generator makes reference to the depth map506. For example, when generating the depth map504, the depth map generator124may reference the regions of depth identified in the depth map506, to determine the manner in which to tailor the search for depth among a pair of images of higher pixel density. Thus, the depth map506may be considered a coarse map for depth reference, and the depth map504may be determined by the depth map generator124by identifying shifts in pixel values among a pair of downscaled images and with reference to the depth map506.

The generation of each of the depth maps504,502, and500may proceed in a similar manner, with the depth map generator124referring to lower or coarser density depth maps for the generation of each relatively finer density depth map. On the basis of or with reference to coarser density depth maps, the depth map generator124may be able to tailor the manner in which finer depth maps are generated, for speed or accuracy, for example. Additionally, a depth map of suitable pixel resolution may be generated using a pyramid of images of increasing pixel density, starting with comparisons of lower or coarser density images and proceeding to higher or finer density images. The progression from lower to higher pixel density in the generation of depth maps, with reference back to coarser depth maps, may assist with the generation of a final depth map having suitable pixel density in relatively less time and using a relatively smaller amounts of memory, for example.

In other embodiments, the depth map generator124may stitch together one or more depth maps. In the context of panoramic images, for example, the depth map generator124may stitch together one or more depth maps to generate a panoramic depth map. As further described below, for panoramic depth maps, smoothing may occur across or among depth maps which have been stitched together, to help remove discontinuities in depth due to occlusions, etc.

After the edge map generator126generates the edge map222and the depth map generator124generates the depth map224, the smoother128smooths the relative depth values of the depth map224using the edge map222. For example, according to one embodiment, the smoother128filters columns (i.e., in a first direction) of depth values of the depth map224between a first pair of edges in the edge map222. The smoother128further filters rows (i.e., in a second direction) of depth values of the depth map224between a second pair edges in the edge map222. The process of filtering along columns and rows may proceed iteratively between filtering columns and rows, until a suitable level of smoothing has been achieved.

FIG. 6illustrates an example process of smoothing performed by the smoother128ofFIG. 1Aaccording to an example embodiment. InFIG. 6, the depth map600is smoothed or filtered along columns (i.e., in a first direction Y) of depth values and between pairs of edges, and the depth map602is smoothed or filtered along rows (i.e., in a second direction X) of depth values and between pairs of edges. With reference toFIGS. 3 and 4, the depth map600is representative, for example, of depth values after a first pass of smoothing depths along columns, using the raw depth map224as a basis for depth values and the edge map222as a basis for edges. The depth map602is representative of smoothed depth values after a second pass of smoothing depths along rows, using the depth map600as a starting basis for depth values.

More particularly, in the generation of the depth map600by the smoother128, the smoother128scans along columns of the depth map600, from a right to a left, for example, of the map. The columns may be scanned according to a column-wise pixel-by-pixel shift of depth values in the map. Along each column, edges which intersect the column are identified, and the depth values within or between adjacent pairs of intersecting edges are filtered. For example, as illustrated inFIG. 6, along the column610of depth values, a pair of adjacent edges612and614is identified by the smoother128. Further, the pair of adjacent edges616and618is identified by the smoother128. Once a pair of adjacent edges is identified along a column, the smoother128filters the depth values between the pair of edges, to provide a smoothed range of depth values between the pair of edges. As illustrated inFIG. 6, smoothing or filtering depth values between pairs of edges is performed by the smoother128along the column610, on a per edge-pair basis. In this way, raw depth values in the raw depth map224(FIG. 4) are smoothed or filtered with reference to the edges in the edge map222(FIG. 3). Thus, depth values are generally extended and smoothed with a certain level of consistency among edges.

As further illustrated inFIG. 6, starting with the depth map600as input, the smoother128scans along rows of the depth map602, from a top to a bottom, for example, of the map. The rows may be scanned according to a row-wise pixel-by-pixel shift of depth values in the map. Along each row, edges which intersect the row are identified, and the depth values within or between adjacent pairs of intersecting edges are filtered. For example, along the row620of depth values, a pair of adjacent edges622and624is identified by the smoother128. Further, the pair of adjacent edges626and628is identified by the smoother128. Once a pair of adjacent edges is identified along a row, the smoother128filters the depth values between the pair of edges, to provide a smoothed range of depth values between the pair of edges. As illustrated inFIG. 6, smoothing or filtering depth values between pairs of edges is performed by the smoother128along the row620, on a per edge-pair basis. In this way, depth values are generally extended and smoothed with a certain level of consistency among edges. It should be appreciated here that several pairs of intersecting edges may be identified along each column610and row620in a depth map, and depth values may be smoothed between each of the pairs of edges.

FIG. 7further illustrates the example process of smoothing performed by the smoother128ofFIG. 1Aaccording to an example embodiment. As illustrated inFIG. 7, the smoother128smooths a depth map along columns and rows of depth values iteratively, alternating between smoothing along columns and rows. In this context, as illustrated inFIG. 7, the smoother128smooths the depth map224along columns to generate the depth map600, then smooths the depth map600along rows to generate the depth map602, then smooths the depth map602along columns to generate the depth map604, and then smooths the depth map604along rows to generate the depth map606. In various embodiments, this iterative process of smoothing along columns and rows may repeat for a predetermined number of times depending upon various factors. Further, it should be appreciated that the process may begin and end with smoothing along rows or smoothing along columns, without limitation. Any of the depth maps600,602,604, or606may be selected as a suitable depth map. Again, as described above, the smoother128generally seeks to smooth or filter depth values from the raw depth map224with reference to, between, and/or among edges in the edge map222, generating a correlation among depth values and edges.

As illustrated among the depth maps600,602,604, or606, the iterative process of smoothing along columns and rows generally spreads, smooths, and/or filters depth values within edge-bounded regions in both column and row (i.e., X and Y) directions. In this sense, the smoother128smooths discontinuities in depth values in the raw depth map224, at least to a certain extent, within edge-bounded regions defined by the edge map222. The discontinuities in depth values may be attributed to occlusions, for example. That is, certain image data may be captured in the first image202(FIG. 2), although such data may be omitted entirely from the second image204(FIG. 2). This discrepancy or occlusion may be due to a parallax difference among the first and second sensors150and152based on their respective positions on the device160(FIG. 1B). To a certain extent, discontinuities in depth values may also be found along the distal sides or edges of the raw depth map224, for example, especially if the field of view of the first image202varies from that of the second image204. Further, in the context of panoramic images, the smoother128may smooth discontinuities in depth values which occur across or among depth maps which have been stitched together, to help remove discontinuities in depth due to occlusions, etc.

Turning toFIG. 8, the results of various smoothing processes performed by the smoother128ofFIG. 1Aare illustrated according to an example embodiment. InFIG. 8, three depth maps800,802, and804are illustrated. The depth map800was generated by the smoother128using a linear fit of depth values between edges, the depth map802was generated by the smoother128using a median of depth values between edges, and the depth map804was generated by the smoother128using a mean of depth values between edges. In other words, to generate the depth map800, the smoother128calculates a linear fit for each depth value along a column or a row, using the depth values between pairs of edges along the column or the row. To generate the depth map802, the smoother128calculates a median of depth values along a column or the row, using the depth values between pairs of edges along the column or the row. To generate the depth map804, the smoother128calculates a mean of depth values along a column or the row, using the depth values between pairs of edges along the column or the row.

As illustrated inFIG. 8, by calculating a linear fit, median, or mean of depth values between edges, the “fit” of depth values among edge-bounded regions varies among the depth maps800,802, and804. It should be appreciated that the smoother128may rely upon ways to smooth depth values other than by calculating a linear fit, median, or mean of values. The smoother128may be configured to operate based on a linear fit, median, mean, or other manner of calculating smoothed or filtered depth values depending, for example, upon the application for use of the resulting depth map, processing capacity, speed, etc. In this context, it should be appreciated that a linear fit of depth values may be preferable for some applications, while a mean fit may be preferable for other applications. Further, it should be appreciated that a median fit of depth values may be preferable over a mean fit, although a tradeoff in processing requirements exists, because identifying a median of depth values between edges depends upon a sort of the values which is not necessary for identifying a mean of the depth values.

Referring back toFIG. 2, after the smoother128smooths the depth values in the depth map224, to provide a smoothed depth map226, the smoother128provides the smoothed depth map226to the scaler120. The scaler120upscales the smoothed depth map226, and provides an upscaled depth map228to the focuser130. Generally, the upscaled depth map228includes a density of depth values which corresponds to the pixel density of the first and/or second images202and204. Using the upscaled depth map228, the focuser130may focus and/or re-focus one or more pixels in the first image202, for example, with reference to corresponding values of depth in the depth map224.

In the context of focusing and/or refocusing,FIG. 9Aillustrates a process flow for focusing and/or refocusing elements of the system10ofFIG. 1Aaccording to an example embodiment. As illustrated inFIG. 9A, the focuser130receives the upscaled depth map228, the first image202, and a point for focus140. Additionally, the image processor132receives the first image202and provides a blurred replica250of the first image202to the focuser130. Generally, the focuser130selectively focuses the first image202according to the point for focus140, by blending portions of the blurred replica250with the first image202, with reference to the relative depth values of the upscaled depth map228as a measure for blending. The focuser130provides an output image260A based on a blend of the first image202and the blurred replica250.

The point for focus140may be received by the device160(FIG. 1B) using any suitable input means, such as by capacitive touch screen, mouse, keyboard, electronic pen, etc. That is, a user of the device160may, after capture of the first and second images202and204by the device160, select a point on the first image202(or the second image204) to be selectively focused using a capacitive touch screen, mouse, keyboard, electronic pen, etc. Here, it is noted that the first image202may be captured by the first sensor150according to a relatively large depth of field. In other words, the first image202may be substantially focused throughout its field of view, for example, based on a sufficiently small optical aperture, etc. Thus, after capture of the first image202, the focuser130may selectively focus areas of the first image202based on depth, by simulating a focal point and associated in-focus depth of field of the first image202along with other depths of field which are out of focus (i.e., blurred).

According to one embodiment, for a certain point of focus140selected by a user, the focuser130identifies a corresponding depth value (i.e., a selected depth value for focus) in the upscaled depth map228, and evaluates a relative difference in depth between the selected depth value and each other depth value in the upscaled depth map228. Thus, the focuser130evaluates the depth values in the upscaled depth map228according to relative differences from the point of focus140. In turn, the focuser130blends the first image202and the blurred replica250based on relative differences in depth, as compared to the point of focus140.

In one embodiment, the blurred replica250may be generated by the image processor132using a Gaussian blur or similar filter, and the focuser130blends the first image202and the blurred replica250according to an alpha blend. For example, at the point of focus140, the focuser130may form a composite of the first image202and the blurred replica250, where the first image202comprises all or substantially all information in the composite and the blurred replica250comprises no or nearly no information in the composite. On the other hand, for a point in the first image202having a relatively significant difference in depth as compared to the point of focus140in the first image202, the focuser130may form another composite of the first image202and the blurred replica250, where the first image202comprises no or nearly no information in the composite and the blurred replica250comprises all or substantially all information in the composite.

The focuser130may evaluate several points among the first image202for difference in depth as compared to the point of focus140, and generate or form a composite image for each point based on relative differences in depth, as compared to the point of focus140for focus140as described above. The composites for the various points may then be formed or joined together by the focuser130into the output image260A. In one embodiment, the focuser130may evaluate individual pixels in the first image202for difference in depth as compared to the point for focus140, and generate or form a composite image for each pixel (or surrounding each pixel) based on relative differences in depth embodied in the depth values of the depth map224, as compared to the point of focus140.

According to the operation of the focuser130, the output image260A includes a region of focus identified by the point for focus140, and a blend of regions of progressively less focus (i.e., more blur) based on increasing difference in depth as compared to the point for focus140. In this manner, the focuser130simulates a focal point and associated in-focus depth of field in the output image260A, along with other depths of field which are out of focus (i.e., blurred). It should be appreciated that, because the depth map224includes several graduated (or nearly continuous) values of depth, the output image260A also includes several graduated ranges of blur or blurriness. In this way, the focuser130simulates the effect of capturing the image202using a relatively larger optical aperture, and the point of focus when capturing the image202may be altered after the image202is captured. Particularly, several points for focus140may be received by the focuser130over time, and the focuser130may generate respective output images260A for each point for focus140.

FIG. 9Billustrates another process flow for focusing and/or refocusing elements of the system10ofFIG. 1Aaccording to an example embodiment. According to the embodiment ofFIG. 9B, rather than relying upon the blurred replica250, the focuser130selectively focuses regions of the first image202without using the blurred replica250. In this context, according to the embodiment illustrated inFIG. 9B, the focuser130determines a point spread per pixel for pixels of the first image202, to generate the output image260B. For example, for pixels with little or no difference in depth relative to the point for focus140, the focuser130may form the output image260using the pixel values in the first image202without (or with little) change to the pixel values. On the other hand, for pixels with larger differences in depth relative to the point for focus140, the focuser130may determine a blend of the value of the pixel and its surrounding pixel values based on a measure of the difference. In this case, rather than relying upon a predetermined blurred replica, the focuser130may determine a blend of each pixel, individually, according to values of neighboring pixels. For any given pixel in the output image260B, the amount of contribution from neighboring pixels for that pixel may depend upon difference in depth from the point for focus. For example, for larger differences in depth, the contribution from surrounding pixels may be greater both in terms weight of contribution and in number of contributing pixels.

In the embodiment ofFIG. 9A, the focuser130again simulates the effect of capturing the image202using a relatively larger optical aperture, and the point of focus when capturing the image202may be altered after the image202is captured. Particularly, several points for focus140may be received by the focuser130over time, and the focuser130may generate respective output images260B for each point for focus140.

It is noted that the use of the blurred replica250when focusing or re-focusing, as in the embodiment ofFIG. 9A, may be less processing-intensive than determining a point spread per pixel, as in the embodiment ofFIG. 9B. Thus, depending upon the processing capabilities of the processing environment100(FIGS. 1A and 1B) and other factors (e.g., battery life, memory constraints, etc.), the device160may be rely upon one or a combination of the focusing and/or refocusing techniques described in connection withFIGS. 9A and 9B, or a combination thereof.

In still another embodiment, focusing and/or refocusing may be achieved by focus stacking using elements of the system10ofFIG. 1A. For example, if multiple images, each having a different depth of field and/or focus point, are captured by the first sensor150, then the focuser130may blend the multiple images to generate a selectively focused image. Particularly, the focuser130may blend the multiple images using relative depth values from a depth map generated based on a pair of the multiple images. Alternatively, the focuser130may blend the multiple images using relative depth values from a depth map generated based on one of the multiple images and another image captured by the second sensor152.

In this context, focus stacking generally consists of capturing multiple images, each focused at a different focal point and/or having a different depth of field, and then selectively blending or combining sharp and blurred regions of the multiple images to simulate the effect of refocusing to a particular depth. In some embodiments, the device160(FIG. 1) may include a focusing mechanism for the sensor150. In such an embodiment, the sensor150may be used to capture multiple images having different focus points and/or different depths of field, and the focuser130may blend the multiple images to generate a selectively focused image. In this case, a depth map, generated as described herein, may be relied upon by the focuser130to select which portions of the images to blend.

FIG. 10illustrates an alternative process flow for depth map generation and post-capture focusing elements of the system10ofFIG. 1Aaccording to an example embodiment. The alternative process flow illustrated inFIG. 10is similar to that ofFIG. 2, although the sensor152generates a second image214A, which is of lower pixel density than the second image204ofFIG. 2, and the scaler120is not relied upon to downscale the second image214A. In this case, the second sensor152captures and generates the second image214A at a pixel density which is lower as compared to the density of the first image202. As such, it is not necessary to downscale the pixel density of the second image214A before it is compared with the downscaled first image212by the depth map generator124.

Here, it is noted that the first and second sensors150and152may be of different pixel density and/or type. For example, the second sensor152may be a lower cost and/or lower resolution sensor, as compared to the first sensor150. Further, the second sensor152may be embodied as a luminance only sensor, for example, or other sensor of relatively limited range of capture. In this case, the image processor132may forward only the luminance data from the first image202, and the edge map generator126and the depth map generator124may compare luminance data values when generating the edge map222and the depth map224. In this case, although the depth map224may be representative of relative depth values based on luminance data, the focuser130may still blend and/or point spread color pixels of the first image202. That is, is should be appreciated that the relative depth values in the depth map224are generally agnostic to any color or types of color for the generation of selectively focused output images.

FIG. 11illustrates another alternative process flow for depth map generation and post-capture focusing elements of the system10ofFIG. 1Aaccording to an example embodiment. The alternative process flow illustrated inFIG. 11is similar to that ofFIG. 2, although the scaler120, the edge map generator126, the depth map generator124, the smoother128, and the focuser130operate on luminance components of any images captured by the first and second sensors152. In other words, rather than processing both chroma and luminance image data, for example, the scaler120, the edge map generator126, the depth map generator124, the smoother128, and the focuser130may process luminance data components only. In this way, processing requirements may be reduced, as relatively less image data is processed.

In the process flow ofFIG. 11, the output of the focuser130is provided to the summer1100, and the summer1100sums the output of the focuser130with chroma image data270from the first sensor150, to generate the output image280. Here, the summer1100may add the chroma image data270to the output of the focuser130. In this context, the output of the focuser130includes luminance image data which has been selectively focused, and the summer1100adds the chroma (i.e., color) image data270to the luminance image data, to provide the output image280. With the process flows illustrated inFIGS. 2, 10, and 11provided by way of example, it should be appreciated that the system10may operate according to other process flows within the scope and spirit of the embodiments described herein.

Before turning to the process flow diagrams ofFIG. 12, it is noted that the embodiments described herein may be practiced using an alternative order of the steps illustrated inFIG. 12. That is, the process flows illustrated inFIG. 12are provided as examples only, and the embodiments may be practiced using process flows that differ from those illustrated. Additionally, it is noted that not all steps are required in every embodiment. In other words, one or more of the steps may be omitted or replaced, without departing from the spirit and scope of the embodiments. Further, steps may be performed in different orders, in parallel with one another, or omitted entirely, and/or certain additional steps may be performed without departing from the scope and spirit of the embodiments. Finally, although the process1200ofFIG. 12is generally described in connection with the system10ofFIG. 1Aand/or the device160ofFIG. 1B, the process1200may be performed by other systems and/or devices.

FIG. 12illustrates a flow diagram for a process1200of depth map generation and post-capture focusing performed by the system10ofFIG. 1Aaccording to an example embodiment. At reference numeral1202, the process1200includes capturing one or more images. In one embodiment, at reference numeral1202, the process1200includes capturing a first image and a second image. In one embodiment, the pixel densities of the first and second images are similar. In other embodiments, a pixel density of the second image is a scalar fraction of the pixel density of the first image.

With reference toFIG. 2, the first and second sensors150and152may capture the first and second images202and204at reference numeral1202. Here, it is noted that the images captured at reference numeral1202may be captured by any suitable sensors or heterogeneous mix of image sensors. For example, a combination of color and luminance image sensors may be relied upon, for example.

At reference numeral1204, the process1200includes calibrating or adjusting one or more sensors used to capture images or images captured by the one or more sensors. For example, one or more of the sensors150and152may be calibrated or adjusted, and/or one or more of the images202or204may be calibrated or adjusted.

At reference numeral1206, the process1200includes downscaling one or more of the images captured at reference numeral1202. For example, at reference numeral1206, the process1200may include downscaling the first image captured at reference numeral1202to a downscaled first image. In this case, if the pixel density of the second image captured at reference numeral1202was a scalar fraction of the pixel density of the first image capture at reference numeral1202, a pixel density of the downscaled first image may be substantially equivalent to the pixel density of the second image. In certain embodiments of the process1200, the downscaling at reference numeral1206may be relied upon to bring the images captured at reference numeral1202into a range of similar pixel density. In some embodiments, the downscaling at reference numeral1206may be omitted, for example, if the images captured at reference numeral1202are of a similar and suitable pixel density. With reference to elements of the system10andFIG. 2, the downscaling at reference numeral1206may be performed by the scaler120.

At reference numeral1208, the process1200includes generating a depth map including a mapping among relative depth values in a field of view, based on a difference between pixels of a first image and pixels of a second image. For example, with reference to elements of the system10andFIG. 2, the depth map generator124may generate the depth map224based on the downscaled first and second images212and214. Alternatively, if downscaling is not relied upon, the depth map generator124may generate a depth map based on the first and second images202and204.

At reference numeral1210, the process1200includes generating an edge map identifying edges in at least one of the images captured at reference numeral1202and/or downscaled at reference numeral1206. For example, with reference to elements of the system10andFIG. 2, the edge map generator126may generate the edge map222based on one or more of the downscaled first or second images212or214, according to any of the techniques for the generation of depth maps described herein. Alternatively, if downscaling is not relied upon, the depth map generator124may generate a depth map based on one or more of the first or second images202or204.

At reference numeral1212, the process1200includes smoothing the relative depth values of the depth map generated at reference numeral1208using the edge map generated at reference numeral1210, and providing a smooth depth map. For example, with reference toFIG. 2, the smoother128may smooth the relative depth values of the depth map224using the edge map222, according to any of the techniques for smoothing described herein.

At reference numeral1214, the process1200includes upscaling the smoothed depth map generated at reference numeral1212. With reference toFIG. 2, the upscaling at reference numeral1214may be performed by the scaler120. In some embodiments, for example, if the downscaling at reference numeral1206is amended from the process1200, the upscaling at reference numeral1214may also be omitted from the process1200. Generally, the upscaling at reference numeral1214is relied upon to increase the density of depth values in the depth map generated at reference numeral1208and smoothed at reference numeral1212. The relatively higher density of depth values may be relied upon in the focusing at reference numeral1218.

At reference numeral1216, the process1200includes receiving a point for focus. As discussed above, the system10and/or the device160may receive a point for focus using any suitable input means, such as by capacitive touch screen, mouse, keyboard, electronic pen, etc. At reference numeral1218, the process1200includes focusing one or more of the images captured at reference numeral1202according to the point for focus received at reference numeral1216, with reference to the depth values of the smoothed depth map generated at reference numeral1212. Further, at reference numeral1218, the process1200includes outputting a selectively focused image. For example, with reference toFIG. 9A, the focuser130may selectively focus the first image202by blending the blurred image250with the first image202, with reference to the relative depth values of the upscaled depth map228(or another depth map) and the point for focus140. Alternatively, with reference toFIG. 9B, the focuser130may focus the first image202by determining a point spread per pixel for one or more pixels of the first image202with reference to the relative depth values of the upscaled depth map228(or another depth map) and the point for focus140.

The process1200may be repeated, for example, for various points of focus. In other words, for each point of focus received at reference numeral1216, the process1200may generate a selectively focused output image. Further, it should be appreciated that the process1200may be performed using downscaled copies of certain images. That is, the process1200may be performed in a manner similar to the process flow illustrated inFIG. 10. Similarly, it should be appreciated that the process1200may be performed using certain components of images or image data, such as luminance data, chroma data, or combinations thereof, at various stages in the process flow. In other words, the process1200may be performed in a manner similar to the process flow illustrated inFIG. 11.

According to various aspects of the process1200, the process1200may be relied upon for focusing and re-focusing images after they are captured. For example, a luminance image provided by a secondary luminance sensor may relied upon, in connection with a luminance component of an image from a main color image sensor, to generate a depth map representative of relative depth values. The depth map may be used for focusing and re-focusing, for example, or for object extraction, scene understanding, gesture recognition, etc.

FIG. 13illustrates an example schematic block diagram of a computing architecture1300that may be employed as the processing environment100of the system10ofFIG. 1A, according to various embodiments described herein. The computing architecture1300may be embodied, in part, using one or more elements of a mixed general and/or specific purpose computer. The computing architecture1300includes a processor1310, a Random Access Memory (RAM)1320, a Read Only Memory (ROM)1330, a memory device1340, and an Input Output (I/O) interface1350. The elements of computing architecture1300are communicatively coupled via one or more local interfaces1302. The elements of the computing architecture1300are not intended to be limiting in nature, as the architecture may omit elements or include additional or alternative elements.

In various embodiments, the processor1310may include or be embodied as a general purpose arithmetic processor, a state machine, or an ASIC, for example. In various embodiments, the processing environment100ofFIGS. 1A and 1Bmay be implemented, at least in part, using a computing architecture1300including the processor1310. The processor1310may include one or more circuits, one or more microprocessors, ASICs, dedicated hardware, or any combination thereof. In certain aspects and embodiments, the processor1310is configured to execute one or more software modules which may be stored, for example, on the memory device1340. The software modules may configure the processor1310to perform the tasks undertaken by the elements of the computing environment100of the system10ofFIG. 1A, for example. In certain embodiments, the process1200described in connection withFIG. 12may be implemented or executed by the processor1310according to instructions stored on the memory device1340.

The RAM and ROM1320and1330may include or be embodied as any random access and read only memory devices that store computer-readable instructions to be executed by the processor1310. The memory device1340stores computer-readable instructions thereon that, when executed by the processor1310, direct the processor1310to execute various aspects of the embodiments described herein.

As a non-limiting example group, the memory device1340includes one or more non-transitory memory devices, such as an optical disc, a magnetic disc, a semiconductor memory (i.e., a semiconductor, floating gate, or similar flash based memory), a magnetic tape memory, a removable memory, combinations thereof, or any other known non-transitory memory device or means for storing computer-readable instructions. The I/O interface1350includes device input and output interfaces, such as keyboard, pointing device, display, communication, and/or other interfaces. The one or more local interfaces1302electrically and communicatively couples the processor1310, the RAM1320, the ROM1330, the memory device1340, and the I/O interface1350, so that data and instructions may be communicated among them.

In certain aspects, the processor1310is configured to retrieve computer-readable instructions and data stored on the memory device1340, the RAM1320, the ROM1330, and/or other storage means, and copy the computer-readable instructions to the RAM1320or the ROM1330for execution, for example. The processor1310is further configured to execute the computer-readable instructions to implement various aspects and features of the embodiments described herein. For example, the processor1310may be adapted or configured to execute the process1200described above in connection withFIG. 12. In embodiments where the processor1310includes a state machine or ASIC, the processor1310may include internal memory and registers for maintenance of data being processed.

The flowchart or process diagram ofFIG. 12is representative of certain processes, functionality, and operations of embodiments described herein. Each block may represent one or a combination of steps or executions in a process. Alternatively or additionally, each block may represent a module, segment, or portion of code that includes program instructions to implement the specified logical function(s). The program instructions may be embodied in the form of source code that includes human-readable statements written in a programming language or machine code that includes numerical instructions recognizable by a suitable execution system such as the processor1310. The machine code may be converted from the source code, etc. Further, each block may represent, or be connected with, a circuit or a number of interconnected circuits to implement a certain logical function or process step.

Although embodiments have been described herein in detail, the descriptions are by way of example. The features of the embodiments described herein are representative and, in alternative embodiments, certain features and elements may be added or omitted. Additionally, modifications to aspects of the embodiments described herein may be made by those skilled in the art without departing from the spirit and scope of the present invention defined in the following claims, the scope of which are to be accorded the broadest interpretation so as to encompass modifications and equivalent structures.