Zero order light removal in active sensing systems

A device for image processing includes an optical receiver configured to receive a reflection of a coded pattern from an object to generate an image, and processing circuitry. The processing circuitry is configured to determine an estimated position of zero order light in the image, determine a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern, map the spatial region to the estimated position of the zero order light in the image to generate a corrected image, and generate a depth map for the coded pattern based on the corrected image.

TECHNICAL FIELD

This disclosure relates to depth map generation and more particularly to depth map generation using an active depth sensing system.

BACKGROUND

Active depth sensing systems transmit and receive light patterns. The received light patterns may be used to generate a depth map for a scene. The farther away an object is from the transmitter and receiver, the closer the received light pattern projection is to its original position at the receiver(s), as the outgoing light pattern projection and reflected incoming light pattern projection are more parallel. Conversely, the closer an object is to the transmitter and receiver, the farther the received light pattern projection is from its original position at the receiver(s). Thus, the difference between a received and a transmitted light pattern position indicates the depth of an object in the scene. Active depth sensing system use these relative depths to generate a depth map, or a three dimensional representation of a scene.

SUMMARY

This disclosure describes example techniques of determining a depth map where a spatial region at a position of zero order light is mapped to a received image. For example, an optical transmitter transmits, towards an object, a coded pattern of light. In this example, an optical receiver receives a reflection from the object of the coded pattern. The received reflection may include zero order light which obscures one or more spatial regions at the position of the zero order light in the received image. In this example, processing circuitry maps the one or more spatial regions at the position of the zero order light using the coded pattern to generate the depth image.

In one example, the disclosure describes a method of image processing, the method comprising transmitting, with an optical transmitter, towards an object, a coded pattern of light, receiving, with an optical receiver, a reflection of the coded pattern from the object to generate an image, determining, with processing circuitry, an estimated position of zero order light in the image, determining, with the processing circuitry, a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern, mapping, with the processing circuitry, the spatial region to the estimated position of the zero order light in the image to generate a corrected image, and generating, with the processing circuitry, a depth map for the coded pattern based on the corrected image.

In one example, the disclosure describes a device for image processing, the device comprising an optical receiver configured to receive a reflection of a coded pattern from an object to generate an image, and processing circuitry. The processing circuitry is configured to determine an estimated position of zero order light in the image, determine a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern, map the spatial region to the estimated position of the zero order light in the image to generate a corrected image, and generate a depth map for the coded pattern based on the corrected image.

In one example, the disclosure describes a computer-readable storage medium including instructions stored thereon that when executed cause one or more processors of a device for image processing to cause an optical transmitter to transmit, towards an object, a coded pattern of light and cause an optical receiver to receive a reflection of the coded pattern from the object to generate an image. The instructions further cause processing circuitry to determine an estimated position of zero order light in the image, determine a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern, map the spatial region to the estimated position of the zero order light in the image to generate a corrected image, and generate a depth map for the coded pattern based on the corrected image.

In one example, the disclosure describes a device for image processing, the device comprising means for transmitting, towards an object, coded pattern of light, means for receiving a reflection of the coded pattern from the object to generate an image, means for determining an estimated position of zero order light in the image, the zero order light indicating undiffracted light output by the optical transmitter when transmitting the coded pattern, means for determining a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern, means for mapping the spatial region to the estimated position of the zero order light in the image to generate a corrected image, and means for generating a depth map for the coded pattern based on the corrected image.

DETAILED DESCRIPTION

To generate images that a user perceives to encompass a three-dimensional space, a device may generate a depth map of the scene or object in the images to be rendered. One way to generate the depth map is in an active sensing system, also referred to as a verged active stereo system. In the coded pattern system, a transmitter device projects a coded pattern on a scene and a receiver device receives the coded pattern to obtain a depth map. For instance, the transmitter device transmits a coded pattern of light that includes the coded pattern on to the scene or object, and the receiver device receives a reflection of the coded pattern from the scene or object.

Then, based on a position of where a receiver received the coded pattern, the receiver device may determine an estimate of the distance of the scene or object from the receiver. Based on the determined distances, the receiver device generates a depth map. A processing circuit (which may be a programmable and/or fixed function processing circuit) may then use the generated depth map to generate graphical data for one or more images (e.g., a graphics processing circuit (GPU) uses the depth map to generate stereoscopic images).

A portion of the coded pattern, referred to herein as “zero order light,” projected to the scene or object may be undiffracted. Such undiffracted, zero order light saturates a brightness level at the optical receiver, which obscures a coded pattern at a spatial region positioned at the zero order light. As such, processing circuitry may not decode the coded pattern at a spatial region where zero order light is received. Moreover, processing circuitry generating a depth map may generate inaccurate depth values at the position of the zero order light due to the lack of decodable information at a spatial region at the position of the received zero order light.

The techniques described in this disclosure provide a way to determine a spatial region of a coded pattern that corresponds to a position of the zero order light in the coded pattern. A coded pattern may include, for example, but not limited to, one or more codewords, a pseudo random pattern of dots, or other information. With the techniques described in this disclosure, processing circuitry may generate a depth map using the coded pattern that correctly specifies depth values at a position of the zero order light, which results in a more accurate depth map compared to systems that do not account zero order light.

FIG. 1is a conceptual diagram illustrating an example transmitter and receiver for generating a depth map.FIG. 1illustrates device10that includes processing circuitry12, transmitter device14that is coupled to optical transmitter16, and receiver device18that is coupled to optical receiver20. Examples of device10include a desktop computer, a laptop computer, a tablet, a wireless communication device, a phone, a television, a camera, a display device, a digital media player, a video game console, a video gaming console, or a video streaming device.

Examples of transmitter device14and receiver device18include a microprocessor, an integrated circuit, a digital signal processor (DSP), a field programmable gate array (FPGA), or application specific integrated circuit (ASIC). In general, transmitter device14and receiver device18include processing circuitry12including programmable circuitry. Examples of optical transmitter16include a laser, and examples of optical receiver20include one or more optical sensors. In some examples, the laser outputs light (i.e., the depth map) in the infrared spectrum and the sensor receives the light (i.e., the depth map) in the infrared spectrum.

Although optical transmitter16is illustrated as part of transmitter device14and optical receiver20is illustrated as part of receiver device18, the techniques described in this disclosure are not so limited. In some examples, transmitter device14and receiver device18may not include respective ones of optical transmitter16and optical receiver20. In some examples, transmitter device14and receiver device18may be formed in the same integrated circuit along with other processing circuits forming a system on chip (SoC).

Transmitter device14may be configured to cause optical transmitter16to transmit a coded pattern of light. For instance, transmitter device14may include a local memory that stores a coded pattern used for depth map generation. In some examples, the coded pattern may be pseudo-random. In some examples, the coded pattern may be a predetermined pattern of codewords. A processing circuit of transmitter device14retrieves a coded pattern and causes optical transmitter16to transmit the coded pattern. The coded pattern reflects from objects and is received, through a lens or aperture, as a coded pattern reflection by optical receiver20.

The reflections of the coded pattern are captured at different locations on optical receiver20. For instance, assume that a first object is a first distance away from device10, and a second object is a second distance away from device10. In this example, the coded pattern that reflects off of the first object would appear at a first location on optical receiver20and the coded pattern that reflects off of the second object would appear at a second location on optical receiver20. In this example, the disparity between the first location and the second location (e.g., the difference in the positions of the first location and the second location) indicates the relative depth of the first and second objects to one another and the positions of the first location and the second location indicate the absolute depth of the first and second objects.

In some examples, the further away an object is from optical transmitter16and optical receiver20, the closer the received projected coded pattern is from its original position at optical receiver20(e.g., the outgoing projection and incoming projection are more parallel). Conversely, the closer an object is from optical transmitter16and optical receiver20, the further the received projected coded pattern is from its original position at optical receiver20. Thus, the difference between received and transmitted coded pattern may be used as an indicator of the depth of an object. In one example, such depth (e.g., relative depth) may provide a depth value for objects depicted by each pixel or grouped pixels (e.g., regions of two or more pixels) in an image.

The coded pattern may be considered as including a plurality of elements, where the elements (e.g., dots, codewords, etc.) in the coded pattern together form the coded pattern. Each element in the coded pattern is located at a transmitting location in the coded pattern transmitted by optical transmitter16at the time of transmission and then located at a receiving location by optical receiver20. Receiver device18may include a local memory that stores a coded pattern used for depth map generation. A processing circuit of receiver device18compares the elements of the received coded pattern to those stored in the local memory to determine the depth map. The processing circuit within the receiver device18determines a disparity (e.g., difference) between the location of each spatial region in the transmitted a coded pattern of light and the received reflected a coded pattern of light and determines the depth map based on the disparity.

FIG. 2is conceptual diagram illustrating an example a of a scanning application that uses a depth map202. For example, optical transmitter16transmits, towards 3D object208, a coded pattern of light. In this example, optical receiver20receives a reflection of the coded pattern from 3D208object to generate an image and processing circuitry12generates depth map202for the coded pattern using the image according to techniques described herein for accounting for zero order light. In the example ofFIG. 2, processing circuitry12uses depth map202to generate 3D point cloud204, which may include a set of point clouds used, for example, to create 3D models. Processing circuitry12or other processing circuitry may use 3D point cloud204for a variety of applications, for example, 3D printing. For example, reconstructed image206illustrates a 3D reconstruction model of the 3D object that may be generated using 3D point cloud204.

FIG. 3is a conceptual diagram illustrating an example of depth mapping. In the example ofFIG. 3, optical receiver20generates image302. In this example, processing circuitry12generates depth map304using image302according to techniques described herein for accounting for zero order light. For example, optical receiver20may generate image302using a disparity between transmitting locations and receiving locations of the coded pattern.

FIG. 4is a conceptual diagram illustrating example applications for depth mapping. In the example ofFIG. 4, processing circuitry12may apply Bokeh and/or segmentation techniques to generate image402. For instance, processing circuitry12may blur features of image402according to a depth map generated according to techniques described herein for accounting for zero order light. In another example, processing circuitry12may perform face liveness detection techniques. For instance, processing circuitry12may generate a 3D model404according to a depth map generated according to techniques described herein for accounting for zero order light. In this instance, processing circuitry12or other processing circuitry12may determine whether the 3D model404corresponds to a known object (e.g. face).

FIG. 5is a conceptual diagram illustrating examples of zero order light at different scales. The zero order light indicates undiffracted light output by the optical transmitter when transmitting the coded pattern.FIG. 5illustrates first scale510, second scale512, third scale514, and fourth scale516. In the example ofFIG. 5, a hand is relatively close to device10for first scale510and is moved further away from device10for second scale512, even further away from device10for third scale514, and even further from device10for fourth scale516.

InFIG. 5, optical receiver20generates an image at first scale510with zero order light502. Zero order light502may be the undiffracted light output by optical transmitter16. In the example ofFIG. 5, zero order light502may saturate optical receiver20such that optical receiver20may not decode a spatial region within a coded pattern positioned underneath zero order light502. As shown, at first scale510, zero order light502relatively large. As shown, at second scale512, zero order light504is smaller than zero order light502. Similarly, zero order light506is smaller than zero order light504and zero order light508is smaller than zero order light506. Moreover, a position of the zero order light changes with distance. For example, a position of zero order light502in first scale510along the horizontal direction (e.g., left to right) is different from a position of zero order light504in second scale512, a position of zero order light506in third scale514, and a position of zero order light508in fourth scale516.

FIG. 6is a conceptual diagram illustrating the effects of zero order light on depth mapping. In the example ofFIG. 6, zero order light602of coded image610may saturate optical receiver20such that optical receiver20may not decode a spatial region within a coded pattern positioned underneath zero order light602. As such, processing circuitry12may generate depth map612that includes inaccurate depth values604, also referred to herein as a “hole.” Inaccurate depth values604may obscure identifiable features in a depth map. This may interfere with certain application that use depth maps. For example, facial recognition application may be adversely affected. For example, in facial recognition applications, depth maps should not include holes on features such as noses, lips, or eyes. Using one or more techniques described herein for accounting for the zero order light602from the coded image610, processing circuitry12may instead generate depth map614that includes accurate depth values606. As shown, depth map614does not include holes, which makes depth map614more suitable for applications, such as, for example, but not limited to, facial recognition, 3D printing, or other applications.

FIG. 7is a conceptual diagram illustrating techniques for determining a probable location window for the zero order light. In some examples, processing circuitry12may use the probable location window for the zero order light to provide a search range for determining the estimated position of the zero order light. However, in some examples, processing circuitry12may directly estimate estimated position of the zero order light without using a probable location window. As used herein, a probable location window may refer to a region of an image that comprises a zero order light for multiple distances between optical transmitter16and the object reflecting the coded pattern. In some examples, the probable location window may be indicative of a region of an image that is likely to comprise the zero order light.

Based on system parameters of device10(e.g., focal length of optical receiver20, baseline, etc.), processing circuitry12may determine probable location window704of image702. As shown, probable location window704of image702extends from inner horizontal edge710to outer horizontal edge712. Processing circuitry12may determine probable location window704based on one or more of a minimum estimated distance between optical transmitter16and an object reflecting the coded pattern, a maximum estimated distance between optical transmitter16and the object reflecting the coded pattern, and a focal length of optical receiver20. As used herein, an estimated distance may be, for example, a derived distance, a calculated distance, or another estimated distance.

For example, processing circuitry12may calculate an inner horizontal edge of the probable location window using the minimum estimated distance between the optical transmitter and the object reflecting the coded pattern. For instance, processing circuitry12may calculate

Δ⁢⁢dmin=fBZ⁢⁢min,
wherein Δdminis inner horizontal edge710of probable location window704, f is the focal length of optical receiver20, and Zminis the minimum estimated distance between optical transmitter16and the object reflecting the coded pattern.

Processing circuitry12may calculate an outer horizontal edge of the probable location window using the maximum estimated distance between the optical transmitter and the object reflecting the coded pattern. For example, processing circuitry12may calculate

Δ⁢⁢dmax=fBZ⁢⁢max,
wherein Δdmaxis outer horizontal edge712of probable location window704and Zmaxis the maximum estimated distance between optical transmitter16and the object reflecting the coded pattern.

FIG. 8is a conceptual diagram illustrating filters that are matched to a size of zero order light. The filters ofFIG. 8are shown for example purposes only and processing circuitry12may apply other filters to account for zero order light. In the example ofFIG. 8, processing circuitry12may apply a filter within a probable location window of an image to estimate estimated position of the zero order. However, in some examples, processing circuitry12may apply a filter directly to an image to determine an estimated position of the zero order without using a probable location window.

In the example ofFIG. 8, the multiscale blob detection may refer to intensity802of a first filter, intensity804of a second filter, intensity806of a third filter, intensity808of a fourth filter, and intensity810of a fifth filter. In the example ofFIG. 8, each intensity of intensities802-810may represent a respective target size of zero order light, which is described further withFIG. 9. Processing circuitry12may apply filters ofFIG. 8to probable location window704ofFIG. 7to determine an estimated position of zero order light. As used herein, an estimated position may be, for example, a derived position, a calculated position, or another estimated position.

In the example ofFIG. 8, processing circuitry12applies a “multiscale blob” detection algorithm to determine an estimated position of the zero order. For example, processing circuitry12may apply multiscale blob detection by applying multiple filters and determining the estimated position of the zero order using the filter that best matches the zero order light. However, in other examples, other techniques may be used to determine an estimated position of the zero order.

FIG. 9is a chart illustrating the filters ofFIG. 8. As shown, intensity902corresponds to intensity802ofFIG. 8, intensity904corresponds to intensity804ofFIG. 8, intensity906corresponds to intensity806ofFIG. 8, intensity908corresponds to intensity808ofFIG. 8, and intensity910corresponds to intensity810ofFIG. 8. The intensity of the filters shown inFIG. 9is for example purposes only and processing circuitry12may apply other filters with other intensities to account for zero order light. In response to determining that a particular intensity of intensities902-910best matches zero order light in a received a coded pattern of light, processing circuitry12may select the filter corresponding to the particular intensity that best matches the zero order light.

FIG. 10is a conceptual diagram illustrating a difference of Gaussian and a Laplacian of Gaussian for the filters ofFIG. 8. As shown, Laplacian of Gaussian of the intensity802ofFIG. 8and difference of Gaussian1002(which overlaps Laplacian of Gaussian) corresponds to intensity802ofFIG. 8, Laplacian of Gaussian and difference of Gaussian1004(which overlaps Laplacian of Gaussian) corresponds to intensity804ofFIG. 8, Laplacian of Gaussian and difference of Gaussian1006(which overlaps Laplacian of Gaussian) corresponds to intensity806ofFIG. 8, Laplacian of Gaussian and difference of Gaussian1008(which overlaps Laplacian of Gaussian) corresponds to intensity808ofFIG. 8, and Laplacian of Gaussian and difference of Gaussian1010(which overlaps Laplacian of Gaussian) corresponds to intensity810ofFIG. 8.

Because of the similarities in Laplacian of Gaussian and difference of Gaussian, processing circuitry12may select a filter using a difference of Gaussian, which may be simpler to calculate compared to difference of Laplacian. Again, while the example ofFIG. 10may apply multiscale blob detection using example filters, ofFIG. 8processing circuitry12may select a filter using a difference of Gaussian instead of a difference of Laplacian when applying other filter techniques and/or other filters.

In some examples, using a difference of Gaussian allows for a separable filter kernel and/or real-time detection. Processing circuitry12may convolve a set of filters represented by the difference of Gaussian1002, difference of Gaussian1004, difference of Gaussian1006, difference of Gaussian1008, and difference of Gaussian1010with the input image. Again, while the example ofFIG. 10may apply multiscale blob detection, processing circuitry12may apply a separable filter kernel and/or real-time detection using a difference of Gaussian instead of a difference of Laplacian when applying other filter techniques.

In this example, processing circuitry12may estimate a location and size of a zero order using the maximum response of the set of filters. In this manner, processing circuitry12may determine an estimated position of zero order light in a received image using a probable location window (e.g., probable location window704ofFIG. 7) and filters (e.g., filters described inFIGS. 7-10or other filters). While the example ofFIG. 10applies difference of Gaussian, in some examples, processing circuitry12may apply other techniques (e.g., Laplacian of Gaussian) to select a filter.

FIG. 11is a conceptual diagram illustrating techniques for interpolating a spatial region within a coded pattern at a position of received zero order light. In this example, in response to determining the position of zero order light of image1102, processing circuitry12may interpolate missing a spatial region within a coded pattern using surrounding depth points to generate corrected image1104. For example, processing circuitry12may apply a median filter to average depth values.

FIG. 12is a conceptual diagram illustrating techniques for mapping a spatial region within a coded pattern of light at a position of zero order light. In the example ofFIG. 12, processing circuitry12has access to a coded pattern1210, which was used by optical transmitter device14to transmit a coded pattern of light. In some examples, the spatial region may include a codeword (e.g., an all ‘1’ codeword, a particular arbitrary codeword, etc.) of the coded pattern of light. As such, processing circuitry12may access the coded pattern1210to determine the spatial region associated with the position X0, which may be known because of an a priori structure of the coded pattern1210.

Due to a highly structured nature of the coded pattern, the position of the zero-order light may always be located in a spatial region of coded pattern1210corresponding to a particular code word, referred to herein as codeword 1 or simply “c1”. Because of the a priori knowledge of c1 in the coded pattern1210, processing circuitry12may omit reconstructing the zero order occlusion using surrounding regions (e.g., the mapping process ofFIG. 11) and instead determine a spatial region of a coded pattern of light that corresponds to the position of the zero ordered light to generate missing depth information directly without interpolation filtering.

For example, a codeword associated with the position of zero order light may be set to an all ‘1’ codeword in a predetermined pattern of codewords. Processing circuitry12may be configured to map the spatial region to the estimated position of the zero order light using the all ‘1’ codeword. For instance, optical transmitter16may transmit the coded pattern with the all ‘1’ codeword (e.g., c1) at the position of the zero order light and processing circuitry12may be configured to map the all ‘1’ codeword at the estimated position of the zero order light in an image received with optical receiver20. For example, processing circuitry12may map codewords associated with the position X0(e.g., the position of the zero order light in the received structed image) to the all ‘1’ codeword in the predetermined pattern of codewords.

While the above example included an all ‘1’ codeword (e.g., c1) at the position of the zero order light, in some examples, the codeword (e.g., c1) at the position of the zero order light may be an arbitrary codeword. In this example, processing circuitry12may be configured to map the spatial region to the estimated position of the zero order light using the arbitrary codeword. For instance, optical transmitter16may transmit the coded pattern with a particular arbitrary codeword (e.g., c1) at the position of the zero order light and processing circuitry12may map the particular arbitrary codeword to the position of the zero order light in an image received with optical receiver20. As used herein, an arbitrary codeword may be based on a random choice. For example, an arbitrary codeword may be random, pseudo-random, or otherwise based on a random choice. In any case, processing circuitry12may determine codewords associated with the position of zero order light using the predetermined codeword to zero-order relationship, which may help to simplify region filling.

Mapping a spatial region to an estimated position of the zero order light may be computational simpler than using interpolating techniques (SeeFIG. 11) to estimate the codewords, which may help to reduce an energy consumption of device10compared to devices configured to use interpolation techniques. Moreover, mapping codewords to the predetermined pattern of codewords may be more accurate compared to codewords estimated using interpolating techniques, which may result in device10generating a more accurate depth map compared to devices configured to use interpolation techniques.

Processing circuitry12may determine a disparity between a transmitting location of a spatial region in the coded pattern and a receiving location of the spatial region in the corrected image. For example, processing circuitry12may generate a depth map based on the disparity. For instance, processing circuitry12may determine the disparity between the transmitting location of the spatial region and the receiving location1220of the spatial region1222based on a period for the spatial region. In some examples, processing circuitry12may calculate

d=mod⁡(X02-c⁢⁢1,P),
wherein d is the disparity between the transmitting location of the spatial region and the receiving location1220of the spatial region1222, X0is the estimated position of the of zero order light in the image, c1 is the spatial region of the coded pattern that corresponds to the position of the zero order light in the coded pattern (e.g., an all ‘1’ codeword), and P is a period for the spatial region. In this way, processing circuitry12may determine a spatial region that is obscured by zero order light without the need for additional processing using the surrounding region, which may help to reduce an energy consumption of device10compared to devices configured to use interpolation techniques.

FIG. 13is a flow chart of a method of image processing for performing one or more example techniques described in this disclosure.FIG. 13is discussed with reference toFIGS. 1-12for example purposes only. Optical transmitter16transmits, towards an object, a coded pattern of light (1302). Optical receiver20receives a reflection of the coded pattern from the object to generate an image (1304). Processing circuitry12determines an estimated position of zero order light in the image (1306). Processing circuitry12determines a spatial region of the coded pattern that corresponds to a position of the zero order light in the coded pattern (1308). Processing circuitry12maps the spatial region to the estimated position of the zero order light in the image to generate a corrected image (1310). Processing circuitry12generates a depth map for the coded pattern based on the corrected image (1312).

In some examples, processing circuitry12may generate a digital representation of a scene based on the depth map. For example, processing circuitry12or other processing circuitry may generate, using the depth map, a 3D print cloud to a create 3D model. In some examples, processing circuitry12or other processing circuitry may apply, using the depth map, Bokeh and/or segmentation techniques to generate an image with out-of-focus portions and in-focus portions for display. In another example, processing circuitry12may generate, using the depth map, a 3D model that may be used to determine whether a 3D model corresponds to a known object (e.g. face).

While some examples herein output a scene as a single object, in some examples, processing circuitry12may generate the digital representation to include a plurality of objects. For example, processing circuitry12or other processing circuitry may generate, using the depth map, a first 3D print cloud for a first object to a create a first 3D model for the first object and generate, using the depth map, a second 3D print cloud for a second object to a create a second 3D model for the second object. In some examples, processing circuitry12or other processing circuitry may apply, using the depth map, Bokeh and/or segmentation techniques to generate a first image with out-of-focus portions for a first object (e.g., a background object) and a second image with in-focus portions for a second object (e.g., a foreground object). In another example, processing circuitry12may generate, using the depth map, a first 3D model for a first object (e.g., a nose) and a second object (e.g., lips) that may be used to determine whether a 3D model corresponds to a known object (e.g. face).

In some examples, processing circuitry12may output the digital representation to a display. For instance, processing circuitry12may output an image with out-of-focus portions to a display (e.g., display58ofFIG. 14.). However, in some examples, processing circuitry12may output the digital representation to an internal application (e.g., for portrait mode, green screening, creating a model for manipulation and/or gaming, semantic understanding of a scene, etc.).

FIG. 14is a block diagram illustrating transmitter device14and receiver device18ofFIG. 1in greater detail.FIG. 14is a block diagram of a device for image processing configured to perform one or more example techniques described in this disclosure.FIG. 14illustrates device10in more detail. As described above, examples of device10include a personal computer, a head-mounted display (e.g., augmented reality, virtual reality, etc.), a desktop computer, a laptop computer, a computer workstation, a video game platform or console, a wireless communication device (such as, e.g., a mobile telephone, a cellular telephone, a table computer, a satellite telephone, and/or a mobile telephone handset), a landline telephone, an Internet telephone, a handheld device such as a portable video game device or a personal digital assistant (PDA), a personal music player, a video player, a display device, a camera, a television, a television set-top box, a server, an intermediate network device, a mainframe computer or any other type of device that processes and/or displays graphical data.

As illustrated in the example ofFIG. 14, device10includes transmitter device14that includes optical transmitter16, receiver device18that includes optical receiver20, a central processing circuit (CPU)45, a graphical processing circuit (GPU)48and local memory50of GPU48, user interface52, memory controller54that provides access to system memory60, and display interface56that outputs signals that cause graphical data to be displayed on display58.

Transmitter device14and receiver device18are similar to those described above with respect toFIG. 1and are not described further. However, in some examples, receiver device18may also function as a camera for device10, and in such examples, receiver device18may be used for depth map generation and for capturing photographic images or device10may include a separate camera to capture photographic images. In this disclosure, receiver device18is described as being used for both generating the depth map and capturing photographic images. The processing circuit of receiver device18may function as a camera processor as well.

Also, although the various components are illustrated as separate components, in some examples the components may be combined to form a system on chip (SoC). As an example, the processing circuit of receiver device18may be formed with one or more of CPU45, GPU48, and display interface56. In such examples, optical receiver20may be separate from receiver device18. Furthermore, the examples described above with respect to the processing circuit of receiver device18generating the depth map are provided merely to ease understanding. In some examples, CPU45, GPU48, or some other device may be configured to perform the examples described above for the processing circuit of receiver device18.

The various components illustrated inFIG. 14may be formed in one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuitry. Also, transmitter device14and receiver device18may include local memory for storage of data such as coded patterns. Examples of such local memory include one or more volatile or non-volatile memories or storage devices, such as, e.g., random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data media or an optical storage media.

The various units illustrated inFIG. 14communicate with each other using bus62. Bus62may be any of a variety of bus structures, such as a third generation bus (e.g., a HyperTransport bus or an InfiniBand bus), a second generation bus (e.g., an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXentisible Interface (AXI) bus) or another type of bus or device interconnect. It should be noted that the specific configuration of buses and communication interfaces between the different components shown inFIG. 14is merely example, and other configurations of computing devices and/or other image processing systems with the same or different components may be used to implement the techniques of this disclosure.

CPU45may comprise a general-purpose or a special-purpose processor that controls operation of device10. A user may provide input to computing device10to cause CPU45to execute one or more software applications. The software applications that execute on CPU45may include, for example, an operating system, a word processor application, an email application, a spread sheet application, a media player application, a video game application, a graphical user interface application or another program. The user may provide input to computing device10via one or more input devices (not shown) such as a keyboard, a mouse, a microphone, a touch pad or another input device that is coupled to computing device10via user interface52.

As one example, the user may execute an application that generates graphical data for stereoscopic images. The application may use images captured by optical receiver20. In such examples, transmitter device14and receiver device18may together perform the example techniques described in this disclosure to generate a depth map. The application executing on CPU45may use the depth map and the captured images.

For instance, CPU45may transmit instructions and data to GPU48to render graphical images. In such examples, the application executing on CPU45may transmit instructions, the depth map, and other data to GPU48instructing GPU48to generate stereoscopic images. For example, GPU48includes a plurality of parallel pipelines which are a combination of fixed-function circuits and programmable circuits, and GPU48processes pixels through the parallel pipelines to generate the stereoscopic images.

Memory controller54facilitates the transfer of data going into and out of system memory60. For example, memory controller54may receive memory read and write commands, and service such commands with respect to system memory60in order to provide memory services for the components in computing device10. Memory controller54is communicatively coupled to system memory60. Although memory controller54is illustrated in the example computing device10ofFIG. 14as being a processing module that is separate from both CPU45and system memory60, in other examples, some or all of the functionality of memory controller54may be implemented on one or both of CPU45and system memory60.

System memory60may store program modules and/or instructions and/or data that are accessible by transmitter device14, receiver device18, CPU45, and GPU48. For example, system memory60may store user applications and graphics data associated with the applications. System memory60may additionally store information for use by and/or generated by other components of computing device10. For example, system memory60may act as a device memory for transmitter device14and receiver device18(e.g., device memory for the camera processor of receiver device18). System memory60may include one or more volatile or non-volatile memories or storage devices, such as, for example, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data media or an optical storage media.

In some aspects, system memory60may include instructions that cause transmitter device14, receiver device18, CPU45, GPU48, and display interface56to perform the functions ascribed in this disclosure to transmitter device14, receiver device18, CPU45, GPU48, and display interface56. For example, system memory60may include zero removal module61configured to cause device10to determine a position of zero order light and to map a spatial region at the position of the zero order. Accordingly, system memory60may be a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors (e.g., processing circuits of transmitter device14and/or receiver device18and CPU45, GPU48, and display interface56) to perform various functions.

In some examples, system memory60is a non-transitory storage medium. The term “non-transitory” indicates that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted to mean that system memory60is non-movable or that its contents are static. As one example, system memory60may be removed from device10, and moved to another device. As another example, memory, similar to system memory60, may be inserted into device10. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM).

Receiver device18, CPU45, and GPU48may store depth maps, image data, rendered image data, and the like in respective buffers that is allocated within system memory60. Display interface56may retrieve the data from system memory60and configure display58to display the image represented by the rendered image data. In some examples, display interface56may include a digital-to-analog converter (DAC) that is configured to convert the digital values retrieved from system memory60into an analog signal consumable by display58. In other examples, display interface56may pass the digital values directly to display58for processing.

Display58may include a monitor, a television, a projection device, a liquid crystal display (LCD), a plasma display panel, a light emitting diode (LED) array, a cathode ray tube (CRT) display, electronic paper, a surface-conduction electron-emitted display (SED), a laser television display, a nanocrystal display or another type of display unit. Display58may be integrated within computing device10. For instance, display58may be a screen of a mobile telephone handset or a tablet computer. Alternatively, display58may be a stand-alone device coupled to computing device10via a wired or wireless communications link. For instance, display58may be a computer monitor or flat panel display connected to a personal computer via a cable or wireless link.

FIG. 15is a conceptual diagram illustrating a received image with zero order light. In the example ofFIG. 15, image includes zero order light1504.FIG. 16is a conceptual diagram illustrating a depth map generated using the image with zero order light ofFIG. 15. In this example, processing circuitry12generates depth map1602to include hole1604.

FIG. 17is a conceptual diagram illustrating a depth map generated using the image with zero order light ofFIG. 15with spatial regions mapped at a position of the zero order light. In the example ofFIG. 17, processing circuitry12may perform one or more steps ofFIG. 13to generate a corrected image that removes the zero order light. In this example, processing circuitry12generates depth map1702. As shown, region1704, which corresponds to hole1604in depth map1602ofFIG. 16does not includes a hole.

Again, holes may cause processing circuitry12or other processing circuitry to obscure key features (e.g., nodal features), such as noses, lips, or eyes, which may be undesirable in applications such as, but not limited to, for example, facial recognition, 3D printing, Bokeh and/or segmentation techniques, face liveness detection techniques, or other applications.

For example, holes may obscure portions of a coded pattern of light, which may reduce a performance of facial recognition. Such a reduction in facial recognition performance may result in facial recognition failing to accurately recognize a user, thereby causing user frustration. In some examples, the reduction in facial recognition performance may cause facial recognition to incorrectly recognize objects as the user, thereby reducing a confidence threshold of the system designer. Reducing the confidence threshold of the system designer may prevent facial recognition from high security applications, such as, for example, but not limited to, mobile payments.

In another example, holes may obscure portions of a coded pattern of light, which may reduce a performance of 3D printing. For instance, the reduction in 3D printing performance may result in a 3D printing system using a 3D print cloud that includes incorrect depth information, thereby resulting in a 3D printing system failing to accurately print. In another example, holes may obscure portions of a coded pattern of light, which may reduce a performance of Bokeh and/or segmentation techniques. For instance, the reduction in Bokeh and/or segmentation techniques performance may result in processed images having foreground features blurred instead of only background features, resulting in user frustration. As such, depth map1702may be more suitable for applications, such as, but not limited to facial recognition, compared to depth map1602.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing circuit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media. In this manner, computer-readable media generally may correspond to tangible computer-readable storage media which is non-transitory. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be understood that computer-readable storage media and data storage media do not include carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.