Patent Description:
Imaging devices that are structured light active sensing systems include a transmitter and a receiver configured to transmit (project) and receive patterns corresponding to spatial codes (or "codewords") to generate a disparity map that indicates the distance of one or more objects in a scene from the imaging device. The farther away an object in a scene is from the transmitter and the receiver, the closer a received codeword reflected from the object is from its original position (compared to the transmitted codeword) because a propagation path of the outgoing codeword and the reflected incoming codeword are more parallel. Conversely, the closer the object is to the transmitter and receiver, the farther the received codeword is from its original position in the transmitted codeword. Accordingly, the difference between the position of a received codeword and the corresponding transmitted codeword may be used to determine the depth of an object in a scene. Structured light active sensing systems may use these determined depths to generate a disparity map of a scene, which may be a three dimensional representation of the scene. Many applications may benefit from determining a disparity map of a scene, including image quality enhancement and computer vision techniques.

Each codeword may be represented by rows and columns of intensity values corresponding to symbols. For example, binary spatial codes may use zeros (<NUM>'s) and ones (<NUM>'s), corresponding to dark and bright intensity values, to represent a binary pattern. Other spatial codes may use more than two different intensity values corresponding to more than two symbols. Other spatial representations also may be used.

Generating a disparity map depends on detecting codewords. To detect codewords made up of an array of symbols, decoding filters may identify spatial boundaries for codewords and symbols, and classify symbols as, for example, "<NUM>" or "<NUM>" based on their intensity values. Decoding filters may use matched filters, corresponding to the set of harmonic basis functions used to define the set of possible codewords, to classify incoming basis functions. Therefore, disparity map accuracy depends on accurately receiving symbols, codewords, and/or basis functions.

Shadows near an object (for example, along one side of the object) can be present in the image due to the relative position of the projector, the object, and a background behind the object, resulting in spatial code gaps (outages) at these pixels (outage pixels). Outages may also come from surfaces with irregular textures (such as hair), object tilt with respect to the camera, or partial occlusions which cause shadow regions. Therefore, there is a need for methods and systems to reconstruct object boundaries in shadow regions. The reconstructed object boundaries could then be used to produce more accurate and complete disparity maps from structured light systems. '<NPL> describes how the depth data provided by Kinect is incomplete because of no-measured depth (NMD for short) pixels, so a preprocessing approach for depth map is necessary. '<NPL> describes a bi-layer inpainting method for synthesizing novel views from a single colour image and its corresponding depth map under the exemplar-based inpainting framework.

Patent document <CIT> discloses a method for analysing depth data in order to segment and identify objects in a scene, wherein blocks of pixels located in between edges detected in a depth map are clustered to identify 3D clusters. Said document also addresses the fact that artefacts may be generated in the depth map due to shadows of foreground objects. More precisely, areas with no depth value that are adjacent to an object are identified as shadows and marked as belonging to that object.

The scope of the present invention is defined by the scope of the appended claims. All embodiments which do not fall under the scope of the appended claims are examples which are useful to understand the invention, but do not form part of the present invention.

A summary of sample aspects of the disclosure follows. For convenience, one or more aspects of the disclosure may be referred to herein simply as "some aspects. " Methods and apparatuses or devices being disclosed herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, for example, as expressed by the claims which follow, its more prominent features will now be discussed briefly. After considering this discussion, and particularly after reading the section entitled "Detailed Description" one will understand how the features being described provide advantages that include efficient ways of object reconstruction in disparity maps using displaced shadow outlines resulting in fewer decoding errors.

Various features, aspects and advantages will become apparent from the description herein and drawings appended hereto, in which like reference symbols generally will identify corresponding aspects or components illustrated in the drawings. As a person of ordinary skill in the art will understand, aspects described or illustrated for an embodiment may be included in one or more other described or illustrated embodiments, if not impractical for the implementation or function of such an embodiment, unless otherwise stated.

The following detailed description is directed to certain specific embodiments. However, the methods and systems disclosed can be embodied in a multitude of different ways. It should be apparent that aspects herein may be embodied in a wide variety of forms and that any specific structure, function, or both being disclosed herein is merely representative. Aspects disclosed herein may be implemented independently of any other aspects. Two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented, or a method may be practiced, using any number of the aspects set forth herein.

Further, the systems and methods described herein may be implemented on a variety of different imaging systems and computing devices and systems. They may use general purpose or special purpose systems.

Structured light active sensing systems project spatial codes with an infrared projector and sense an image having the spatial codes reflected from the surface of an object with a camera to generate points of a disparity map of the object. Disparity maps provide relative depth information associated with the surface of the object. Shadows near an object (for example, along one side of the object) can be present in the image due to the relative position of the projector, the object, and a background behind the object, resulting in spatial code gaps (outages) at these pixels. Depth can't be directly calculated at pixel positions with code outages; instead, depths can only be estimated at these locations. For example, if there is a single pixel outage and all neighboring pixels in the disparity map are within a tolerance of the same depth, the depth at the pixel with the outage can be accurately estimated based on the neighboring depths. However, if the neighboring pixels have different depths, because the pixel with the code outage is at or near an object boundary, then the depth at the pixel with the outage may be inaccurate because foreground and background pixels at substantially different depths will be combined or averaged, resulting in inaccurate depth estimates. More accurate estimates would be possible in shadow regions if the object boundary can be estimated, so that depth estimates are based on only those pixels on the same side of the object boundary line are considered. Outages may also come from surfaces with irregular textures (such as hair), object tilt with respect to the camera, or partial occlusions. Where there are outages, it may be difficult to accurately classify symbols, codewords, and basis functions, resulting in inaccurate disparity maps with inaccurate object boundaries.

Existing methods and system to estimate disparity in the presence of shadows may not account for object shape and boundaries. The disclosed technology includes systems and methods to fill code gaps to produce more accurate and complete disparity maps. The method includes generating a disparity map of the object, detecting a first boundary of the object in the disparity map, identifying a shadow region in the disparity map adjoining the first boundary (the shadow region including pixels with codeword outages), determining a boundary of the shadow region, determining a width of the shadow region, displacing a representation of the shadow boundary towards the object by the width of the shadow region, the displaced representation forming a second boundary of the object, and changing the shape of the object in the disparity map based on the second boundary. This results in a more accurate representation of the object boundary for filling in outages in the disparity map, resulting in better defined object boundaries in the disparity map with more accurate depths on both sides of the boundaries.

<FIG> illustrates an example of an active sensing system <NUM> that generates three dimensional information, such as a depth map <NUM> (disparity map), from two dimensional images. The active sensing system <NUM> includes a transmitter <NUM> and a receiver <NUM>. The transmitter <NUM> projects light through a code mask to form a projected image <NUM>. A section <NUM> of projected image <NUM> includes a unique codeword <NUM> that is projected onto the scene <NUM>. The surface of an object or objects in the scene <NUM> is illuminated by spatial pattern <NUM>, which forms part of reflected image that is sensed by receiver <NUM>. Receiver <NUM> senses a portion <NUM> (segment) of the reflected image <NUM>, including unique codeword <NUM>, and compares the relative position of unique codeword <NUM> to other unique codewords in the code mask to determine depth information, for generating a depth map <NUM>, of the surface of object in scene <NUM>, as described below with regard to <FIG>. The receiver <NUM> forms a depth map <NUM> based on depth estimates over the surfaces of the objects in the scene, which reflect other identifiable codewords from other segments of reflected image <NUM>. Each segment <NUM> that is captured may be uniquely identifiable at the receiver <NUM> and its location relative to other segments ascertained from the known pattern of the coded mask. The receiver <NUM> may use pattern segmentation techniques to address distortion, decoding techniques to identify codes, and triangulation to ascertain orientation and/or depth. In an embodiment, the transmitter <NUM> may be an infrared transmitter. In an embodiment, a single housing may include both the transmitter <NUM> and the receiver <NUM>.

<FIG> illustrates another example of a system for active sensing to generate depth maps (disparity maps) and display three dimensional representations of scenes. An encoder/shape modulator <NUM> may generate a code mask which is then projected by a transmitter device <NUM> over a transmission channel <NUM>. The code mask may be projected onto a target (e.g., a scene) and the reflected light is captured by a receiver sensor <NUM> as a projected code mask image. The receiver sensor <NUM> (e.g., receiver <NUM> in <FIG>), captures the reflected image of the target, which segmentation/decoder <NUM> segments and decodes to determine depth information used to generate depth map <NUM>. The depth map <NUM> may then be used to present, generate, and/or provide a 3D image version of, for example, a person 210a, a living room 210b, or a person holding a camera 210c.

Active sensing relies on being able to recognize (at the receiver sensor <NUM> and/or segmentation/decoder <NUM>) spatial codes (e.g., codewords) from the code mask being projected by the transmitter device <NUM> on a scene. If a scene is too close to the transmitter and receiver, the surface of the scene may be angled or curved, a baseline reference plane <NUM> may be tilted, and the codes may be modified under an unknown affine transformation (e.g., rotation, skew, compression, elongation, etc.). One or more aspects or features described herein may be implemented within the exemplary environments of <FIG> and <FIG>.

<FIG> illustrates an example of how depth may be sensed for one or more objects in a scene. <FIG> shows a device <NUM> that illuminates two objects <NUM> and <NUM> with structured light as codeword projection <NUM>. The codeword projection <NUM> reflects from objects <NUM> and/or <NUM> and is received as a reflected codeword <NUM> on sensor plane <NUM>.

As illustrated in <FIG>, the device <NUM> projects codeword projection <NUM> through transmitter aperture <NUM> on lens plane <NUM>. The device <NUM> receives reflected light from objects <NUM> and/or <NUM> via receiver aperture <NUM> on lens plane <NUM> and focuses the received codeword reflection <NUM> on sensor plane <NUM>. Therefore, device <NUM> illustrates transmission and reflection of structured light in a single device. In some embodiments, the transmitter and receiver functions are performed by two separate devices.

The codeword projection <NUM> illuminates the object <NUM> as projected segment <NUM>', and illuminates the object <NUM> as projected segment <NUM>". When the projected segments <NUM>' and <NUM>'' are received by the device <NUM> through receiver aperture <NUM>, the reflected codeword <NUM> may show reflections generated from the object <NUM> at a first distance d1 and reflections generated from the object <NUM> at a second distance d2.

As illustrated in <FIG>, the object <NUM> is located closer to the device <NUM> (e.g., a first distance from the device <NUM>) and the projected segment <NUM>' appears at a distance d2 from its initial location. In contrast, the object <NUM> is located further away (e.g., a second distance from the device <NUM>), and the projected segment <NUM>" appears at a distance d1 from its initial location (where d1 < d2). That is, the further away an object is from the device <NUM>, the closer the received projected segment/portion/window is from its original position at the device <NUM> (e.g., the outgoing projection and incoming projection are more parallel). Conversely, the closer an object is from the device <NUM>, the further the received projected segment/portion/window is from its original position at the device <NUM>. Thus, the difference between received and transmitted codeword position may be used as an indicator of the depth of an object. In one example, such depth (e.g., relative depth) may provide a depth value for objects depicted by each pixel or grouped pixels (e.g., regions of two or more pixels) in an image.

Various types of modulation and coding schemes may be used to generate a codeword projection or code mask. These modulation and coding schemes include, for example, temporal coding, spatial coding, and direct codification.

In temporal coding, patterns are successively projected onto the measuring surface over time. This technique has high accuracy and resolution but is less suitable for dynamic scenes.

In spatial coding, information is encoded in a local neighborhood based on shapes and patterns. Pseudorandom codes may be based on De-Bruijn or M-arrays define the codebook of valid codewords (e.g., m-ary intensity or color modulation). Pattern segmentation may not be easily attained, for example, where the shapes and patterns are distorted.

In direct codification, both horizontal and vertical pixel coordinates are encoded. Modulation may be by a monotonic phase or an intensity waveform. However, this scheme may utilize a codebook that is larger than the codebook utilized for other methods. In most methods, received codewords (sensed codewords) may be correlated against a defined set of possible codewords (e.g., in a codebook). Thus, use of a small set of codewords (e.g., small codebook) may provide better performance than a larger codebook. Also, since a larger codebook results in smaller distances between codewords, additional errors may be experienced by implementations using larger codebooks.

Structured light patterns may be projected onto a scene by shining light through a codemask. Light projected through the codemask may contain one or more tessellated codemask primitives. Each codemask primitive may contain an array of spatial codes. A codebook or data structure may include the set of codes. Spatial codes, the codemask, and codemask primitives may be generated using basis functions. The periodicities of the basis functions may be chosen to meet the requirements for the aggregate pattern of Hermitian symmetry (for eliminating ghost images and simplifying manufacturing), minimum duty cycle (to ensure a minimum power per codeword), perfect window property (for optimum contour resolution and code packing for high resolution), and randomized shifting (for improved detection on object boundaries). A receiver may make use of the codebook and/or the attributes of the design intended to conform to the constraints when demodulating, decoding, and correcting errors in received patterns.

The size and corresponding resolution of the spatial codes corresponds to a physical spatial extent of a spatial code on a codemask. Size may correspond to the number of rows and columns in a matrix that represents each codeword. The smaller a codeword, the smaller an object that can be detected. For example, to detect and determine a depth difference between a button on a shirt and the shirt fabric, the codeword should be no larger than the size of the button. In some embodiments, each spatial code may occupy four rows and four columns. In some embodiments, the codes may occupy more or fewer rows and columns (rows x columns), to occupy, for example, 3x3, 4x4, 4x5, 5x5, 6x4, or 10x10 rows and columns.

The spatial representation of spatial codes corresponds to how each codeword element is patterned on the codemask and then projected onto a scene. For example, each codeword element may be represented using one or more dots, one or more line segments, one or more grids, some other shape, or some combination thereof.

A spatial code may include bright bits or portions (e.g., "<NUM>") transmitting light through the codemask and dark bits or portions (e.g., "<NUM>") not transmitting light through the codemask. The "duty cycle" of spatial codes corresponds to the percentage of bright bits or portions projecting light. For example, a spatial code for which <NUM>% of the bits or portions are bright has a lower duty cycle than one for which <NUM>% of the bits or portions are bright. Codewords with too low a duty cycle may be difficult to detect.

The "contour resolution" or "perfect window" characteristic of codes indicates that when a codeword is shifted by an amount, for example, a one-bit rotation, the resulting data represents another codeword.

<FIG> is a block diagram illustrating an example of a transmitter device <NUM> that configured to generate a composite code mask and/or project such composite code mask. The transmitter device <NUM> in this example includes a processing circuit <NUM> coupled to a memory/storage device <NUM> (memory device), an image projecting device <NUM>, and/or a tangible medium <NUM>. The transmitter device <NUM> may correspond to the codeword projection <NUM> and transmitter aperture <NUM> portions of device <NUM> discussed above with respect to <FIG>.

In a first example, the transmitter device <NUM> may include a tangible medium <NUM>. The tangible medium may define, include, and/or store a composite code mask <NUM>. The tangible medium <NUM> may be a diffractive optical element (DOE) that encodes the code mask, such that when light from a laser or other light source is projected through the DOE at, for example, a near infrared frequency, a codeword pattern image is projected from the transmitter <NUM>. The composite code mask <NUM> may include a code layer combined with a carrier layer. The code layer may include uniquely identifiable spatially-coded codewords defined by a plurality of symbols. The carrier layer may be independently ascertainable and distinct from the code layer. The carrier layer may include a plurality of reference objects that are robust to distortion upon projection. At least one of the code layer and carrier layer may be pre-shaped by a synthetic point spread function prior to projection.

In a second example, the processing circuit (or processor) <NUM> may include a code layer generator/selector <NUM>, a carrier layer generator/selector <NUM>, a composite code mask generator/selector <NUM> and/or a pre-shaping circuit <NUM>. The code layer generator/selector <NUM> may select a pre-stored code layer <NUM> and/or may generate such code layer. The carrier layer generator/selector <NUM> may select a pre-stored carrier layer <NUM> and/or may generate such carrier layer. The composite code mask generator/selector <NUM> may select a pre-stored composite code mask <NUM> and/or may combine the code layer <NUM> and carrier layer <NUM> to generate the composite code mask <NUM>. Optionally, the processing circuit <NUM> may include a pre-shaping circuit <NUM> that pre-shapes the composite code mask <NUM>, the code layer <NUM>, and/or the carrier layer <NUM>, to compensate for expected distortion in the channel through which the composite code mask <NUM> is to be projected.

In some implementations, a plurality of different code layers and/or carrier layers may be available, where each such carrier or code layers may be configured for different conditions (e.g., for objects at different distances, or different configurations between the transmitter device and receiver device). For instance, for objects within a first distance or range, a different combination of code and carrier layers may be used than for objects at a second distance or range, where the second distance is greater than the first distance. In another example, different combination of code and carrier layers may be used depending on the relative orientation of the transmitter device and receiver device.

The image projecting device <NUM> may serve to project the generated/selected composite code mask onto an object of interest. For instance, a laser or other light source (not shown) of the image projecting device <NUM> may be used to project the composite code mask onto the object of interest (e.g., through a projection channel). In one example, the composite code mask <NUM> may be projected in an infrared spectrum, so it may not be visible to the naked eye. Instead, a receiver sensor in the infrared spectrum range may be used to capture such projected composite code mask.

<FIG> is a block diagram illustrating an example of a receiver device <NUM> that is configured to receive a composite code mask reflected from an object and to determine be depth information from a composite code mask. The receiver device <NUM> may include a processing circuit <NUM> coupled to a memory/storage device and a receiver sensor <NUM> (e.g., an image capturing device <NUM>). In some aspects, the receiver device <NUM> illustrated in <FIG> may correspond to the receiver aperture <NUM> and codeword reflection <NUM> of device <NUM> discussed above with respect to <FIG>. In some embodiments, the receiver sensor <NUM> is an image capture device, for example, a camera.

The receiver sensor <NUM> may be configured to obtain at least a portion of a composite code mask projected on the surface of an object. For instance, the receiver sensor may capture an image of at least a portion of a composite code mask <NUM> projected on the surface of a target object. The composite code mask <NUM> may be defined by: (a) a code layer of uniquely identifiable spatially-coded codewords defined by a plurality of symbols, and (b) a carrier layer independently ascertainable and distinct from the code layer and including a plurality of reference objects that are robust to distortion upon projection. At least one of the code layer and carrier layer may have been pre-shaped by a synthetic point spread function prior to projection. In one example, the receiver sensor <NUM> may capture (sense) the composite code mask in the infrared spectrum.

Still referring to <FIG>, in some embodiments, the code layer may include n1 by n2 binary symbols, where n1 and n2 are integers greater than two. In the composite code mask, each symbol may be a line segment in one of two gray-scale shades distinct from the reference objects. The symbols of the code layer may be staggered in at least one dimension. The carrier layer reference objects may include a plurality of equally spaced reference stripes with a guard interval in between. The reference stripes and the guard interval may be of different widths. The width of each reference stripe relative to a guard interval width may be determined by an expected optical spreading of a transmitter device and/or a receiver device.

The processing circuit <NUM> may include a reference stripe detector circuit/module <NUM>, a distortion adjustment circuit/module <NUM>, a codeword identifier circuit/module <NUM>, a depth detection circuit/module <NUM>, and/or a depth map generation circuit/module <NUM>.

The reference stripe detector circuit/module <NUM> may be configured to detect reference stripes within the portion of the composite code mask. The distortion adjustment circuit/module <NUM> may be configured to adjust a distortion of the portion of the composite code mask based on an expected orientation of the reference stripes relative to an actual orientation of the reference stripes. The codeword identifier circuit/module <NUM> may be configured to obtain a codeword from a window defined within the portion of the composite code mask. The depth detection circuit/module <NUM> may be configured to obtain depth information for a surface portion of the target object corresponding to the window based on: (a) a single projection of the composite code mask, and (b) a displacement of the window relative to a known reference code mask.

The depth map generation circuit/module <NUM> may be configured to assemble a depth map for the object based on a plurality of codewords detected as different overlapping windows within the portion of the undistorted composite code mask.

<FIG> is a block diagram illustrating an embodiment of an apparatus configured to perform one or more of the error correction methods disclosed herein. Apparatus <NUM> includes a light emitter <NUM>, a light receiving element <NUM>, a processor <NUM>, and a memory <NUM>. The light emitter <NUM>, light receiving element <NUM>, processor <NUM>, and the memory <NUM> are operably connected via a bus <NUM>. In some aspects, the light receiving element <NUM> may correspond to the receiver device <NUM> discussed above with respect to <FIG>. In some aspects, the light emitter <NUM> may correspond to the transmitter device <NUM> discussed above with respect to <FIG>.

The memory <NUM> may store instructions that configure the processor <NUM> to perform one or more functions of the methods discussed herein. For example, instructions stored in the memory may configure the processor <NUM> to generate a disparity map of the object, detect a first boundary of the object in the disparity map, identify a shadow region in the disparity map adjoining the first boundary (the shadow region including pixels with codeword outages), determine a boundary of the shadow region, determine a width of the shadow region, displace a representation of the shadow boundary towards the object by the width of the shadow region, the displaced representation forming a second boundary of the object, and change the shape of the object in the disparity map based on the second boundary. Instructions stored in the memory may further configure the processor change a shape of the object in the disparity map based on the detected shadow region according to the method <NUM> discussed below.

<FIG> is a picture of an example of a code mask <NUM> with arrays of symbols corresponding to bright and dark spots. In this example, the bright spots correspond to "<NUM>" symbols. In other embodiments, each bright spot may represent a zero ("<NUM>") symbol and the dark spots may represent a one ("<NUM>") symbol. The bright spots are aligned in rows and columns, and separated by black guard intervals and guard bands that give structure to the projected codes and make it possible to determine spatial boundaries of individual symbols and codewords. Codewords occupy a rectangular spatial area that includes rows and columns of symbols. For example, a codeword may include sixteen symbols in four rows and four columns. The "<NUM>" symbols with bright spots are visible, but the "<NUM>" symbols with dark spots blend into the guard intervals and guard bands.

<FIG> is a diagram of a top view of an example scene <NUM> with an infrared projector <NUM> that projects codewords over a field of view <NUM> onto a scene with a rectangular object <NUM> and background <NUM>. A camera <NUM> with field of view <NUM> captures an image of object <NUM> and background <NUM> to generate a disparity map <NUM>.

<FIG> is an example of a disparity map <NUM> of the rectangular object <NUM> of <FIG>, its shadow, and background. Disparity map <NUM> includes object image <NUM> of object <NUM>, background image <NUM> of background <NUM>, and first shadow region <NUM>, and second shadow region <NUM>. First shadow region <NUM> and second shadow region <NUM> form to the right of the object <NUM> in this example because the camera <NUM> is horizontally aligned with, and positioned to the right of, the infrared projector <NUM>.

The rightmost boundary of first shadow region <NUM> has the same shape of the rightmost boundary of object <NUM>. Second shadow region <NUM> may be formed by an irregularity or texture of the surface of object <NUM>; it does not reflect the shape of the boundary of object <NUM>. The rightmost boundary <NUM> of the first shadow region <NUM>, which has the same shape as the rightmost boundary of object <NUM> (<FIG>); can be displaced (shifted) to the left by the number of pixels corresponding to the width (W) of the first shadow region <NUM> to coincide with the rightmost edge of the object <NUM> to correct for irregularity or texture of the surface of object <NUM> that causes second shadow region <NUM>.

The width (W) of the first shadow region <NUM> can be estimated by calculating a median, mode, or average number of adjacent shadow pixels (also referred to as "outage pixels"). Outliers may be excluded when determining width. Shadow pixels to the left of the displaced boundary correspond to the second shadow region <NUM> pixels that occur because of an irregularity or surface texture of the object <NUM>. These outage pixels may be "corrected" or filled in as object pixels. The disparity values of the object pixels may be approximated by interpolating disparity values of neighboring object pixel disparity values.

<FIG> is an example of a disparity map <NUM> that represents a man's head and shoulders, including outage pixels from a projected light shadow. An "outage pixel" is defined as a pixel for which a valid spatial code has not been received. The pixels <NUM> (represented with a first pattern) include the man's head and shoulders, with the brightest pixels at the tip of the man's nose because it is closest to the camera. The pixels <NUM> (represented by a second pattern) are illustrated as being darker because they are farther from the camera than the man in the foreground. The pixels <NUM> (represented by a third pattern) correspond to outage pixels without codes, for which depth has not been estimated. The code outages at the pixels <NUM> may correspond to the shadow region <NUM> to the right of the man's head in the image, under his chin <NUM>, towards the top of his head <NUM>, along his collar line <NUM>, and elsewhere along the perimeter <NUM> of the man's head and shoulders. The pixels <NUM> towards the top of the man's head <NUM> and at/near his chin <NUM> may be due to the texture of his hair and beard. Shadow region <NUM> includes outage pixels from the shadow formed by the man's head and shoulders (corresponding to first shadow region <NUM> of <FIG>), as well as outage pixels that occur because of an irregularity or surface texture (such as hair) of the man (corresponding to second shadow region <NUM> of <FIG>).

Depth (and disparity) can be calculated for both the pixels <NUM> in the foreground and the pixels <NUM> in the background by the structured light techniques described above with respect to <FIG>. However, depth can't be directly calculated at pixel positions with code outages; instead, depths can only be estimated at these locations. For example, if there is a single outage pixel (a single pixel lacking a depth estimate because a valid code was not received) and all neighboring pixels in the disparity map are within a tolerance of the same depth, the depth at the pixel with the outage can be estimated accurately based on the neighboring depths.

If the neighboring pixels have different depths, because the outage pixel is at or near an object boundary, then a depth estimate for the outage pixel based on the neighboring pixels may be inaccurate because foreground and background pixels at substantially different depths will be combined or averaged. This would tend to blur or low pass filter boundaries, resulting in less distinct object boundaries with less accurate depth estimates close to the boundary.

More accurate estimates would be possible in shadow regions if the object boundary is reconstructed so that depth estimates are based on only those pixels on the same side of the object boundary line are considered Therefore, the outage pixels can be filled in more accurately by first reconstructing the boundary of his head, and then estimating depths at each pixel on the object side of the boundary using known depths for neighboring object pixels, and estimating depths at each pixel on the background side of the boundary using known depths for neighboring background pixels.

<FIG> is an image with boundaries (edges) of the disparity map <NUM> of the man of <FIG>, including object/outage and outage/background boundaries. In the example of <FIG>, boundary image <NUM> includes object/outage boundary <NUM> between object pixels and outage pixels and outage/background boundary <NUM> between outage pixels and background pixels. Object pixels, outage pixels, and background pixels correspond to pixels <NUM>, pixels <NUM>, and pixels <NUM>, respectively, illustrated in <FIG>. The outage/background boundary <NUM> follows the contour of the man's head and left shoulder because shadow outlines correspond in shape to the object that formed the shadow.

In the example of <FIG>, the outage/background boundary <NUM> to the right of the man's head more closely corresponds to the shape of his head than the object/outage boundary of his head. The outage/background boundary for the man's head may be used to reconstruct the shape of the man's head by displacing the outage/background boundary by the width <NUM> of the shadow region so that the two boundaries overlap.

<FIG> illustrates the reconstructed object boundary <NUM> of the man's head <NUM> and shoulders <NUM> after displacing the outage/background boundary <NUM> by a width <NUM> of the shadow region. Once displaced, the outage/background boundary <NUM> forms a more accurate representation of the object boundary.

<FIG> is a disparity map <NUM> with estimates for outage pixels based on the reconstructed object boundary <NUM> of <FIG>. Depths for each outage pixel within the new object boundary <NUM> of the man's head <NUM> and shoulders <NUM> are estimated using neighboring object pixel depths (and not background pixel depths). Depths of outage pixels on the other side of the reconstructed boundary <NUM> are estimated using neighboring background pixel depths (and not object pixel depths). This results in a more accurate disparity map <NUM> than is possible without object boundary reconstruction.

<FIG> illustrates an example of a process <NUM> for reconstructing an object boundary in a disparity map.

At block <NUM>, process <NUM> projects codewords with an image processing device. Process <NUM> projects laser light through a codemask to project codewords onto a scene. The codemask has the same codewords, associated symbols, and are formed by the same harmonic basis functions as the codemask described above with respect to <FIG> and <FIG>. The codewords are continuously projected for a time interval. The projected codewords may be projected onto a scene, or objects in a scene. This may be performed, for example, by the image projecting device <NUM> of <FIG> or the light emitter <NUM> of <FIG>.

At block <NUM>, process <NUM> senses the projected codewords reflected from an object with a receiver including a sensor. The received codewords may be received in an image of the scene or objects in the scene. This may be performed by a receiver sensor <NUM> of <FIG>, or a sensor integrated with a light source for example, light receiving element <NUM> integrated with a light emitter <NUM> of <FIG>.

At block <NUM>, process <NUM> generates a disparity map. Process <NUM> may use structured light methods as described with regard to <FIG>, in which codeword displacements are used to generate depth information. Process <NUM> may generate disparity map information from a single structured light frame, or multiple structured light frames. This may be performed, for example, by processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> stores the disparity map with a memory device. This may be performed, for example by the memory storage device <NUM> of <FIG>, memory <NUM> of <FIG>, or memory/storage device <NUM> of <FIG>.

At block <NUM>, process <NUM> detects a first boundary of the object in the disparity map. Process <NUM> may determine the first boundary of the object by locating object pixels with a neighboring shadow (outage) pixel. The first boundary of the object may exclude outage pixels and shaded regions from within the bounded object, such as a single outage pixel (or small groups of pixels) surrounded by object pixels. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> identifies a shadow region in the disparity map adjoining the first boundary. Process <NUM> may include outage pixels without codewords or calculated depths in the shadow region. Process <NUM> may, for each row of the disparity map, include adjacent outage pixels between the first boundary and a background pixel in the shadow region. Process <NUM> may exclude object pixels and background pixels from the shadow region. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> determines a boundary of the shadow region. Each row with pixels in the shadow region includes adjacent outage pixel between the first boundary of the object and a boundary of the shadow region adjoining a background pixel. In various embodiments, process <NUM> determines the boundary of the shadow region by starting at the first boundary of an object (an object pixel on the boundary with the shadow region) and traversing the adjacent pixels in the shadow region until reaching the outage pixel that adjoins a background pixel, to determine the pixel in the shadow region boundary for each row, to determine the boundary of the shadow region. The shadow boundary may be determined to be where there is a transition between the invalid (outage) pixel and a background pixel. The width of the shadow in a single row is the number of adjacent outage pixels. For example, as discussed above with respect to <FIG>, the shadow pixels may include pixels from both first shadow region <NUM> and second shadow region <NUM>. The rightmost boundary of first shadow region <NUM> has the same shape of the rightmost boundary of rectangular object <NUM>. Second shadow region <NUM> is formed by an irregularity or texture of the surface of rectangular object <NUM>; it does not reflect the shape of the boundary of rectangular object <NUM>. The rightmost boundary of the first shadow region <NUM>, which has the same shape as the rightmost boundary of rectangular object <NUM>; can be displaced (shifted) to the left by the number of pixels corresponding to the width of the first shadow region <NUM> to coincide with the rightmost boundary of the rectangular object <NUM> to correct for irregularity or texture of the surface of rectangular object <NUM> that causes second shadow region <NUM>. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> determines a width of the shadow region. The width of the shadow region may be determined based on the number of adjacent outage pixels, which corresponds to the number of pixels between the first boundary and the boundary of the shadow region in each row. The widths of a plurality of rows may be combined using statistical techniques, such as determining a median, mode or mean width across multiple rows. Process <NUM> may exclude rows with large numbers of adjacent outage pixels within an object boundary after correction. In various embodiments, the width is adjusted to more closely align portions of the shadow region boundary to the first boundary for these rows. If an object is leaning forward, the width in its top rows is larger than in its bottom rows. This means that the shadow outline is displaced more on the top than on the bottom. In some embodiments, a shadow that occupies less than a threshold number of rows may be excluded from consideration. For example, if there are outages in one row, but not in adjoining rows, there may not be sufficient information to correct an object boundary. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> forms a second boundary of the object. This is done by displacing a representation of the shadow boundary towards the object by the width of the shadow region, the displaced representation forming a second boundary of the object. As discussed above with respect to <FIG>, the rightmost boundary of the rectangular object <NUM> is changed by displacing the rightmost boundary of the first shadow region <NUM> to the left by the width of the first shadow region <NUM>. For example, if the determined width is <NUM> pixels, and the column location of the shadow boundary in row <NUM> is at column <NUM>, corresponding to coordinate (<NUM>, <NUM>), then the displaced shadow boundary in row <NUM> is at column <NUM>, corresponding to coordinate (<NUM>, <NUM>). Once the rightmost boundary of the first shadow region <NUM> (shadow boundary) is displaced to the left by the determined width of the first shadow region <NUM>, this displaced boundary may largely coincide with the rightmost boundary of the rectangular object <NUM>. Shadow pixels to the left of the displaced boundary, for example, at coordinate (<NUM>, <NUM>), may correspond to the second shadow region <NUM> pixels that occur because of an irregularity or surface texture of the rectangular object <NUM>. In various embodiments, the displacement takes place in increments and the degree of alignment between the shadow boundary and the first boundary are maximized. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>.

At block <NUM>, process <NUM> changes the shape of the object in the disparity map based on the second boundary. Process <NUM> may change a shape of the object by adjusting its first boundary as detected in block <NUM>. In various embodiments, the first boundary is replaced by the boundary of the shadow region, and outage pixels between the two boundaries are assumed to be object pixels, while outage pixels not between the two pixels are assumed to be background pixels. Depths of the outage pixels between the two boundaries that are assumed to be object pixels may be estimated based on the depths of neighboring object pixels only (and not background pixel depths). Depths of the outage pixels not between the two pixels are assumed to be background pixels may be estimated based on the depths of neighboring background pixels only (and not object pixel depths). In various embodiments, process <NUM> smooths portions of the first boundary that do not adjoin the shadow region using disparity map values of adjoining object pixels In various embodiments, process <NUM> updates the disparity map based on the changed shape of the object, and stores the updated disparity map with a memory device. This may be performed, for example by the processing circuit <NUM> of <FIG> or processor <NUM> of <FIG>, and by the memory storage device <NUM> of <FIG>, memory <NUM> of <FIG>, or memory/storage device <NUM> of <FIG>.

The various operations of methods described above may be performed by any suitable means capable of performing the operations, for example, various hardware and/or software component(s), circuits, and/or module(s). Generally, any operations illustrated in the Figures may be performed by corresponding functional means capable of performing the operations.

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.

In one or more aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Thus, in some aspects computer readable medium may comprise non-transitory computer readable medium (e.g., tangible media).

Claim 1:
A structured light system (<NUM>), comprising:
an image projecting device (<NUM>) configured to project codewords (<NUM>);
a receiver device (<NUM>) including a sensor (<NUM>), the receiver device configured to sense the projected codewords reflected from an object (<NUM>);
a processor configured to:
generate a disparity map (<NUM>) of the object from the sensed codewords,
detect a first boundary (<NUM>) of the object in the disparity map,
identify a shadow region (<NUM>) in the disparity map adjoining the first boundary, the shadow region including pixels with codeword outages,
determine a boundary of the shadow region (<NUM>), wherein the boundary of the shadow region is a boundary between the shadow and the background,
determine a width (<NUM>) of the shadow region, wherein the first boundary of the object is a boundary between the object and the shadow region,
determine a second boundary of the object in the disparity map, wherein the second boundary of the object is computed by displacing the boundary of the shadow by the width of the shadow toward the first boundary of the object, said second boundary representing a more accurate representation of the object boundary and
change the shape of the object in the disparity map based on the second boundary; and
a memory device configured to store the disparity map