Patent ID: 12188759

DETAILED DESCRIPTION

To make the technical problems to be resolved by embodiments of the specification, and the technical solutions and beneficial effects of the embodiments of the specification clearer and more comprehensible, the following further describes the specification in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain the specification but does not limit the specification.

It should be noted that, when an element is described as being “fixed on” or “disposed on” another element, the element may be directly located on the another element, or indirectly located on the another element. When an element is described as being “connected to” another element, the element may be directly connected to the another element, or indirectly connected to the another element. In addition, the connection may be used for fixation or circuit connection.

It should be understood that orientation or position relationships indicated by the terms such as “length,” “width,” “above,” “below,” “front,” “back,” “left,” “right,” “vertical,” “horizontal” “top,” “bottom,” “inside,” and “outside” are based on orientation or position relationships shown in the accompanying drawings, and are used only for ease and brevity of illustration and description of embodiments of the specification, rather than indicating or implying that the mentioned apparatus or component needs to have a particular orientation or needs to be constructed and operated in a particular orientation. Therefore, such terms should not be construed as limiting the specification.

In addition, terms “first” and “second” are used merely for the purpose of description, and shall not be construed as indicating or implying relative importance or implying a quantity of indicated technical features. In view of this, a feature defined by “first” or “second” may explicitly or implicitly include one or more features. In the descriptions of the embodiments of the specification, unless otherwise specified, “a plurality of” means two or more than two.

FIG.1is a schematic diagram of a structure of a depth imaging device10based on temporal speckles, according to an embodiment of the specification. The depth imaging device10includes an emission module11, an acquisition module12, and a control and processing device13separately connected to the emission module11and the acquisition module12. The emission module11comprises a light emitting device and is configured to emit at least two speckle patterns that change temporally to a target object20. The acquisition module12comprises a light sensor and is configured to acquire the corresponding reflected speckle patterns of the at least two speckle patterns reflected by the target object20. The control and processing device13is configured to perform spatial-temporal stereo matching by using the reflected speckle patterns and at least two prestored reference speckle patterns to calculate offsets of pixel points between speckles of the at least two prestored reference speckle patterns and speckles of the reflected speckle patterns, and calculate depth values of the pixel points according to the offsets. It may be understood that a disparity for emitting the at least two speckle patterns may be preset.

The emission module11includes a light source111and an optical element112. The light source111may be a light source such as a light-emitting diode (LED), an edge-emitting laser (EEL), or a VCSEL, or may be an array light source including a plurality of light sources. Preferably, a plurality of VCSEL light sources are used to form a VCSEL array light source. Because the VCSEL has characteristics, such as a small volume, a small light source emission angle, and good stability, a plurality of VCSEL light sources can be simultaneously arranged on a semiconductor substrate. A VCSEL light source array chip formed in this way has a small volume and low power consumption, and is more conducive to generating spot pattern light beams. Moreover, because the plurality of VCSEL light sources have different spatial positions, speckle patterns corresponding to the plurality of VCSEL light sources are also different. In this way, speckle patterns that change temporally can be generated.

The light source array includes a plurality of discrete sub-light source arrays, and the discrete sub-light source arrays may be grouped and independently controlled. Through grouping, the control and processing device13independently controls the discrete sub-light source arrays of the emission module11to emit at least two speckle patterns that change temporally to the target object20. The grouping and controlling include that the sub-light source arrays are independently or synchronously turned on.

FIG.2is a schematic diagram of a VCSEL light source array, according to an embodiment of the specification. In the embodiment shown inFIG.2, a plurality of sub-light sources201(the light sources shown in hollow circles) jointly form a first sub-light source array. The first sub-light source array forms a first two-dimensional pattern, and can independently emit a first speckle pattern under the control of the control and processing device13. A plurality of sub-light sources202(the light sources shown in solid circles) jointly form a second sub-light source array. The second sub-light source array forms a second two-dimensional pattern, and can independently emit a second speckle pattern under the control of the control and processing device13(e.g., a processor). In addition, the first sub-light source array and the second sub-light source array are spatially separated. The hollow circles201and the solid circles202in the figure are only for distinguishing, and both the hollow circles201and the solid circles202are light sources. In addition, all the hollow circles201are jointly controlled, and all the solid circles202are jointly controlled. That is, sub-light source arrays represented by the solid circles and the hollow circles may be independently controlled. The first two-dimensional pattern and the second two-dimensional pattern may be the same or different. It may be understood that the first sub-light source array and the second sub-light source array may alternatively be synchronously turned on to form a third two-dimensional pattern. The description herein is merely exemplary. The light source array may include a plurality of sub-light source arrays, and the sub-light source arrays may be individually turned on or two or more of the sub-light source arrays may be jointly turned on.

It may be understood that, the plurality of sub-light source arrays may be arranged separately, crosswise, and compounded in space. For example, the first sub-light source array is in a region A, the second sub-light source array is in a region B, and a third sub-light source array is in a region A+B. In addition, an arrangement manner of the plurality of sub-light source arrays may be set according to requirements, and patterns, quantities, densities, and arrangement manners of the sub-light source arrays may be the same or different. For example, an arrangement density of first sub-light sources is higher than that of second sub-light sources, a quantity of the first sub-light sources is less than that of the second sub-light sources. Because different arrangement manners result in an output of different speckle patterns, speckle patterns that change temporally can be generated.

The optical element112receives light beams from the light source111, performs modulation such as diffraction or transmission on the light beams, and then emits modulated light beams to the target object20. The optical element may be one or a combination of two or more of forms such as a lens, a DOE, and a microlens array, and is set according to specific usage scenarios.

In another embodiment of the specification, the depth imaging device10includes a light beam deflection unit114(e.g., one or more actuators301explained below) connected to the emission module. The light beam deflection unit114may be a specific piece of hardware or a combination of pieces of hardware, and is configured to deflect light beams emitted from the light source array of the emission module to emit the speckle patterns that change temporally to the target object20. It may be understood that a deflection angle and a deflection time may be set according to specific requirements, and different speckle patterns that change temporally are generated by deflecting the emitted light beam. Details are as follows.

In an embodiment, the emission module11is connected to one or more actuators301. The actuator301may be connected to either of the VCSEL array and the lens or DOE, and is configured to move or deflect the VCSEL array or the lens or DOE in a single direction or a plurality of directions, thereby generating speckle patterns that change temporally.

As shown inFIG.3, the VCSEL array emits a light beam303. The lens310receives the light beam303and converges the light beam into a light beam304. The DOE312receives the light beam304and diffracts the light beam304into a zeroth-order diffracted light beam305, positive first-order diffracted light beams306a, and negative first-order diffracted light beams306bto form a speckle pattern. It may be understood that the zeroth-order diffracted light beam and the positive and negative first-order diffracted light beams are only used as an example herein for ease of description, and the DOE312may generate light beams into a larger quantity of diffraction orders. The actuator301is separately connected to the lens310and the control and processing device13, and is configured to translate the lens310transversely along an optical axis of the lens, for the light beam303to be translated or deflected.

As shown inFIG.4, the actuator301is configured to translate the lens310to the right (e.g., downward inFIG.4), and the light beam304rotates clockwise (as shown by an arrow402in the figure) for a specific angle θ to form a light beam401. The angle is determined according to a ratio of a translation amount of the lens to a focal length of the lens. The rotation continues to be transferred to the DOE312, and the zeroth-order diffracted light beam305, the positive first-order diffracted light beams306a, and negative first-order diffracted light beams306bof the DOE312also rotate clockwise (as shown by an arrow403in the figure) for the specific angle θ, for the speckle pattern to transversely move, thereby achieving a temporal change effect. It may be understood that a diffraction order fulcrum also moves transversely, and a movement amount of the fulcrum can be omitted when compared with the deflection angle. In this embodiment, only rightward translation of the lens is used as an example, and actually the lens may alternatively move to the left or in other directions. The movement direction of the lens is not limited herein.

It may be understood that the description herein is merely exemplary. The deflection unit includes the hardware or hardware combination that is capable of achieving a function similar to that of the actuator, such as a function of a scanning component and a function of a beam splitter, and the hardware or the hardware combination with a suitable size may be selected in consideration of specific application scenarios.

In another embodiment of the specification, the emission module11emits at least two speckle patterns that change temporally to the target object20, and the acquisition module12acquires the corresponding reflected speckle patterns reflected by the target object20. The control and processing device13matches the reflected speckle patterns and prestored reference speckle patterns to calculate offsets of pixel points, and calculates depth values of the pixel points according to the offsets. It may be understood that the plurality of prestored reference speckle patterns are acquired by using a calibration process. That is, a flat plate is placed at one or more preset distances, then a speckle pattern is projected onto the flat plate, and the acquisition module12acquires the speckle pattern and stores the speckle pattern in a memory (not shown).

FIG.5(a)is a schematic diagram of stereo matching in a general stereo matching technology, andFIG.5(b)is a schematic diagram of spatial-temporal stereo matching according to embodiments of the specification. The commonly used stereo matching technology in the existing technologies is to establish a point correspondence between a pair of stereo images, and three-dimensional coordinates of a corresponding point can be obtained by calculating a correspondence between pixels. Stereo correction is performed on the stereo images according to pre-calibrated camera parameters, so that search of the corresponding point only needs to be performed on a horizontal line on which the pixel point is located. As shown inFIG.5(a), the left part is a reference image, the right part is a captured target image, and an arrow in the figure is a search direction of the corresponding point. By means of this method, when texture features on a target surface are not rich enough, it is difficult to establish a correct point correspondence, and it becomes very difficult to obtain accurate stereo matching.

It may be understood that there are a plurality of methods for performing matching by using the captured speckle patterns and the prestored reference speckle patterns, such as a sum of squared differences (SSD) similarity method and an NCC matching method. The matching method is not limited herein. In this embodiment, a spatial-temporal stereo matching method is adopted by adding temporal information into a stereo matching method. By establishing a point correspondence between the reference image and the captured target image, a depth of the target object is calculated by using a triangulation principle. As shown inFIG.5(b), the left parts are reference images, the right parts are captured target images, and an arrow in the figure is a search direction of the corresponding point. The emission module11emits a plurality of speckle patterns that change temporally to the target object, and stereo matching is performed according to a spatial-temporal stereo matching principle by using three-dimensional windows. The three-dimensional windows contain rich image information. Therefore, even if a radius of a matching window is very small (for example, 5×5, or even 3×3), a compact disparity map can be obtained by using the NCC matching method. The NCC matching is to calculate a matching degree between a speckle pattern and a prestored reference speckle pattern based on an NCC measurement formula by using grayscales of the speckle pattern and the prestored reference speckle pattern. A formula of the method for calculating the NCC matching degree between the three-dimensional windows is as follows:

ncc⁡(x,y,d)=∑i=1K∑u,v∈Ω⁡(x,y)(Ii,O(u,v)-I¯O))⁢(Ii,R(u+d,v)-I¯R)∑Ki=1∑u,v∈Ω⁡(x,y)(Ii,O(u,v)-I¯O))2⁢∑i=1K(Ii,R(u+d,v)-I_R)2(1)
where d is a disparity value, i is an image sequence index, ncc(x, y, d) represents an NCC value between image blocks in K different temporal speckle images centered around a pixel point (x, y) in reference speckle images Ii,Rand image blocks in K different temporal speckle images centered around a pixel point (x-d, y) in captured speckle images Ii,O, Ω(x, y) is a neighborhood centered around (x, y), and ÏOand ÏRare respectively average pixel grayscale values in three-dimensional windows of the reference speckle images and the captured speckle images.

To improve a matching speed, a pyramid search strategy is adopted, to implement coarse-to-fine matching. By using a three-layer pyramid, a width or height of an image at a current layer is ½ of a width or height of an image at a next layer. Mismatching points are eliminated through bidirectional matching. That is, it is assumed that a corresponding point PO is found in a captured target speckle image for a pixel point PR in a reference speckle image, and then reverse matching is performed in the reference image for the point PO in the target captured speckle image, to obtain a corresponding point PR1. The process is considered as mismatching if (PR−PR1≤1) is not met

A disparity map obtained based on NCC is of a pixel level, and to obtain matching accuracy of a sub-pixel level, two pixels that are obtained based on NCC and that are at a center of matched positions are subdivided, where a subdivision interval is 0.1 pixel to obtain 21 positions. Subsequently, the 21 positions are searched for a position with a highest NCC score by using NCC similarities, and the position with the highest NCC score is a final sub-pixel matching result. By using this method, matching accuracy of 1/10 pixel can be obtained theoretically.

As shown inFIG.6, the emission module11emits speckle patterns to the target object, and the acquisition module12acquires the reflected speckle patterns reflected by the target object. After a disparity map is obtained through speckle matching, the depth values of the pixel points can be calculated according to disparities of the pixel points in the disparity map by using a triangulation method, and an expression is as follows:

Z=Z01+Z0f×b×d
where d is a disparity value, b is a length of a baseline from a light source of the emission module to a camera of the acquisition module, Z0is a distance between a plane on which the emission module and the acquisition module are located and a reference plane, f is a focal length of the camera, and Z is a depth value of a pixel point. It may be understood that the emission module and the acquisition module are generally arranged on the same baseline, and a plane on which the emission module and the acquisition module are located is actually a plane on which the baseline is located.

It may be understood that in the foregoing embodiment, a depth image of the target object20is calculated by acquiring a plurality of frames of images. For example, the emission module11sequentially emits a temporal speckle image A and a temporal speckle image B to the target object20, that is, the two temporal speckle images are temporally emitted to the target object20in an order of A1, B1, A2, B2, A3, and B3. If a depth image D is outputted by A+B sequentially, that is, a frame of the depth image is calculated based on A1and B1, a frame of image is calculated based on A2and B2, and a frame of image is calculated based on A3and B3. There are three frames of images in total. Therefore, a frame rate of the depth image is reduced by half compared with a frame rate for acquisition of the speckle image. However, in an embodiment, a front-back-frame sequential manner may be used. In this way, a quantity of acquisition frames will not be reduced. A front-back-frame sequential acquisition method according to an embodiment of the specification shown inFIG.7is to calculate a frame of the depth image based on a previous frame and a next frame. A frame of the depth image is calculated based on A1and B1, and then a frame of depth image is calculated based on B1and A2, and so on. The first frame A1has no corresponding depth image, but each subsequent frame of speckle image has a corresponding depth image, so that a measurement frame rate is not reduced.

It may be understood that the foregoing two methods are merely exemplarily described based on an improvement of the software of the control and processing units, and/or an improvement of a combination of the hardware and the software of the depth imaging device. Other methods that can implement the concept of the specification, that is, methods in which the emission module emits at least two speckle patterns that change temporally to a target object, temporal information is added based on a stereo matching method, and stereo matching is performed according to a spatial-temporal stereo matching principle by using three-dimensional windows, should all fall within the protection scope of the specification.

Based on the depth imaging device based on temporal speckles disclosed in the foregoing embodiments, the specification further provides a corresponding depth imaging method.FIG.8shows a depth imaging method based on temporal speckles, according to an embodiment of the specification. The method includes the following steps.

S1: Controlling an emission module to emit at least two speckle patterns that change temporally to a target object.

In an embodiment, the emission module includes a VCSEL array, a lens, and a DOE. Each sub-array in the VCSEL array emits a different speckle pattern, thereby generating temporal speckle patterns.

In another embodiment, the emission module includes a VCSEL array, a lens, a DOE, and an actuator. The actuator is connected to any one of the VCSEL array, the lens, or the DOE, to move the VCSEL array, the lens, or the DOE in a single direction or a plurality of directions, thereby generating speckle patterns that change temporally.

S2: Controlling an acquisition module to acquire the reflected speckle patterns of the at least two speckle patterns reflected by the target object.

S3: Performing spatial-temporal stereo matching by using the reflected speckle patterns and at least two prestored reference speckle patterns, to calculate offsets of pixel points between speckles of the at least two prestored reference speckle patterns and speckles of the reflected speckle patterns, and calculating depth values of the pixel points according to the offsets.

In an embodiment of the specification, the plurality of discrete sub-light source arrays in the emission module are controlled to be independently turned on or to be synchronously turned on, so that the at least two speckle patterns that change temporally are emitted to the target object. The specific implementation is as described above, and details are not described herein again.

In another embodiment of the specification, after a light beam emitted from the light source array of the emission module is controlled to be deflected, the at least two speckle patterns that change temporally are emitted to the target object. The specific implementation is as described above, and details are not described herein again.

The control and processing device performs matching on the reference speckle patterns and the captured speckle patterns by using an NCC matching method, to calculate to obtain a disparity map. An expression of the NCC matching calculation method is as follows:

ncc⁡(x,y,d)=∑i=1K∑u,v∈Ω⁡(x,y)(Ii,O(u,v)-I¯O))⁢(Ii,R(u+d,v)-I¯R)∑Ki=1∑u,v∈Ω⁡(x,y)(Ii,O(u,v)-I¯O))2⁢∑i=1K(Ii,R(u+d,v)-I_R)2
where d is a disparity value, i is an image sequence index, ncc(x, y, d) represents an NCC value between image blocks in K different temporal speckle images centered around a pixel point (x, y) in reference speckle images Ii,Rand image blocks in K different temporal speckle images centered around a pixel point (x-d, y) in captured speckle images Ii,O, Ω(x, y) is a neighborhood centered around (x, y), and ÏOare average pixel grayscale values in three-dimensional windows of the reference speckle images and ÏRare average pixel grayscale values in three-dimensional windows of the captured speckle images.

After the disparity map is obtained according to the speckle matching, the depth values of the pixel points can be calculated according to disparities of the pixel points in the disparity map by using a triangulation method, and an expression is as follows:

Z=Z01+Z0f×b×d
where d is a disparity value, b is a length of a baseline from a light source of the emission module to a camera of the acquisition module, Z0is a distance between a plane on which the emission module and the acquisition module are located and a reference plane, f is a focal length of the camera, and Z is a depth value of a pixel point.

An embodiment of the specification further provides a control device, including a processor and a storage medium configured to store a computer program, where the processor is configured to at least perform the foregoing method when executing the computer program.

An embodiment of the specification further provides a storage medium, configured to store a computer program, where the computer program, when being executed, at least performs the foregoing method.

An embodiment of the specification further provides a processor, where the processor, when executing a computer program, at least performs the foregoing method.

The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. The non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, a compact disc, or a compact disc ROM (CD-ROM); and the magnetic surface memory may be a magnetic disk storage or a magnetic tape storage. The volatile memory may be a random access memory (RAM), used as an external cache. Through exemplary but non-limitative descriptions, RAMs in lots of forms may be used, for example, a static RAM (SRAM), a synchronous SRAM (SSRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a SyncLink DRAM (SLDRAM), and a direct Rambus RAM (DR RAM). The storage medium according to this embodiment of the specification includes, but not limited to, above and any other suitable types of memories.

In the several embodiments provided in the specification, it should be understood that the disclosed system and method may be implemented in other manners. The described device embodiments are merely exemplary. For example, the unit division is merely logical function division and may be other division during actual implementation. For example, multiple units or components may be merged or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the components may be implemented through some interfaces, indirect couplings or communication connections between the devices or units, or electrical connections, mechanical connections, or connections in other forms.

The units described as separation parts may be or may not be physically separated. The part used as display unit may be or may not be a physical unit. That is, the units may be located in the same place, or may be distributed to a plurality of network units. Some or all of the units need to be selected according to actual requirements to implement the purpose of the solution of the embodiments.

In addition, functional units in the embodiments of the specification may be all integrated in a second processing unit, each unit is separately used as a unit, or two or more units are integrated in a unit. The integrated unit may be implemented in a form of hardware, or may be implemented in form of hardware plus a software functional unit.

A person of ordinary skill in the art may understand that all or some of the steps of the method embodiments may be implemented by a program instructing relevant hardware. The program may be stored in a computer-readable storage medium. When the program runs, the steps of the method embodiments are performed. The foregoing storage medium includes: any medium that can store program code, such as a portable storage device, a ROM, a RAM, a magnetic disk, or a compact disc.

Alternatively, when the integrated module in the specification is implemented in the form of a software functional module and sold or used as an independent product, the integrated module may be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the specification essentially, or the part contributing to the existing technologies may be implemented in a form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the methods described in the embodiments of the specification. The foregoing storage medium includes: any medium that can store program code, such as a portable storage device, a ROM, a RAM, a magnetic disk, or a compact disc.

The methods disclosed in the method embodiments provided in the specification may be combined with each other arbitrarily if there is no conflict, to obtain a new method embodiment.

The features disclosed in the product embodiments provided in the specification may be combined with each other arbitrarily if there is no conflict, to obtain a new product embodiment.

The features disclosed in the method or device embodiments provided in the specification may be combined with each other arbitrarily if there is no conflict, to obtain a new method embodiment or device embodiment.

The foregoing contents are detailed descriptions of the specification with reference to specific exemplary embodiments, and it should not be considered that the specific implementation of the specification is limited to these descriptions. A person skilled in the art, to which the specification belongs, may further make some equivalent replacements or obvious variations without departing from the concept of the specification. Performance or functions of the replacements or variations are the same as those in the specification, and all the replacements or variations should be considered as falling within the protection scope of the specification.