Patent ID: 12249108

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention will now be described more specifically with reference to the following embodiments. It is to be noted that the following descriptions of preferred embodiments of this invention are presented herein for purpose of illustration and description only. It is not intended to be exhaustive or to be limited to the precise form disclosed.

The present invention provides a 3D image sensing device with a 3D image processing function and a 3D image processing method for the 3D image sensing device. A first embodiment of the 3D image sensing device and the 3D image processing method will be described as follows.

FIG.1is a schematic view illustrating the appearance of a 3D image sensing device according to an embodiment of the present invention. As shown inFIG.1, the 3D image sensing device1comprises a housing10, a first camera lens11, a second camera lens12and a light source13. The first camera lens11, the second camera lens12and the light source13are installed on the housing10. The 3D image sensing device1further comprises a processing unit (not shown), associated electronic components and associated circuitry structures, which are disposed within the housing10.

For example, the 3D image sensing device1is a mobile electronic device (e.g., a smart phone, a wearable device or a laptop), a smart car, a robot and or any other appropriate device that employs the 3D image sensing technology. The two camera lenses11and12are used to implement the stereo vision technology, and the two camera lenses and the light source13are cooperatively used to implement the active stereo vision technology.

In an embodiment, the two camera lenses (i.e., the first camera lens11and the second camera lens12) are used to photograph a target (not shown). For example, the first camera lens11is the left lens, and the second camera lens12is the right lens. The light source13can emit plural feature points to the target. After the images photographed according to the stereo vision technology and the active stereo vision technology are processed by the processing unit, the corresponding depth maps are generated. The principles and applications of the stereo vision technology and the active stereo vision technology are well known to those skilled in the art, and not redundantly described herein.

The 3D image processing method of the present invention is executed by a firmware component that is installed in the 3D image sensing device1. Alternatively, the 3D image processing method is written as operation program codes and stored in a flash memory of the 3D image sensing device1. After the operation program codes are accessed and executed by the processing unit, the 3D image processing method is operated.

Please refer toFIGS.2and3.FIG.2illustrates a flowchart of a 3D image processing method according to a first embodiment of the present invention.FIG.3schematically illustrates an implementation example of selecting and synthesizing the corresponding depth maps by using the 3D image processing method of the first embodiment.

Please refer toFIG.2. Firstly, in a step S11, a target is photographed by the first camera lens11and the second camera lens12, so that a first image and a second image are obtained. Then, in a step S12, the 3D image sensing device1is operated in a stereo vision mode to process the first image and the second image, so that a first depth map A1is obtained. Then, in a step S13, the light source13emits plural feature points to the target, and the target is photographed by the first camera lens11and the second camera lens12, so that a third image and a fourth image are obtained. Then, in a step S14, the 3D image sensing device1is operated in an active stereo vision mode to process the third image and the fourth image, so that a second depth map A2is obtained. Afterwards, in a step S15, the first depth map A1and the second depth map A2are selected and synthesized according to a synthetization strategy, so that a synthesized depth map B1is obtained.

In the steps S11and S12, the first image (not shown) is the left image, and the second image (not shown) is the right image. Moreover, in the stereo vision mode, the processing unit is operated according to the stereo vision technology, and the corresponding depth map (i.e., the first depth map A1as shown inFIG.3) is obtained after the associated calculation is performed on the parallax between the first image and the second image. In the first depth map A1, the depth data are presented in grayscale, and the part without depth data is displayed in white. The first depth map A1in this embodiment is presented for simple illustration only. That is, the first depth map A1has one type of depth, but is not limited thereto.

In the steps S13and S14, the light source13is a light-emitting diode (LED), a laser diode (LD) or a vertical cavity surface emitting laser (VCSEL). The light source13can emit plural feature points. The plural feature points are uniformly scattered on the target in order for facilitating recognition. Similarly, the third image (not shown) is the left image, and the fourth image (not shown) is the right image. In the active stereo vision mode, the processing unit is operated according to the active stereo vision technology, and the corresponding depth map (i.e., the second depth map A2as shown inFIG.3) is obtained after the associated calculation is performed on the parallax between the third image and the fourth image. The presentation method of the second depth map A2is identical to the presentation method of the first depth map A1.

In this embodiment, the steps S11˜S12of generating the first depth map A1according to the stereo vision technology are performed before the steps S13˜S14of generating the second depth map A2according to the active stereo vision technology. It is noted that the sequences of the steps S11˜S12and the steps S13˜S14are not restricted. In some other embodiments, the steps of generating the corresponding depth map according to the stereo vision technology are performed after the steps of generating the corresponding depth map according to the active stereo vision technology. Likely, the purpose of the 3D image processing method of the present invention can be achieved.

In practice, the first camera lens11and the second camera lens12photograph the target by using a static method or a dynamic method. In case that the target is photographed by using the static method, the acquired images (i.e., the first, second, third and fourth images) are static image frames. In case that the target is photographed by using the dynamic method, the acquired images (i.e., the first, second, third and fourth images) are streaming image frames. That is, each image frame can be processed into the corresponding depth map in real time.

As previously described in the prior art, the stereo vision technology and the active stereo vision technology can supplement each other to a considerable extent in the identification application. For example, if the target has many features, the stereo vision technology is suitable for identification, but the active stereo vision technology is not suitable for identification. Whereas, if the target has few features, the active stereo vision technology is suitable for identification, but the stereo vision technology is not suitable for identification. For example, since the edge features of the wire target are obvious, the stereo vision technology is suitable for photographing the target directly. Moreover, since the content of the flat target is simple, it is better to use the active stereo vision technology to photograph the feature points.

In accordance with a feature of the present invention, the first depth map A1and the second depth map A2are obtained according to different technologies. Moreover, the final result is not determined according to one depth map only. Since the final result is determined according to the combination of the first depth map A1and the second depth map A2, the information in the first depth map A1and the information in the second depth map A2can be effectively utilized.

In the step S15, the first depth map A1and the second depth map A2are selected and synthesized according to the synthetization strategy. Particularly, the target is photographed by the two camera lenses11and12at the same viewing angle when the 3D image sensing device1is operated in the stereo vision mode and the active stereo vision mode. Consequently, the first depth map A1has a first range R1, and the second depth map A2has a second range R2. The first range R1and the second range R2are related to the same spatial range. In other words, the first depth map A1and the second depth map A2have the same dimension. Any coordinate (or any point) in the first depth map A1and the corresponding coordinate in the second depth map A2are related to the identical actual photographed point. For example, as shown inFIG.3, a specified location P1in the first depth map A1and a specified location P1′ in the second depth map A2have the same coordinate. In other words, the locations P1and P1′ are related to the same actual photographed point.

As mentioned above, the synthetization strategy of the present invention is specially defined. In accordance with the synthetization strategy, the depth data corresponding to each of the same locations of the two depth maps A1and A2are checked, and then the qualified depth data are selected from the first depth map A1and/or the second depth map A2. Consequently, the selected depth data of the first depth map A1and the selected depth data of the second depth map A2are synthesized as the synthesized depth map B1(seeFIG.3). For example, the procedure of checking the depth data of the specified location P1and the depth data of the specified location P1′ by the processing unit is repeatedly done. After the depth data of all locations of the first depth map A1and the depth data of the corresponding locations of the second depth map A2are checked, the selected depth data of the first depth map A1and the selected depth data of the second depth map A2are synthesized as the synthesized depth map B1by the processing unit.

In this embodiment, if the corresponding location of the first depth map A1or the second depth map A2contains the depth data, it means that the information about the distance between the 3D image sensing device1and the target is obtained. Whereas, if the corresponding location of the first depth map A1or the second depth map A2does not contain the depth data, it means that the depth map has a vacant site. That is, the information about the distance between the 3D image sensing device1and the target cannot be obtained.

In other words, the synthetization strategy in the step S15includes the following situations.

In a first situation, a specified location of the first depth map A1has a depth data, but the corresponding location of the second depth map A2has no depth data. In this situation, the corresponding depth data of the first depth map A1is selected.

In a second situation, a specified location of the first depth map A1has no depth data, but the corresponding location of the second depth map A2has a depth data. In this situation, the corresponding depth data of the second depth map A2is selected.

As mentioned above, one of the first depth map A1and the second depth map A2is selected in the first situation or the second situation of the synthetization strategy. That is, if a specified point has the depth data in one of the first depth map A1and the second depth map A2but the specified point has no depth data in the other of the first depth map A1and the second depth map A2, the depth map with the depth data is selected. There are two possible scenarios in the first situation or the second situation. In the first scenario, the depth data corresponding to the edges, the contours, the patterns or other sites with obvious features can be obtained according to the stereo vision technology, but the depth map of obtained according to the active stereo vision technology has vacant sites. In the second scenario, the depth data corresponding to the flat surfaces, the curved surfaces or other sites without obvious features can be obtained according to the active stereo vision technology, but the depth map of obtained according to the stereo vision technology has vacant sites.

In a third situation, a specified location of the first depth map A1has a depth data, and the corresponding location of the second depth map A2has a depth data. In this situation, the corresponding depth data of the first depth map A1and the corresponding depth data of the second depth map A2are subjected to an average calculating operation. According to this selecting method, if one point contains depth data in both of the first depth map A1and the second depth map A2, the two depth maps A1and A2are both selected. In case that the features are neither very obvious nor very non-obvious, the depth can be obtained according to both of the stereo vision technology and the active stereo vision technology.

As mentioned above, in the third situation, the depth can be obtained according to both of the stereo vision technology and the active stereo vision technology. Since the distance of the target obtained according to the stereo vision technology and the distance of the target obtained according to the active stereo vision technology are not much different, the use of the average calculating operation to process the depth data is feasible. In an embodiment, the average calculating operation is an arithmetic average calculating operation. That is, the final result of the arithmetic average calculating operation is the summed average of the depth data obtained according to the stereo vision technology and the depth data obtained according to the active stereo vision technology (i.e., directly added and then divided by 2).

The average calculating operation is not restricted to the arithmetic average calculating operation. For example, in some other embodiments, the average calculating operation is a geometric average calculating operation or a weighted average calculating operation. The final result of the geometric average calculating operation is the multiplicated average of the depth data obtained according to the stereo vision technology and the depth data obtained according to the active stereo vision technology (i.e., directly multiplied and then the square root is found). The final result of the weighted average calculating operation is determined according to weights and the final result of the arithmetic average calculating operation. The weights is determined according to the practical requirements. Consequently, the final result of the weighted average calculating operation is close to the depth data of one depth map.

In a fourth situation, a specified location of the first depth map A1has no depth data, and the corresponding location of the second depth map A2has no depth data. In this situation, the selecting action is not done. That is, if a specified point has no depth data in both of the first depth map A1and the second depth map A2, the selecting action is not performed on the depth maps A1and A2. This situation may be related to the sensitivity of the device or the photographing effect of the actual scene, and the depth data cannot be obtained according to these two technologies. In the 3D imaging sensing applications, there are still some situations that cannot be overcome by the stereo vision technology or the active stereo vision technology.

As mentioned above, in the fourth situation, the selecting action and the image processing action are not performed. Consequently, the corresponding location of the synthesized depth map B1has a vacant site. It is noted that this situation is seldomly generated. In practice, the stereo vision technology and the active stereo vision technology can effectively supplement each other in the identification application.

As mentioned above, the synthetization strategy in the step S15includes four situations. After the depth data of all locations of the first depth map A1and the depth data of the corresponding locations of the second depth map A2are checked, selected or further calculated according to the synthetization strategy, the synthesized depth map B1is obtained. In other words, the synthesized depth map B1contains the depth data that can best reflect the actual distance. That is, the depth data of the two depth maps A1and A2can be effectively used.

In the above embodiment, the target is photographed under a general ambient light source. It is noted that numerous modifications and alterations may be made while retaining the teachings of the invention. For example, in another embodiment, the 3D image sensing device1further comprises an illumination unit14for providing supplementary light. As shown inFIG.1, the light source13and the illumination unit14are installed on the housing10. If the intensity of the light beam from the ambient light source where the target is located is weak or the distance from the target is relatively long, the illumination unit14is enabled. After the illumination unit14is enabled to provide supplementary light, the target is photographed by the first camera lens11and the second camera lens12.

A second embodiment of a 3D image sensing device with a 3D image processing function and a 3D image processing method for the 3D image sensing device will be described as follows.FIGS.4A and4Billustrate a flowchart of a 3D image processing method according to a second embodiment of the present invention.

Please refer toFIGS.4A and4B. Firstly, in a step S21, a target is photographed by the first camera lens11and the second camera lens12, so that a first image and a second image are obtained. Then, in a step S211, an edge detection process is performed on the first image and the second image, so that a first edge detection result is generated. Then, a step S212is performed to judge whether the first edge detection result contains an edge feature. If the judging result of the step S212indicates that the first edge detection result contains any edge feature, the 3D image sensing device1is operated in a stereo vision mode to process the first image and the second image, so that a first depth map is obtained (Step S22). Whereas, if the judging result of the step S212indicates that the first edge detection result contains no edge feature, the illumination unit14is enabled (Step S213). After the illumination unit14is enabled to provide supplementary light, the target is photographed by the first camera lens11and the second camera lens12, so that an additional first image and an additional second image are obtained (Step S214). Then, in a step S215, the 3D image sensing device1is operated in the stereo vision mode to process the additional first image and the additional second image, so that an additional first depth map is obtained.

After the step S22or the step S215is completed, the light source13emits plural feature points to the target, and the target is photographed by the first camera lens11and the second camera lens12, so that a third image and a fourth image are obtained (Step S23). Then, in a step S231, the edge detection process is performed on the third image and the fourth image, so that a second edge detection result is generated. Then, a step S232is performed to judge whether the second edge detection result contains an edge feature. If the judging result of the step S232indicates that the second edge detection result contains any edge feature, the 3D image sensing device1is operated in an active stereo vision mode to process the third image and the fourth image, so that a second depth map is obtained (Step S24).

Whereas, if the judging result of the step S232indicates that the second edge detection result contains no edge feature, the illumination unit14is enabled (Step S233). After the illumination unit14is enabled to provide supplementary light and the light source13emits plural feature points to the target, the target is photographed by the first camera lens11and the second camera lens12, so that an additional third image and an additional fourth image are obtained (Step S234). Then, in a step S235, the 3D image sensing device1is operated in the active stereo vision mode to process the additional third image and the additional fourth image, so that an additional second depth map is obtained. After the step S24or the step S235is completed, the first depth map (or the additional first depth map) and the second depth map (or the additional second depth map) are selected and synthesized according to a synthetization strategy, so that a synthesized depth map is obtained (Step S25).

The steps S21, S22, S23and S24in the flowchart ofFIGS.4A and4Bare respectively identical to the steps S11, S12, S13and S14in the flowchart ofFIG.2. In comparison with the first embodiment, the steps S211˜S215and the steps S231˜S235in the 3D image processing method of this embodiment are distinguished. In this embodiment, the left image and the right image obtained in the stereo vision mode and the active stereo vision mode are subjected to the edge detection process by the processing unit, and then the processing unit judges whether the edge detection result contains an edge feature. If the edge detection result contains any edge feature, the flowchart ofFIG.2is performed. If the edge detection result contains no edge feature, the steps S213˜S215or the steps S233˜S235are performed.

In this embodiment, the edge detection process is an edge extraction process such as a Sobel edge detection process or a Laplacian edge detection process.

If the intensity of the light beam from the ambient light source where the target is located is weak or the distance from the target is relatively long, the edge part or the pattern of the object shown in the image is not obvious, the edge detection process is performed or a high-frequency filtering process is performed. In the edge detection process, a gradient change of a mask window exceeding a threshold value indicates that there is at least one edge feature. If a low-frequency signal is removed and a high-frequency signal with edge information is retained after the high-frequency filtering process is performed, it means that there is at least one edge feature. If no edge feature is contained in the image, the 3D image sensing device in each of the stereo vision technology and the active stereo vision technology is unable to generate the depth map. For solving the above problems, the illumination unit14is enabled to provide supplementary light. Then, the target is photographed again, and the left image and the right image are obtained. Consequently, a better depth map can be obtained according to the left image and the right image.

The step S25is similar to the step S15ofFIG.2. That is, the depth map obtained in the step S22or the step S215and the depth map obtained in the step S24or the step S235are selected and synthesized. Consequently, regardless of whether the supplementary light is provided or not provided, the synthesized depth map can be generated. Generally, the provision of the supplementary light can overcome environmental factors to increase the sharpness of the captured image. That is, the depth sensing efficacy is enhanced, but the depth calculation is not affected.

In this embodiment, the illumination unit14is disabled after the step S215is completed. In other words, the third image and the fourth image are obtained when no supplementary light is provided. Under the condition that no supplementary light is provided, the processing unit judges whether the third image and the fourth image contain the edge features.

In practice, the illumination unit14is selectively in the enabled state (i.e., the supplementary light is provided) or the disabled state (i.e., the supplementary light is not provided). That is, in the flowchart ofFIG.4A, either the step S22or the steps S213˜S215will be performed. Similarly, in the flowchart ofFIG.4B, either the step S24or the steps S233˜S235will be performed. Consequently, the synthesized depth map is selected and synthesized in one of four possible combined conditions.

As mentioned above, the 3D image sensing device1can be operated in one of four possible combined conditions. In the first combined condition, the supplementary light is needed in the stereo vision mode, but the supplementary light is not needed in the active stereo vision mode. In the second combined condition, the supplementary light is needed in the active stereo vision mode, but the supplementary light is not needed in the stereo vision mode. In the third combined condition, the supplementary light is needed in the stereo vision mode, and the supplementary light is needed in the active stereo vision mode. In the fourth combined condition, the supplementary light is not needed in the stereo vision mode, and the supplementary light is not needed in the active stereo vision mode. In other words, the depth maps to be selected and synthesized in the step S25are determined according to the corresponding combined condition. For example, if the first depth map and the second depth map are selected and synthesized, it means that the flowchart ofFIG.2is carried out. On the other hand, if the supplementary light is provided, the first depth map and the additional second depth map are selected and synthesized, or the additional first depth map and the second depth map are selected and synthesized, or the additional first depth map and the additional second depth map are selected and synthesized.

It is noted that numerous modifications and alterations may be made while retaining the teachings of the invention. For example, in another embodiment, the illumination unit14is enabled to provide supplementary light in both of the stereo vision mode and the active stereo vision mode. Consequently, the steps S231˜S233may be omitted. In case that the illumination unit14is maintained in the enabled state, the step S234is performed immediately after the step S215is completed.

Consequently, the additional first depth map and the additional second depth map are selected and synthesized.

From the above descriptions, the present invention provides a 3D image sensing device with a 3D image processing function and a 3D image processing method for the 3D image sensing device. When compared with the conventional technologies, the 3D image sensing device and the 3D image processing method can well utilize the stereo vision technology and the active stereo vision technology. Consequently, the vacant sites in the synthesized depth map are largely reduced. Moreover, since the depth data that best reflects the actual distance is obtained, the identification result is more accurate.

In other words, the 3D image sensing device and the 3D image processing method of the present invention are capable of effectively overcoming the drawbacks of the conventional technologies and achieving the purposes of the present invention.

While the invention has been described in terms of what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention needs not be limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements included within the spirit and scope of the appended claims which are to be accorded with the broadest interpretation so as to encompass all such modifications and similar structures.