Patent Description:
A structured light 3D system consists of a projecting device, an image capturing device and a processing device. The processing device generates a depth map according to a reference image associated with the projecting device and a captured image from the image capturing device. The image capturing device and the projecting device are sensitive to changes in the temperature and the component warm-up. The thermal effect leads to the distortion in the projected image of the projecting device and the pixel drifts in the captured image and the reference image. Thus, how to handle the thermal compensations on the projecting device and the image capturing device to obtain an accurate depth map is an important problem to be solved. <CIT> discloses a method and apparatus for performing temperature compensation for thermal distortions in a camera system. <CIT> discloses a high-accuracy three-dimensional measurement by performing an appropriate calibration in accordance with various temperature changes. <CIT> discloses a medium for temperature drift correction of a depth camera based on structured light.

This in mind, the present invention aims at providing a system and method for handling a thermal compensation to solve the abovementioned problem.

This is achieved by a system and method for handling a thermal compensation according to the independent claims, respectively. The dependent claims pertain to corresponding further developments and improvements.

As will be seen more clearly from the detailed description following below, a claimed system for handling a thermal compensation comprises: an image capturing device comprising a capturing circuit, configured for capturing a first image, and a first sensing circuit, configured for detecting a first temperature of the image capturing device; a projecting device comprising a second sensing circuit, configured for detecting a second temperature of the projecting device; a storage device, configured for storing a plurality of first parameters associated with the image capturing device, a plurality of second parameters associated with the projecting device and a reference image associated with the projecting device; and a processing device comprising a processing circuit, configured for compensating the first image according to the first temperature and the plurality of first parameters, to generate a first compensated image, compensating the reference image according to the second temperature and the plurality of second parameters, to generate a second compensated image, and generating a second image according to the first compensated image and the second compensated image.

A claimed method for handling a thermal compensation comprises: capturing a first image; detecting a first temperature of an image capturing device; detecting a second temperature of a projecting device; storing a plurality of first parameters associated with the image capturing device, a plurality of second parameters associated with the projecting device and a reference image associated with the projecting device; compensating the first image according to the first temperature and the plurality of first parameters, to generate a first compensated image; compensating the reference image according to the second temperature and the plurality of second parameters, to generate a second compensated image; and generating a second image according to the first compensated image and the second compensated image.

In the following, the invention is further illustrated by way of example, taking reference to the following drawings.

<FIG> is a schematic diagram of a system <NUM> according to an embodiment of the present invention. The system <NUM> may be a structured light 3D system. In <FIG>, the system <NUM> comprises an image capturing device <NUM>, a projecting device <NUM>, a storage device <NUM> and a processing device <NUM>. The image capturing device <NUM> comprises a capturing circuit <NUM> and a first sensing circuit <NUM>. The capturing circuit <NUM> is configured for capturing a first image. The first sensing circuit <NUM> is configured for detecting (e.g., measuring) a first temperature of the image capturing device <NUM>. The projecting device <NUM> comprises a second sensing circuit <NUM>. The second sensing circuit <NUM> is configured for detecting (e.g., measuring) a second temperature of the projecting device <NUM>. The storage device <NUM> is configured for storing a plurality of first parameters associated with the image capturing device <NUM>, a plurality of second parameters associated with the projecting device <NUM> and a reference image (e.g., a ground truth image) associated with the projecting device <NUM>. The processing device <NUM> is coupled to the image capturing device <NUM>, the projecting device <NUM> and the storage device <NUM>, and comprises a processing circuit <NUM>. The processing circuit <NUM> is configured for compensating the first image according to the first temperature and the plurality of first parameters, to generate a first compensated image, compensating the reference image according to the second temperature and the plurality of second parameters, to generate a second compensated image, and generating a second image (e.g., a depth map) according to the first compensated image and the second compensated image.

In one embodiment, the image capturing device <NUM> is a monitor, a video camera, a camera or any of the above combinations, but is not limited herein. In one embodiment, the projecting device <NUM> is a bioscope, a projector or any of the above combinations, but is not limited herein. In one embodiment, the storage device <NUM> is externally connected to the processing device <NUM>, or arranged in the processing device <NUM>, but is not limited herein. In one embodiment, the first/second sensing circuit <NUM>/<NUM> is a sensor (e.g., a thermometer) for detecting a temperature of a device, but is not limited herein.

In one embodiment, the projecting device <NUM> further comprises a projecting circuit. The projecting circuit is configured for projecting a design image (e.g., a reference image). In one embodiment, the processing device <NUM> further comprises a receiving circuit. The receiving circuit is configured for receiving the first image and the first temperature from the image capturing device <NUM>, receiving the second temperature from the projecting device <NUM>, and receiving the plurality of first parameters, the plurality of second parameters and the reference image from the storage device <NUM>. In one embodiment, the receiving circuit is a central processing unit (CPU) which receives (e.g., obtains, loads) data from other device/circuit, but is not limited herein.

In one embodiment, the step of compensating the first image according to the first temperature and the plurality of first parameters to generate the first compensated image comprises: generating a first interpolated image according to the first temperature and the plurality of first parameters (e.g., by using an interpolation method), and compensating the first image according to the first interpolated image, to generate the first compensated image. In one embodiment, the step of compensating the reference image according to the second temperature and the plurality of second parameters to generate the second compensated image comprises: generating a second interpolated image according to the second temperature and the plurality of second parameters (e.g., by using the interpolation method), and compensating the reference image according to the second interpolated image, to generate the second compensated image. In one embodiment, the processing circuit <NUM> generates the second image by using a depth decoding for the first compensated image and the second compensated image.

In one embodiment, the plurality of first parameters comprises a first relation (e.g., a table) between the first temperature and a first pixel shift of the first image. In one embodiment, the plurality of first parameters comprises a plurality of first pixel shift parameters of the first image, and the plurality of first pixel shift parameters are associated with the first temperature. In one embodiment, the plurality of first parameters comprises a plurality of first lens distortion coefficients of the image capturing device <NUM>, and the plurality of first lens distortion coefficients are associated with the first temperature. The first relation and the plurality of first lens distortion coefficients may be combined into a table stored by the storage device <NUM>, but are not limited herein.

In one embodiment, the plurality of second parameters comprises a second relation (e.g., a table) between the second temperature and a second pixel shift of the reference image. In one embodiment, the plurality of second parameters comprises a plurality of second pixel shift parameters of the reference image, and the plurality of second pixel shift parameters are associated with the second temperature.

<FIG> is a schematic diagram of a system <NUM> according to an embodiment of the present invention. The system <NUM> may be a structured light 3D system. In <FIG>, the system <NUM> comprises an image capturing device <NUM>, a projecting device <NUM>, a storage device <NUM>, a processing device <NUM> and a third sensing circuit <NUM>. The image capturing device <NUM> comprising a capturing circuit <NUM> and a first sensing circuit <NUM>, the projecting device <NUM> comprising a second sensing circuit <NUM>, the storage device <NUM> and the processing device <NUM> comprising a processing device <NUM> can be referred to the image capturing device <NUM>, the projecting device <NUM>, the storage device <NUM> and the processing device <NUM> in <FIG>, respectively. The embodiments in <FIG> can be applied to <FIG>, and are not narrated herein for brevity. In <FIG>, the third sensing circuit <NUM> is configured for detecting (e.g., measuring) an ambient temperature (e.g., an environment temperature). In one embodiment, the third sensing circuit <NUM> is a sensor (e.g., a thermometer) for detecting the temperature, but is not limited herein.

In <FIG>, the storage device <NUM> stores a plurality of third parameters associated with the image capturing device <NUM> and the ambient temperature, and stores a plurality of fourth parameters associated with the projecting device <NUM> and the ambient temperature. The processing circuit <NUM> compensates the first image according to the first temperature, the plurality of first parameters, the ambient temperature and the plurality of third parameters, to generate the first compensated image. The processing circuit <NUM> compensates the reference image according to the second temperature, the plurality of second parameters, the ambient temperature and the plurality of fourth parameters, to generate the second compensated image. Then, the processing circuit <NUM> generates the second image (e.g., the depth map) according to the first compensated image and the second compensated image.

In one embodiment, the plurality of third parameters comprises a third relation (e.g., a table) between the ambient temperature and a third pixel shift of the first image. In one embodiment, the plurality of third parameters comprises a plurality of third pixel shift parameters of the first image, and the plurality of third pixel shift parameters are associated with the ambient temperature. In one embodiment, the plurality of third parameters comprises a plurality of second lens distortion coefficients of the image capturing device <NUM>, and the plurality of second lens distortion coefficients are associated with the ambient temperature. The third relation and the plurality of second lens distortion coefficients may be combined into a table stored by the storage device <NUM>, but are not limited herein.

In one embodiment, the plurality of fourth parameters comprises a fourth relation (e.g., a table) between the ambient temperature and a fourth pixel shift of the reference image. In one embodiment, the plurality of fourth parameters comprises a plurality of fourth pixel shift parameters of the reference image, and the plurality of fourth pixel shift parameters are associated with the ambient temperature.

The above pixel shift parameters (e.g., the plurality of first pixel shift parameters, the plurality of second pixel shift parameters, the plurality of third pixel shift parameters and the plurality of fourth pixel shift parameters) may be measured by the polynomial equation(s) for the image capturing device <NUM>/<NUM> and/or the projecting device <NUM>/<NUM>, but are not limited herein. The above pixel shift parameters may be measured in a chamber, but are not limited herein.

In <FIG>, the system <NUM> considers the component self-heating of the image capturing device <NUM> and the projecting device <NUM>, and compensates the image captured by the image capturing device <NUM> and the reference image associated with the projecting device <NUM> according to the parameters associated with the temperatures of the image capturing device <NUM> and the projecting device <NUM>. In <FIG>, the system <NUM> not only considers the component self-heating of the image capturing device <NUM> and the projecting device <NUM>, but also considers the ambient temperature. Accordingly, the system <NUM> compensates the image captured by the image capturing device <NUM> and the reference image associated with the projecting device <NUM> according to the parameters associated with the ambient temperature and the parameters associated with the temperatures of the image capturing device <NUM> and the projecting device <NUM>. That is, the system <NUM> may be realized when the ambient temperature is unchanged (e.g., the ambient temperature is a constant) or the change in the ambient temperature is negligibly small (e.g., the temperature difference is smaller than <NUM> degree), while the system <NUM> may be realized when the ambient temperature is changed (e.g., the ambient temperature is not a constant).

<FIG> is a relationship diagram between pixel shift parameters and temperatures of an image capturing device according to an embodiment of the present invention. <FIG> may be the plurality of first parameters in <FIG>, and shows an X-axis and a Y-axis. The X-axis represents a temperature difference (ΔT) of a reference temperature (e.g., <NUM> degrees) and a temperature of the image capturing device <NUM>/<NUM>. The Y-axis represents a pixel shift parameter (a variation of a scale factor) for the image capturing device <NUM>/<NUM>. The pixel shift parameter is measured in a chamber according to modules M1-M3 and three times per module. The modules M1-M3 are performed via the same scale factor. The modules M1-M3 represent different conditions (e.g., different polynomial equations) for the image capturing device <NUM>/<NUM>.

<FIG> is a relationship diagram between pixel shift parameters and temperatures of a projecting device according to an embodiment of the present invention. <FIG> may be the plurality of second parameters in <FIG>, and shows an X-axis and a Y-axis. The X-axis represents a temperature difference (ΔT) of a reference temperature (e.g., <NUM> degrees) and a temperature of the projecting device <NUM>/<NUM>. The Y-axis represents a pixel shift parameter (a variation of a scale factor) for the projecting device <NUM>/<NUM>. The pixel shift parameter is measured in a chamber according to modules M4-M6, and the modules M4-M6 are performed via scale factors SF1 and SF2. The modules M4-M6 represent different conditions (e.g., different polynomial equations) for the projecting device <NUM>/<NUM>.

<FIG> is a flowchart of a process <NUM> according to an example of the present invention, for illustrating the operations of the system <NUM>. The process <NUM> includes the following steps:.

The process <NUM> is used for illustrating the operations of the system <NUM>. Detailed description and variations of the process <NUM> can be referred to the previous description, and are not narrated herein.

It should be noted that there are various realizations of the systems <NUM> and <NUM>. For example, the devices/circuits mentioned above may be integrated into one or more devices/circuits. In addition, the systems <NUM> and <NUM> may be realized by hardware (e.g., circuit), software, firmware (known as a combination of a hardware device, computer instructions and data that reside as read-only software on the hardware device), an electronic system or a combination of the devices mentioned above, but are not limited herein.

Claim 1:
A system (<NUM>) for handling a thermal compensation, comprising:
an image capturing device (<NUM>), comprising:
a capturing circuit (<NUM>), configured for capturing a first image; and
a first sensing circuit (<NUM>), configured for detecting a first temperature of the image capturing device;
a projecting device (<NUM>), comprising:
a second sensing circuit (<NUM>), configured for detecting a second temperature of the projecting device;
a storage device (<NUM>), configured for storing a plurality of first parameters associated with the image capturing device, a plurality of second parameters associated with the projecting device and a reference image associated with the projecting device; and
a processing device (<NUM>), coupled to the image capturing device, the projecting device and the storage device, characterised by:
a processing circuit (<NUM>), configured for:
compensating the first image according to the first temperature and the plurality of first parameters, to generate a first compensated image;
compensating the reference image according to the second temperature and the plurality of second parameters, to generate a second compensated image; and
generating a second image according to the first compensated image and the second compensated image.