Patent Description:
There is a stereo camera technology as a three-dimensional object recognition technology. This technology is a technology in which parallax is detected based on trigonometry using a difference in how images of two cameras arranged at different positions are reflected, and a depth and a position of an object are detected using the parallax. When this technology is used, a position of an observation target can be accurately detected. The stereo camera technology is applied to detect and recognize a three-dimensional object in in-vehicle applications, robot applications, and the like. The images of the two cameras of the stereo camera are transformed by predetermined projection (for example, a central projection system), and the distance to an object is measured by detecting a difference between the images as the parallax.

Meanwhile, when a camera detection image changes due to a change with time or a change in temperature, a distance measurement error occurs as a result. For example, the camera detection image changes with a change in refraction of a lens due to the temperature or a change in the lens surface spacing. Further, when a lens barrel elongates/contracts according to the temperature so that a distance between the lens and an image sensor changes, the camera detection image is enlarged/reduced. When a distance error occurs due to these factors, accurate control (for example, vehicle control, robot control, and the like) based on a measurement result becomes difficult. Further, this hinders provision of accurate information to a driver in the case of an in-vehicle stereo camera.

In order to solve the problem of the occurrence of the distance measurement error due to the temperature change, PTL <NUM> below has an object of "performing highly accurate three-dimensional measurement even in a case where zoom and focus change in a three-dimensional measurement device using a stereo camera having a zoom and focus function" and describes a technique in which "the three-dimensional measurement device includes: a plurality of imaging means including one or more imaging means capable of changing zoom and focus; a control means for zooming and focusing; a stereo image processing unit that performs stereo image processing on captured images output from the plurality of imaging means and measuring three-dimensional information; and a camera information storage means for storing camera information required to perform the stereo image processing. The camera information storage means has a plurality of pieces of camera information corresponding to states of camera information that changes depending on a zoom state and a focus state of each imaging means, and the stereo image processing unit acquires the zoom state and the focus state of each imaging means and acquires the corresponding camera information from the camera information storage means for use in the stereo image processing".

The following PTL <NUM> has an object of "performing distortion correction processing in response to a temperature change while reducing necessary storage capacity as compared with a conventional method for correcting distortion of an image using LUT", and describes a technique in which "a coefficient of a distortion correction formula including a polynomial expression capable of transforming an image position before distortion correction processing in a captured image to an ideal position over a predetermined temperature range by changing the coefficient is stored in a correction coefficient storage unit as a correction coefficient for each of a plurality of specific temperatures different from each other. A distortion correction processing unit reads a correction coefficient, which minimizes an index value (deviation amount) indicating the degree of distortion of the captured image at a detected temperature, from the correction coefficient storage unit based on the temperature detected by a temperature sensor, and performs the distortion correction processing by the distortion correction formula using the read correction coefficient".

PTL <NUM> proposes a stereo camera device having an image acquisition part for acquiring two images from two imaging parts, a temperature information acquisition section for acquiring temperature information of the two imaging parts, an image correction part for correcting the image based on the acquired temperature information, and a parallax image calculation part for generating a parallax image based on the image corrected by the image correction part.

PTL <NUM> proposes a data capture device including: an imaging module including an image sensor and a lens; a temperature sensor associated with the imaging module; a memory storing, for a plurality of temperatures, respective imaging module response parameters corresponding to the temperatures; an imaging controller connected to the image sensor and the temperature sensor; the imaging controller including a frame generator configured to: receive image data from the image sensor and a temperature measurement from the temperature sensor; generate a raw image frame containing the temperature measurement and the image data; and provide the raw image frame to a calibrator for generation of a calibrated image frame based on the raw image frame and one of the imaging module response parameters corresponding to the temperature matching the temperature measurement.

PTL <NUM> proposes an imaging apparatus including a plurality of imaging units, and a radiator operable to uniformize temperatures of the plurality of the imaging units.

PTL <NUM> proposes an electronic device including: a housing; a first image sensor configured to provide a first angle of view; a second image sensor configured to provide a second angle of view; and a processor, wherein the processor is configured to obtain a first image having first resolution for a plurality of objects outside the electronic device, the plurality of objects corresponding to the first angle of view, to obtain a second image having second resolution for some objects corresponding to the second angle of view among the plurality of objects, to crop a third image having the second resolution, corresponding to at least part of the some objects, from the second image based on at least a depth-map using the first image and the second image, and to compose the third image with a region corresponding to the at least part of the some objects in the first image.

In the related art, a frequency of changing correction with respect to a temperature change is high from the viewpoint of correcting a temperature of each lens, and thus, there is a problem that the processing becomes complicated. The problem of the frequency of changing the correction processing is not described in the above patent literatures, and thus, a countermeasure for the problem is not disclosed. For example, the three-dimensional measurement device described in Patent Literatures <NUM> and <NUM> performs image processing according to temperatures of two cameras with respect to a change in a camera detection image caused by the temperature. Therefore, it is necessary to change the image processing each time the temperature of each camera changes. Since the change of the image processing causes a delay in time until the control, it is desired to minimize the change.

The present invention has been made in view of the above problems, and an object thereof to provide an image processing device capable of reducing a change frequency of a processing unit with respect to a temperature change of a lens of the image processing device.

The present invention provides an image processing device according to claim <NUM>.

According to the image processing device of the present invention, it is possible to reduce the change frequency of the processing unit with respect to the temperature change of the lens of the image processing device.

<FIG> is a block diagram illustrating a configuration example of an image processing device <NUM> according to a first comparative example of the present invention. The image processing device <NUM> is configured to enable input of a plurality of images, and is connected so as to be capable of receiving images from a camera <NUM> (first camera) and a camera <NUM> (second camera) constituting a stereo camera, respectively, in the example of <FIG>.

The camera <NUM> includes a lens <NUM> (first lens) and an image sensor <NUM> (first image sensor). The camera <NUM> includes a lens <NUM> (second lens) and an image sensor <NUM> (second image sensor). The camera <NUM> detects an image of an object or the like by the image sensor <NUM> via the lens <NUM>. The camera <NUM> detects an image of an object or the like with the image sensor <NUM> via the lens <NUM>.

The camera <NUM> includes a temperature sensor <NUM> (first temperature sensor). The temperature sensor <NUM> detects a temperature (first temperature) related to the camera <NUM>. The camera <NUM> includes a temperature sensor <NUM> (second temperature sensor). The temperature sensor <NUM> detects a temperature (second temperature) of the camera <NUM>. The meaning of the "temperature related to the camera" can be appropriately defined by those skilled in the art, and may be, for example, a temperature of a lens, may be a temperature in the vicinity of the lens, may be a temperature of an image sensor (particularly a temperature of a CMOS element), may be temperatures of other components of the camera or a temperature of a space in the camera. The temperature sensors <NUM> and <NUM> may be temperature sensors outside the camera.

The image processing device <NUM> has a configuration as a known computer, and includes a computing device <NUM> and a storage device <NUM>. The computing device <NUM> includes, for example, a microprocessor and the like, and the storage device <NUM> includes, for example, at least one of a volatile memory and a nonvolatile memory. The storage device <NUM> can store an image and another information, and functions as, for example, an image storage means.

<FIG> is a block diagram illustrating a configuration example of the computing device <NUM>. The computing device <NUM> includes affine processing units 20a and 20b, brightness correction units 21a and 21b, pixel interpolation units 22a and 22b, brightness information generation units 23a and 23b, a parallax image generation unit <NUM>, a distance calculation unit <NUM>, and a temperature difference acquisition unit <NUM>. The computing device <NUM> acquires an image P1 (first image) from the camera <NUM> and acquires an image P2 (second image) from the camera <NUM>.

The affine processing unit 20a performs affine processing on the image P1. The affine processing is, for example, a linear coordinate transformation process, but may include a non-linear operation. As a result of performing the affine processing, the affine processing unit 20a outputs an image P3 (third image). Similarly, the affine processing unit 20b performs affine processing on the image P2 to acquire an image P4 (fourth image). The affine processing unit 20a and the affine processing unit 20b may also execute distortion transformation processing other than the affine processing. For example, in a case where a lens projection system is fsinθ, projective transformation to ftanθ may be performed. However, a format of an image after the projective transformation may be a system other than ftanθ.

The brightness correction unit 21a corrects the brightness of each pixel of the image P3. For example, correction is performed based on a difference between a gain of the camera <NUM> and a gain of each pixel in the image P3, and the like. Similarly, the brightness correction unit 21b corrects the brightness of each pixel of the image P4.

The pixel interpolation unit 22a performs demosaicing processing on the image P3. For example, a RAW image is transformed into a color image. Similarly, the pixel interpolation unit 22b performs the demosaicing processing on the image P4.

The brightness information generation unit 23a generates brightness information of the image P3. The brightness information is information indicating a brightness value of each pixel. For example, the brightness information is generated by transforming a color value of each pixel of a color image into a brightness value for generating a parallax image. Similarly, the brightness information generation unit 23b generates brightness information of the image P4.

A parallax image generation unit <NUM> calculates parallax between two images based on the image P3 and the image P4 (or information generated based on these). A distance calculation unit <NUM> calculates a distance to an object appearing in an image based on the calculated parallax.

Considering the processing in the brightness correction unit 21a and the brightness correction unit 21b, the pixel interpolation unit 22a and the pixel interpolation unit 22b, the brightness information generation unit 23a and the brightness information generation unit 23b, and the parallax image generation unit <NUM>, it can be said that the distance calculation unit <NUM> detects the distance to the object in the image based on the image P3 and the image P4. Those skilled in the art can arbitrarily design a distance measurement criterion, and the distance to the object can be measured from a predetermined distance reference point defined according to a positional relationship between the camera <NUM> and the camera <NUM>, for example.

The temperature difference acquisition unit <NUM> includes a temperature acquisition unit 30a (first temperature acquisition unit) and a temperature acquisition unit 30b (second temperature acquisition unit). The temperature acquisition unit 30a acquires a temperature related to the camera <NUM> from the temperature sensor <NUM>, and the temperature acquisition unit 30b acquires a temperature related to the camera <NUM> from the temperature sensor <NUM>. The temperature difference acquisition unit <NUM> acquires and outputs a temperature difference (temperature difference between cameras) between the temperature related to the camera <NUM> and the temperature related to the camera <NUM>.

When an absolute value of the temperature difference is equal to or smaller than a predetermined value (α°C in <FIG>), the computing device <NUM> outputs distance information output from the distance calculation unit <NUM> to a control processing unit (not illustrated) (referred to as normal processing). The control processing unit controls a vehicle, a robot, and the like based on the distance information output from the distance calculation unit <NUM>. For example, in the case of a vehicle, collision damage reduction braking, lane departure prevention assistance, and the like are performed.

When the absolute value of the temperature difference is equal to or larger than the predetermined value, it is considered that the reliability of the distance information is low, and thus, the computing device <NUM> outputs error information indicating that an error has occurred at the time of calculating the parallax image or the distance is output to the control processing unit (error processing). For example, in the case of a vehicle, when the error information is output, the control processing is temporarily stopped, and the information is notified to a driver through an instrument panel or the like. Thereafter, when the absolute value of the temperature difference becomes smaller than the predetermined value, the processing is changed to the normal processing. Such notification improves the safety consciousness of the driver and urges safe driving. For example, when the image processing device <NUM> outputs the error information in a case where another distance sensor is used together in an in-vehicle application, a robot application, or the like, the control based on the distance information of the image processing device <NUM> may be stopped, and the control may be performed using distance information of another distance sensor. In the first comparative example, the processing is changed according to the temperature difference between the cameras, and thus, it is possible to avoid inappropriate control accompanying a distance measurement error.

<FIG> is a view for describing the influence of the temperature difference in the first comparative example. A reason for detecting the temperature difference between the cameras will be described with reference to <FIG>. A left view in <FIG> illustrates a state where a temperature difference between the lens <NUM> and the lens <NUM> occurs, and a right view in <FIG> illustrates a state where the temperature difference between the lens <NUM> and the lens <NUM> disappears. Here, a distance to an object <NUM> is detected based on images acquired by the image sensor <NUM> and the image sensor <NUM> via the lens <NUM> and the lens <NUM>.

In the left view of <FIG>, a light beam L1 incident on the lens <NUM> from the object <NUM> and a light beam L2 incident on the lens <NUM> from the object <NUM> when measurement is performed in a state where there is no temperature difference (for example, a state where the camera <NUM> and the camera <NUM> are at a predetermined reference temperature) are indicated by one-dot chain lines. A distance D0 to the object <NUM> is detected based on a difference (parallax) between an incident position of the light beam L1 detected by the image sensor <NUM> and an incident position of the light beam L2 detected by the image sensor <NUM>. In this case, there occurs no distance measurement error, and the distance D0 accurately represents the distance to the object <NUM>.

In the left view of <FIG>, it is assumed that a temperature difference occurs between cameras and the temperature of the lens <NUM> rises, for example. As a result, optical characteristics (a refractive index, a lens surface spacing, and the like) of the lens <NUM> change, and a state different from optical characteristics of the lens <NUM> is formed. As a result, the light beam L2 is incident on the image sensor <NUM> at a position shifted by an arrow A1. The computing device <NUM> determines that a light beam L4 is incident on the image sensor <NUM> from the viewpoint of parallax calculation, and erroneously detects that the light beam from the object <NUM> exists at an angle of view of the virtual light beam L4 indicated by a two-dot chain line. As a result, an intersection <NUM> between the light beam L1 and the virtual light beam L4 is erroneously detected as a position of the object <NUM>, and the distance to the object <NUM> is detected as a distance D1. As can be seen from <FIG>, the distance D0 and the distance D1 do not match, and a distance measurement error occurs. This is a mechanism of occurrence of the distance measurement error accompanying the temperature change of the lens.

Next, a case where there is no temperature difference between the cameras will be described with reference to the right view of <FIG>. When the temperatures of the lens <NUM> and the lens <NUM> similarly rise, a light beam is incident on a position shifted by the arrow A1 on the image sensor <NUM> similarly to the left view of <FIG>. The computing device <NUM> determines that the virtual light beam L4 is incident on the image sensor <NUM> from the viewpoint of parallax calculation, and erroneously detects that the light beam from the object <NUM> exists at the angle of view of the virtual light beam L4 indicated by the two-dot chain line. Further, a light beam is incident on a position shifted by an arrow B1 on the image sensor <NUM> similarly for the camera <NUM>. The computing device <NUM> determines that a virtual light beam L5 is incident on the image sensor <NUM>, and erroneously detects that the light beam from the object <NUM> exists at an angle of view of the virtual light beam L5 indicated by a two-dot chain line.

The computing device <NUM> detects an intersection <NUM> between the virtual light beam L5 and the virtual light beam L4 as a position of the object <NUM>, and detects a distance D2. Since a difference between the distance D2 and the distance D0 is smaller than a difference between the distance D1 and the distance D0, the distance measurement error is improved. In particular, a temperature difference condition that the distance D2 and the distance D0 are set to be substantially the same also exists, and thus, a residual error is not a problem in practical distance measurement.

<FIG> is a graph illustrating an example of the distance measurement error when the temperature difference between the cameras changes. An effect of the first comparative example will be quantitatively described with reference to <FIG>. The following condition examples were used at the time of calculating the graph of <FIG>.

The above temperature sensitivity corresponds to displacement of a position of an image by <NUM> when an angle of view is <NUM> deg and a temperature change is <NUM>. Here, it is assumed that the lens temperature sensitivity changes linearly with respect to an angle of view and a temperature. When considering manufacturing variations of a lens or the like, the temperature sensitivity actually varies depending on the lens, but the same effect can be obtained even if the temperature sensitivity varies (not that it is assumed that the temperature sensitivities of the lens <NUM> and the lens <NUM> are the same in this condition example). The lens temperature sensitivity is not necessarily linear, and a similar effect can be obtained even if the lens temperature sensitivity is non-linear.

From <FIG>, it can be seen that the distance measurement error occurs depending on the temperature difference between the cameras regardless of the temperature of the lens <NUM>. Further, it can be seen that the distance measurement error is small if the temperature difference is small. Although the temperature change of the lens has been described herein, the same applies to the influence of elongation/contraction of a lens barrel with the temperature. In this case, it is considered that a distance between a lens and an image sensor changes, and a position of a light beam incident on the image sensor is shifted, so that the same result as in <FIG> can be obtained.

For the above reasons, the image processing device <NUM> of the first comparative example determines that the reliability of the distance information is high and performs the normal processing when the absolute value of the temperature difference is equal to or smaller than the predetermined value. When the absolute value of the temperature difference is equal to or larger than the predetermined value, it is determined that the reliability of the distance information is low, and the error processing is performed. Since the temperature difference between the cameras is detected to change the processing according to the temperature difference in this manner, it is possible to avoid the inappropriate control accompanying the distance measurement error.

<FIG> is an image view for describing an effect of the present invention in comparison with the related art. A left view in <FIG> illustrates processing with respect to a temperature change in PTL <NUM> or PTL <NUM>. A right view in <FIG> illustrates processing with respect to a temperature change in the first comparative example. In <FIG>, the vertical axis represents the temperature, the horizontal axis represents time, and the time at which the image processing device <NUM> is activated is defined as time zero. When the image processing device <NUM> is activated, the temperature rises, and converges to a predetermined temperature after a lapse of a predetermined time as heat generation and heat dissipation become the same. A solid line and a dotted line indicate the temperature of the camera <NUM> and the temperature of the camera <NUM>, respectively, and the temperatures of the two cameras change in the same manner. In the drawing, ▲ indicates a change timing of the processing.

In the case of the left view of <FIG>, it is necessary to change the processing a plurality of times assuming that the processing is changed for each predetermined temperature Δt. On the other hand, in the case of the right view of <FIG>, the error processing does not occur since no temperature difference occurs between the two cameras. Since the temperature change has a long time constant, the temperatures of the two cameras change slowly. Therefore, the temperature difference between the two cameras is less likely to occur as compared with the temperature change, a change frequency of the processing is low. In this manner, the image processing device <NUM> according to the present first comparative example has a feature that the change frequency of the processing can be reduced as compared with an image processing device in the related art (PTL <NUM> and PTL <NUM>).

A temperature α of the first comparative example is not a fixed value and may be changed based on the temperature of the camera. Although the absolute value of the temperature difference is compared with α in the first comparative example, the processing may be changed based on whether the temperature difference falls within a predetermined temperature range. For example, there is a case where a distance error is asymmetric with respect to positive and negative temperature differences between cameras as illustrated in <FIG>. Further, in a case where the temperature sensitivity is different between lenses or in a case where the temperature sensitivity of each lens is non-linear, there is a condition that the distance measurement accuracy is the best when a temperature difference is a predetermined value that is not zero.

<FIG> is a configuration example of the computing device <NUM> in a case where processing is changed based on whether the temperature difference falls within the predetermined temperature range. In this case, assuming a temperature TA of the camera <NUM> and a temperature TB of the camera <NUM>, the normal processing may be performed if TA - TB falls within a range of a lower limit temperature β and an upper limit temperature γ, and the error processing may be performed if TA - TB does not fall within the range. At this time, the computing device <NUM> may refer to, for example, a temperature (at least any of TA and TB) and change at least any of the lower limit temperature β and the upper limit temperature γ. The temperature α, the lower limit temperature β, and the upper limit temperature γ can be determined by a distance measurement tolerance in a system to which the present comparative example is applied. The computing device <NUM> stores data describing such a determination rule in the storage device <NUM>, for example, and can determine the temperature α, the lower limit temperature β, and the upper limit temperature γ according to the data.

<FIG> is a block diagram illustrating a configuration example of the computing device <NUM> included in the image processing device <NUM> according to a second comparative example of the present invention. In the first comparative example, the error processing is performed when the absolute value of the temperature difference between the cameras is equal to or larger than the predetermined value. In the second comparative example, a temperature of a camera is changed so as to decrease a temperature difference when an absolute value of the temperature difference is equal to or larger than a predetermined value. The other configurations are the same as those in the first comparative example.

In the case of the first comparative example, the error processing is continued until the absolute value of the temperature difference between the cameras becomes equal to or smaller than the predetermined value. It is considered that it takes time until normal processing since a time constant of a temperature change is long if the temperature difference between the cameras increases. Therefore, the temperature difference is forcibly reduced by heating or cooling the camera in the second comparative example. A camera at a higher temperature may be cooled or a camera at a lower temperature may be heated. The computing device <NUM> instructs a cooling device or a heating device to adjust the temperature of the camera.

<FIG> is an image view for describing an effect of the present comparative example in comparison with the first comparative example. A left view in <FIG> illustrates processing with respect to a temperature change in the first comparative example, and a right view in <FIG> illustrates processing with respect to a temperature change in the second comparative example. A difference from <FIG> is that temperatures of the camera <NUM> and the camera <NUM> at the time of activation are different. The temperatures at the time of activation may be different when the two cameras are mounted on both ends of a vehicle, for example, in the case of an in-vehicle application.

In the case of the image processing device <NUM> of the first comparative example, the temperature difference between the cameras at the time of activation is larger than the temperature α, and thus, the error processing is performed. When the temperature difference between the cameras becomes smaller than the temperature α after a lapse of time t1, the processing transitions to the normal processing. On the other hand, the temperature difference between the cameras is larger than the temperature α at the time of activation in the case of the image processing device <NUM> of the second comparative example, and thus, the camera <NUM> is heated. As a result, the processing can transition to the normal processing within time t2 shorter than the time in the left view of <FIG>.

A method for changing the camera temperature in the second comparative example is not limited. For example, in a case where a lens is heated, a heating wire may be used. Further, a fan may be used in a case where cooling is performed. Further, a temperature changing mechanism mounted for other applications may be used. For example, in the case of an in-vehicle application, air conditioning or the like is used. Furthermore, the amount of heat generated by another element may be used. For example, in the case of an in-vehicle application, a head lamp may be used. The camera <NUM> and the camera <NUM> can be mounted in or near two headlamps, and the camera can be forcibly warmed using heat generated from the headlamp. In this case, the head lamp on a side having a lower temperature may be turned on.

The image processing device <NUM> according to the second comparative example can generate highly reliable distance information by forcibly changing the temperature of the camera when the temperature difference between the cameras does not fall within a reference range. Furthermore, it is possible to suppress the number of times of changing image processing according to a temperature as in PTLs <NUM> and <NUM>, and to eliminate a delay in time accompanying the change in the image processing.

Although the absolute value of the temperature difference is compared with the temperature α in the second comparative example, normal processing may be performed if TA - TB falls within the range between the lower limit temperature β and the upper limit temperature γ, and the temperature of the camera may be changed if TA - TB does not fall within the range in the same manner as in <FIG> Further, the temperature of the camera is changed in the description of the second comparative example, but a temperature of a lens may be changed.

<FIG> is a block diagram illustrating a configuration example of the computing device <NUM> included in the image processing device <NUM> according to a third comparative example of the present invention. In the first comparative example, the error processing is performed when the absolute value of the temperature difference between the cameras is equal to or larger than the predetermined value. In the third comparative example, an image processing method is changed when an absolute value of a temperature difference is equal to or larger than a predetermined value. In the case of the first comparative example, the affine processing is performed using a coefficient as a fixed value. On the other hand, a coefficient of affine processing is changed when the temperature difference becomes equal to or larger than the predetermined value in the third comparative example. Hereinafter, the processing of the third comparative example will be described.

The computing device <NUM> includes a coefficient changing unit <NUM> in addition to the configuration described with reference to <FIG>. The coefficient changing unit <NUM> further includes a coefficient changing unit 31a and a coefficient changing unit 31b. When the absolute value of the temperature difference between cameras is equal to or smaller than the predetermined value (α°C in <FIG>), transformation processing is performed by a predetermined affine coefficient. When the absolute value of the temperature difference is equal to or larger than the predetermined value, the affine transformation is performed after changing the affine coefficient as follows.

The coefficient changing unit <NUM> changes coefficients of transformation processing performed by the affine processing unit 20a and the affine processing unit 20b (a first correction unit and a second correction unit), respectively. For example, the coefficient changing unit 31a and the coefficient changing unit 31b generate a transformation table representing transformation with respect to the images P1 and P2 based on temperature of the cameras and a predetermined transformation reference table. The transformation reference table may be stored in advance in the storage device <NUM>, for example. In this case, it can be said that the storage device <NUM> functions as a transformation reference coefficient storage unit. This transformation table may be a coefficient of affine processing in PTLs <NUM> and <NUM>.

The images P3 and P4 respectively output from the affine processing unit 20a and the affine processing unit 20b are input to the parallax image generation unit <NUM> via the brightness correction unit 21a, the brightness correction unit 21b, the brightness information generation unit 23a, and the brightness information generation unit 23b. The parallax image generation unit <NUM> calculates parallax between two pieces of image information based on the image P3 and the image P4 (or information generated based on these). The distance calculation unit <NUM> calculates a distance to an object appearing in an image based on the parallax.

In the third comparative example, the coefficient of the affine processing is changed only when the absolute value of the temperature difference is equal to or larger than the predetermined value. The image processing device <NUM> of the third comparative example has a feature that a processing change frequency can be reduced as compared with the image processing device in the related art (PTLs <NUM> and <NUM>) as described in the first comparative example.

<FIG> illustrates processing with respect to a temperature change in the third comparative example. Here, it is assumed that there is a temperature difference at the time of activation. In this case, a difference from <FIG> is that temperatures of the camera <NUM> and the camera <NUM> at the time of activation are different. In the image processing device <NUM> of the third comparative example, the affine processing coefficients of the affine processing unit 20a and the affine processing unit 20b are changed when the temperature difference between the cameras at the time of activation is larger than a temperature α. When the temperature difference between the cameras becomes smaller than the temperature α (time t1), the processing transitions to processing using an affine coefficient which is a predetermined value. The similar processing can be also performed in the first and second comparative examples.

Although the affine processing coefficients of the affine processing unit 20a and the affine processing unit 20b are changed in the third comparative example, for example, the affine processing coefficient of only one of the affine processing unit 20a and the affine processing unit 20b may be changed according to the temperature difference. This is based on the same reason as the description that the distance measurement error is reduced in the case of the virtual light beam L5 in the right view of <FIG> while the light beam from the object <NUM> is erroneously detected to exist at the angle of view of the virtual light beam L4 indicated by the two-dot chain line in the left view of <FIG>. That is, highly accurate distance measurement can be implemented by changing an affine processing method only for the image P1 of an image sensor <NUM> and transforming the image to obtain the virtual light beam L5. As a predetermined offset is added by the parallax image generation unit <NUM> based on a concept similar to this, an effect similar to that of changing the affine processing coefficient can be obtained.

Although the absolute value of the temperature difference is compared with the temperature α in the third comparative example, normal processing may be performed if TA-TB falls within the range of the lower limit temperature β and the upper limit temperature γ, and affine processing coefficients of the affine processing unit 20a and the affine processing unit 20b may be changed if TA-TB does not fall within the range in the same manner as in <FIG>.

<FIG> is a block diagram illustrating a configuration example of the image processing device <NUM> according to a fourth comparative example of the present invention. In the fourth comparative example, the image processing device <NUM> acquires images from cameras <NUM> and <NUM>, respectively. Although the cameras <NUM> and <NUM> are not stereo cameras, a fact that the image processing device <NUM> acquires images from the two cameras is similar to the first comparative example.

The camera <NUM> includes a lens <NUM> (first lens) and an image sensor <NUM> (first image sensor). The camera <NUM> includes a lens <NUM> (second lens) and an image sensor <NUM> (second image sensor). The camera <NUM> detects an image of an object or the like with the image sensor <NUM> via the lens <NUM>. The camera <NUM> detects an image of the object or the like with the image sensor <NUM> via the lens <NUM>. The camera <NUM> is a camera having a wider field of view than the camera <NUM>. The camera <NUM> is a camera capable of detecting a farther side than the camera <NUM>.

<FIG> illustrates a synthesized image of two camera images in the case of being used for an in-vehicle application as an example. An upper view of <FIG> illustrates a case where a temperature difference is small, and a lower view of <FIG> illustrates a case where the temperature difference is large. An image P13 is an output image of the camera <NUM>. An image P14 is an output result of the camera <NUM>. Since such a synthesized image is generated, the resolution in a progressing direction of a vehicle can be improved, and a wide field of view can be realized. In this example, a white line WL1 and a white line WL2 are reflected on the image P13 and the image P14, respectively.

When there is no temperature difference, the white line WL1 and the white line WL2 are smoothly connected as illustrated in the upper view of <FIG>. When the temperature difference occurs, a detection position of an image on one side is changed (shifted by the arrow A1) as illustrated in the left view of <FIG>, and thus, the white line WL1 and the white line WL2 are not smoothly connected as illustrated in the lower view of <FIG>. The synthesized image as illustrated in the lower view of <FIG> is different from the actual image.

<FIG> is a block diagram illustrating a configuration example of the computing device <NUM> in the fourth comparative example. The computing device <NUM> in the fourth comparative example includes an image synthesis unit <NUM> instead of the parallax image generation unit <NUM> and the distance calculation unit <NUM>. The affine processing units 20a and 20b output the images P13 and P14, respectively.

The temperature acquisition unit 30a acquires a temperature related to the camera <NUM> from the temperature sensor <NUM>, and the temperature acquisition unit 30b acquires a temperature related to the camera <NUM> from the temperature sensor <NUM>. The temperature difference acquisition unit <NUM> acquires and outputs a temperature difference (temperature difference between cameras) between the temperature related to the camera <NUM> and the temperature related to the camera <NUM>.

When an absolute value of the temperature difference is equal to or smaller than a predetermined value, the computing device <NUM> (image synthesis unit <NUM>) synthesizes the image P3 and the image P4 to generate a synthesized image as described with reference to <FIG>. Thereafter, normal processing according to a system is performed. When the absolute value of the temperature difference is equal to or larger than the predetermined value, it is notified that an error has occurred at the time of generating the synthesized image. Through the above procedure, a correct image can be output to the system side.

Although error processing is performed in the same manner as in the first comparative example in the above description, temperature adjustment or coefficient change may be performed as in the second or third comparative example. Although the image P14 is included in the image P13 in <FIG>, the images P13 and P14 may be images having similar sizes or may be images having a partially common region without being limited thereto. In any case, the same effect as that of the fourth comparative example can be obtained.

<FIG> is a block diagram illustrating an example of a configuration including the image processing device <NUM> according to a first embodiment of the present invention. In the first embodiment, the image processing device <NUM> converts a separately acquired distance error into a temperature difference instead of using the temperature sensors <NUM> and <NUM>. Since the other configurations are the same as those of the first to fourth comparative examples, the fact of converting the distance error into the temperature difference will be mainly described hereinafter.

A distance detection unit <NUM> detects a distance to an object and outputs the distance to the image processing device <NUM>. The distance detection unit <NUM> can be configured using, for example, a distance measurement device capable of measuring a distance to an object such as a radar and light detection and ranging (LiDAR). The distance detection unit <NUM> can be configured as a part of the image processing device <NUM>, or can be configured as a device separate from the image processing device <NUM>.

<FIG> is a block diagram illustrating a configuration example of the computing device <NUM> in the first embodiment. The computing device <NUM> includes a distance comparison unit <NUM> and a temperature difference conversion unit <NUM>, instead of the temperature difference acquisition unit <NUM>. The distance calculation unit <NUM> calculates a distance to the object and outputs the result as distance information G1. The distance comparison unit <NUM> receives the distance information G1 and receives distance information G2 detected by the distance detection unit <NUM>. The distance comparison unit <NUM> compares the distance information G1 with the distance information G2, and notifies the temperature difference conversion unit <NUM> of the result. The temperature difference conversion unit <NUM> converts the comparison result into the temperature difference between the cameras <NUM> and <NUM>. The other processing is similar to that described in the first comparative example.

The temperature difference conversion unit <NUM> converts the distance information into the temperature difference by the following procedure. A correspondence relationship between the temperature difference between the cameras and a distance measurement error can be acquired in advance as described with reference to <FIG>. For example, in a case where it is assumed that the distance information G2 is accurate, a distance measurement error of the distance information G1 can be calculated using the distance information G2 as a reference. The temperature difference can be estimated by applying the distance measurement error to the correspondence relationship as illustrated in <FIG>.

According to the first embodiment, if the distance information G2 can be acquired from the distance detection unit <NUM>, the distance measurement error can be converted into the temperature difference using the separately calculated distance information G1. Therefore, it is advantageous that the temperature sensors <NUM> and <NUM> are not required. Even in a case where the temperature sensors <NUM> and <NUM> are provided, it is also possible to consider an application in which a measurement result is correct by comparing the temperature difference converted by a technique of the first embodiment with a measurement result obtained by each temperature sensor.

Instead of converting a difference between the distance information G1 and G2 into the temperature difference, the difference may be directly regarded as the distance measurement error. Alternatively, the difference may be output to the outside of the image processing device <NUM>, and the difference may be used when an external device that has received the difference determines the distance measurement error.

When an absolute value of the temperature difference is equal to or larger than a predetermined value, the temperature adjustment described in the second comparative example or the coefficient adjustment described in the third comparative example may be performed instead of or in combination with error processing. Since the temperature adjustment is performed on any of two lenses at a higher (or lower) temperature, it is necessary to determine the sign of the temperature difference (that is, which lens has a higher (or lower) temperature). Since the coefficient adjustment is performed using a lens closer to a standard operating temperature as a reference (that is, a correction coefficient of a lens farther from the standard operating temperature is mainly changed), it is necessary to determine the sign of the temperature difference. Therefore, it is necessary to determine whether the temperature difference is positive or negative in any case of performing either the temperature adjustment or the coefficient adjustment. Such a sign determination procedure will be described hereinafter.

<FIG> is a view illustrating a detection distance when the temperature difference between the cameras becomes positive and a detection distance when the temperature difference becomes negative. A left view in <FIG> illustrates a case where a temperature of the lens <NUM> rises (temperature difference is negative), and a right view in <FIG> illustrates a case where a temperature of the lens <NUM> rises (temperature difference is positive). When the object <NUM> is on the right side of the image processing device <NUM>, the detected distance D1 is longer than the actual distance D0 in the case where the temperature difference is negative. In the case where the temperature difference is positive, the detected distance D2 is shorter than the actual distance D0. In a case where the distance information G2 does not largely depend on the temperature, the actual distance D0 can be detected by the distance detection unit <NUM>, and thus, whether the temperature difference is positive or negative can be determined by comparing D0 with D1 or D0 with D2. If the sign of the temperature difference is known, which of the lenses <NUM> and <NUM> has a higher (or lower) temperature can be determined, and thus, the processing described in the second and third comparative examples can be performed based on this.

It is desirable to apply the configuration using the distance detection unit <NUM> as in the first embodiment particularly in a state where the temperature is unstable such as at the time of activation. This is because the temperature difference between the cameras decreases as time elapses, and thus, it is possible to perform distance measurement to the same extent as the distance detection unit <NUM>. The configuration of the first embodiment is effective in a case where a distance sensor is used in automatic driving of a vehicle or the like.

In the above embodiment and comparative examples, for example, the error processing as in the first comparative example may be performed when the temperature difference between the cameras occurs, and then, the lens temperature may be adjusted as in the second comparative example, or the image processing method may be changed as in the third comparative example.

In the above embodiment, the computing device <NUM> may stop processing for calculating the parallax image and the distance, instead of outputting the error or together with the error output when the temperature difference does not fall within the reference range.

Claim 1:
An image processing device (<NUM>) that processes an image captured by a camera, the image processing device (<NUM>) comprising:
first and second cameras (<NUM>, <NUM>) that capture images (P1, P2) of an object to be captured; and
a computing device (<NUM>) that performs first processing on the images (P1, P2) captured by the first and second cameras (<NUM>, <NUM>) and outputs a result of the first processing,
wherein the computing device (<NUM>) generates a parallax image from the image (P1) captured by the first camera (<NUM>) and the image (P2) captured by the second camera (<NUM>) as the first processing to calculate a first distance (G1) to the object in the image,
the computing device (<NUM>) acquires a temperature difference between a first temperature of the first camera (<NUM>) and a second temperature of the second camera,
the computing device (<NUM>) outputs the result if the temperature difference falls within a reference range, and
the computing device (<NUM>) performs second processing different from the first processing, instead of or together with the outputting of the result if the temperature difference does not fall within the reference range, characterised in that:
the computing device (<NUM>) acquires a second distance (G2) to the object from a distance measurement device, and
the computing device (<NUM>) calculates the temperature difference in accordance with a difference between the first distance (G1) and the second distance (G2).