Patent Description:
In order to improve traveling safety of a vehicle, a system has been studied in which an obstacle ahead is detected by a camera mounted on the vehicle, and when there is a possibility of collision with the obstacle, an alarm to a driver or automatic braking is performed.

As a sensor for monitoring the front of the vehicle, there are a millimeter wave radar, a laser radar, a camera, and the like. As a type of camera, there are a monocular camera and a stereo camera using a plurality of cameras. The stereo camera can measure a distance to a photographed object by using parallax of an overlapping region of images photographed by two cameras at a predetermined interval (for example, see <CIT>). Therefore, it is possible to accurately grasp the risk of collision to the object ahead. The stereo camera obtains parallax between images captured by the two cameras and transforms the parallax into a distance.

As an exposure type of an image sensor for photographing an image, there is a rolling shutter type. The rolling shutter type is a type of photographing pixels of sensors disposed two-dimensionally while shifting exposure timings in units of rasters. In this type, it is known that distortion occurs in an image when a fast moving subject is captured. For this reason, when the traveling direction is captured by the stereo camera mounted on the moving vehicle, there may be a difference in the measured distance between the upper side and the lower side of the image. <CIT> proposes a method of correcting a distance difference between the upper and lower sides of the image.

<CIT> discloses a stereo camera apparatus for obtaining distance information of an object. The stereo camera apparatus includes a camera unit that includes a left image sensor and a right image sensor respectively obtaining a left-side image and a right-side image.

<CIT> proposes a method of adjusting all the parallaxes in an image to the parallax of a reference time by correcting the parallax based on the reference time by a delay amount of the exposure timing for each pixel from the reference time. However, the influence of the rolling shutter type on the parallax of the stereo camera has the following problems in addition to the above problems.

That is, an image captured using the optical lens is distorted. Therefore, in the stereo camera, it is necessary to correct the distortion of the left and right cameras so that the same object appears at the same raster position in the left and right images (images captured by the left and right cameras). This is because the parallax is calculated in the stereo matching processing while scanning the block matching in the raster direction. Meanwhile, since the distortion of the right and left lenses is not the same, the exposure timings of pixels of the same raster of the left and right images after correcting the lens distortion may be different between the left and right images. In this case, if the vehicle moves during the period of the deviation in the exposure timing, the same object may not appear in the same raster of the left and right images. In this case, there is a possibility that an erroneous calculation occurs in the stereo matching processing and an error occurs in the parallax. <CIT> does not disclose such a problem.

An object of the present invention is to provide an image processing device capable of reducing an error in parallax calculation of stereo matching processing, and a stereo camera device using the image processing device.

The dependent claims describe optional embodiments of the invention. An image processing device includes a geometric correction unit configured to correct distortion of images captured by a plurality of rolling shutter-type image sensors mounted in a vehicle, a parallax calculation unit configured to calculate a parallax using a corrected image that is corrected by the geometric correction unit, and an exposure time difference adjusting unit configured to adjust a geometric correction position of at least one image based on a difference between a geometric correction amount used for distortion correction of an image serving as a reference and a geometric correction amount used for distortion correction of other images and vehicle information regarding a position or movement of the vehicle.

According to the present invention, for example, in a stereo camera device, pixels on the same raster of the respective images (left and right images) become images with the same exposure timing, and thus, there is an effect that an error in parallax calculation of the stereo matching processing can be reduced.

Objects, configurations, and effects besides the above description will be apparent through the explanation on the following embodiments.

Hereinafter, embodiments of the present invention will be described using the drawings and the like. Although the following description illustrates specific examples of the content of the invention, the invention is not limited to the described embodiments but is defined by the appended claims. In all the drawings for describing the invention, components having the same function are designated by the same reference numeral, and the repeated description thereof may be omitted.

<FIG> illustrates a system configuration diagram of a stereo camera device <NUM> on which an image processing device <NUM> according to a first embodiment is mounted. The stereo camera device <NUM> is mounted on a vehicle (hereinafter, it may be referred to as an own vehicle) not illustrated, and mainly includes a pair of left and right cameras (a left camera <NUM> and a right camera <NUM>) arranged at predetermined intervals (side by side) so as to photograph the periphery (for example, the front) of the vehicle, and the image processing device <NUM> that processes images (hereinafter, the image may be referred to as a camera image) of the cameras.

In this example, two cameras of the right camera <NUM> and the left camera <NUM> capture images around the vehicle. An optical lens <NUM> and an image sensor <NUM> are mounted in the right camera <NUM>, and an optical lens <NUM> and an image sensor <NUM> are mounted in the left camera <NUM>. The image sensors <NUM> and <NUM> are a rolling shutter type. An angle of view and a focal distance of the image captured by the camera are determined by an optical lens. An image captured through the optical lens is distorted. The geometric correction is performed to correct the distortion.

The images (images with distortion) captured by the right camera <NUM> and the left camera <NUM> are input to the image processing device <NUM>. The image processing device <NUM> includes a microcomputer including a CPU (Central Processing Unit), a memory, and the like, and the CPU executes various processes described below according to instructions of a program stored in the memory.

In the present embodiment, the image processing device <NUM> includes a geometric correction (left) unit <NUM>, a geometric correction (right) unit <NUM>, a geometric correction table left <NUM>, a geometric correction table right <NUM>, a parallax calculation unit <NUM>, an object detection unit <NUM>, a vehicle control unit <NUM>, an exposure time difference adjusting unit <NUM>, a road surface estimation unit <NUM>, and a table reference coordinate generation unit <NUM>, which are communicably connected via a communication line.

The geometric correction (right) unit <NUM> corrects the image of the right camera <NUM> (which may be referred to as a right image or a right camera image), and the geometric correction (left) unit <NUM> corrects the image of the left camera <NUM> (which may be referred to as a left image or a left camera image). In the present example, the geometric correction processing of the image is performed using the geometric correction table. The geometric correction table right <NUM> is a table for correcting the photographed image of the right camera <NUM>, and is input to the geometric correction (right) unit <NUM>. The geometric correction table left <NUM> is a table for correcting the photographed image of the left camera <NUM>, and is input to the geometric correction (left) unit <NUM>. The images (corrected images) corrected by the geometric correction (right) unit <NUM> and the geometric correction (left) unit <NUM> are input to the parallax calculation unit <NUM>.

Note that, here, a table method using a geometric correction table is adopted as the geometric correction processing of the image, but for example, a geometric correction value used for the geometric correction processing (distortion correction processing) may be calculated and obtained using a mathematical expression or the like.

The exposure time difference adjusting unit <NUM> is a unit that corrects a deviation between the left and right images caused by a deviation in the exposure time (exposure timing) between the image captured by the right camera <NUM> and the image captured by the left camera <NUM>. The deviation between the left and right images is caused by a deviation in the exposure time in a part of both images when the left and right images are geometrically corrected. The exposure time difference adjusting unit <NUM> includes a time difference calculation unit <NUM> that calculates an exposure time difference by using a difference in correction amount between the geometric correction table right <NUM> and the geometric correction table left <NUM>, and a coordinate adjustment unit <NUM> that corrects an image by using the exposure time difference and a vehicle signal (vehicle information) such as a vehicle speed and a steering angle (details will be described later).

The table reference coordinate generation unit <NUM> sequentially generates coordinates (x, y) to refer to the geometric correction table (the geometric correction table right <NUM> and the geometric correction table left <NUM>) in order to perform geometric correction on the entire image from the upper left to the lower right.

The parallax calculation unit <NUM> uses the image corrected by the geometric correction (right) unit <NUM> as a reference image, and performs stereo matching processing in order to calculate parallax data with the image corrected by the geometric correction (left) unit <NUM>. The parallax data is generated for each pixel of the entire image. The parallax data generated by the parallax calculation unit <NUM> is input to the road surface estimation unit <NUM> and the object detection unit <NUM>.

The road surface estimation unit <NUM> extracts a portion of the road surface on which the own vehicle is traveling in the parallax data from the image, and calculates a parameter of the road surface. The parameter of the road surface calculated by road surface estimation unit <NUM> is input to the object detection unit <NUM>.

Based on the parallax data from the parallax calculation unit <NUM> and the parameter of the road surface from the road surface estimation unit <NUM>, the object detection unit <NUM> detects a three-dimensional object on the road surface using the parallax data and the image data. The information detected by the object detection unit <NUM> is input to the vehicle control unit <NUM>.

When the three-dimensional object detected by the object detection unit <NUM> becomes an obstacle to traveling, the vehicle control unit <NUM> generates and outputs a control instruction for controlling a brake, a suspension, and the like provided in the vehicle in order to avoid a collision with the obstacle or reduce an impact due to the collision.

<FIG> is a diagram illustrating images before and after geometric correction of lens distortion of a camera image. When a square lattice pattern is photographed, a right camera image <NUM> becomes an image distorted by the optical lens <NUM>. When this image is corrected by the geometric correction (right) unit <NUM>, a geometrically transformed right image <NUM> is obtained. A curve <NUM> in the right camera image <NUM> is a straight line <NUM> in the geometrically transformed right image <NUM>. Similarly, when a square lattice pattern is photographed, a left camera image <NUM> becomes an image distorted by the optical lens <NUM>. When this image is corrected by the geometric correction (left) unit <NUM>, a geometrically transformed left image <NUM> is obtained. A curve <NUM> in the left camera image <NUM> is a straight line <NUM> in the geometrically transformed left image <NUM>.

Normally, since the distortion characteristics of the right optical lens <NUM> and the left optical lens <NUM> are different, for example, even if the straight line <NUM> and the straight line <NUM> have the same y coordinate, the y coordinate of the curve <NUM> and the y coordinate of the curve <NUM> before transformation of these straight lines may be different. Since the image sensors <NUM> and <NUM> as optical sensors (such as a CCD sensor or a CMOS sensor) of the right camera <NUM> and the left camera <NUM> are the rolling shutter type, the exposure time of the curve <NUM> and the exposure time of the curve <NUM> are shifted from each other. This is because the exposure time of the rolling shutter is shifted in units of rasters. When the vehicle is moving, the subject is photographed while shifted to the same y coordinate in the left and right images (the geometrically transformed right image <NUM> and the geometrically transformed left image <NUM>) after the geometric correction.

<FIG> is a diagram illustrating an example in which a subject is shifted and photographed at the same y coordinate of left and right images. The object <NUM> on the road surface of the geometrically transformed left image <NUM> and the object <NUM> on the road surface of the geometrically transformed right image <NUM> are the same object but have a shift in the y coordinate on the image. Since the stereo matching processing is based on the premise that the same subject is photographed at the same y coordinate on the left and right when the parallax between the left and right images is calculated, an error occurs in the stereo matching in this case. The reason why the y coordinates of the object <NUM> and the object <NUM> are shifted as illustrated in <FIG> is that the exposure time is shifted although the y coordinates of the straight line <NUM> and the straight line <NUM> illustrated in <FIG> are the same. Therefore, it is necessary to correct the geometric transformation so that the data of the same coordinates on the left and right of the geometrically transformed image becomes pixel data with the same exposure timing.

<FIG> is a diagram illustrating an example of geometric correction processing using a geometric correction table. In <FIG>, the left camera image <NUM> has <NUM> pixels of <NUM> pixels in the x direction and <NUM> pixels in the y direction, and the geometrically transformed left image <NUM> has <NUM> pixels of <NUM> pixels in the x direction and <NUM> pixels in the y direction. The geometric correction table left <NUM> indicates which pixel of the left camera image <NUM> (before the geometric transformation) is used for each pixel of the geometrically transformed left image <NUM>. This example indicates that the pixel data of x = <NUM> and y = <NUM> of the left camera image <NUM> (before the geometric transformation) is used for the coordinates of x = <NUM> and y = <NUM> of the geometrically transformed left image <NUM>. Similarly, for the right camera image <NUM>, pixel data of x = <NUM> and y = <NUM> of the right camera image <NUM> (before the geometric transformation) is used for the coordinates of x = <NUM> and y = <NUM> of the geometrically transformed right image <NUM>, as indicated by the geometric correction table right <NUM>. According to this example, in the pixel data of x = <NUM> and y = <NUM> of the left and right geometrically transformed images, the right camera image <NUM> has y = <NUM> and the left camera image <NUM> has y = <NUM>, and there is a difference of two rasters. If the difference between the two rasters of the image sensors <NUM> and <NUM> is multiplied by the time difference of the exposure timing per raster, the deviation amount (difference) of the exposure timing of the pixel is obtained. That is, by comparing the correction data (correction amount) indicated at the same coordinate position in the left and right geometric correction tables, the deviation amount (exposure time difference) of the exposure timing between the same coordinates of the left and right geometrically corrected images can be known.

The moving distance and the moving direction of the vehicle are calculated based on the deviation amount, and the positional deviation of one of the left and right images is corrected. In this embodiment, the right image is used as the reference image for the stereo processing, and the left image is corrected. In the present embodiment, the numerical value of the geometric correction table is expressed in an integer format, but it is also possible to express the numerical value in a small number format for improving accuracy. In addition, the geometric correction table may be thinned out to about <NUM> pixel units, and the pixels therebetween may be interpolated to reduce the table size.

The correction of the positional deviation of the image described above will be described. <FIG> is a diagram illustrating correspondence between a two-dimensional image <NUM> indicated by x and y coordinates and a three-dimensional space indicated by X, Y, and Z. When the point Q in the three-dimensional space is displayed at the point q of the two-dimensional image <NUM>, the following Expressions (<NUM>) and (<NUM>) are obtained. In Expressions (<NUM>) and (<NUM>), f represents a focal distance. If (x, y) on the image and Z of the pixel are known, X and Y are determined by converting Expressions (<NUM>) and (<NUM>). <NUM>) <MAT> (Math. <NUM>) <MAT>.

<FIG> is a diagram illustrating an example of an image obtained by photographing a road surface <NUM> during traveling. Assuming that the road surface <NUM> is a flat surface and has no inclination in the lateral direction, the relationship between the y coordinate and the Z value on the road surface <NUM> is represented by the following Expression (<NUM>). In Expression (<NUM>), α is a y coordinate of a vanishing point <NUM>, and β is a parameter of the road surface obtained by a ground height, a focal distance, or the like of the camera. <NUM>) <MAT>.

When Expression (<NUM>) is converted into an expression for obtaining a Z value to calculate a Z value from the y coordinate on the road surface <NUM>, and x and y on the road surface <NUM> and the Z value are applied to the expressions obtained by converting Expressions (<NUM>) and (<NUM>) to obtain X and Y, the coordinates (X, Y, Z) of a three-dimensional space corresponding to a point (x, y) of the object <NUM> on the road surface <NUM> can be calculated. When the moving distance and the moving direction of the vehicle obtained from the difference between the y coordinate values of the left and right geometric correction table illustrated in <FIG> are added to the (X, Y, Z) and the (x, y) of the left camera image is obtained again by Expressions (<NUM>) and (<NUM>), it is possible to obtain an image in which the deviation between the exposure times of the geometrically transformed right image <NUM> and the geometrically transformed left image <NUM> is corrected.

As described above, the time difference calculation unit <NUM> of the exposure time difference adjusting unit <NUM> illustrated in <FIG> compares the correction data (correction amount) indicated at the same coordinate position in the left and right geometric correction tables (the geometric correction table right <NUM> and the geometric correction table left <NUM>) to obtain the difference, and multiplies (in consideration of) the time difference of the exposure timing per raster to obtain the deviation amount (exposure time difference) of the exposure timing between the same coordinates of the left and right geometrically corrected images.

In addition, as described above, the coordinate adjustment unit <NUM> of the exposure time difference adjusting unit <NUM> corrects the positional deviation of the image, in other words, adjusts the geometric correction position of the left image, using the deviation amount (exposure time difference) of the exposure timing and the vehicle information regarding the position or movement of the vehicle. The vehicle information includes a turning angle, an own vehicle speed, and the like, and is obtained from, for example, a steering angle, a yaw rate, and a speed sensor mounted on the vehicle. In addition, as the vehicle information, a turning angle, an own vehicle speed, and the like may be calculated from an own vehicle position. As a result, it is possible to calculate the moving distance and the moving direction of the vehicle that has moved in the deviation period of the exposure timing.

<FIG> illustrates an operation flow of the stereo camera device <NUM>.

First, images are captured by the right camera <NUM> and the left camera <NUM> (S100). Next, the geometric transformation processing is performed to correct lens distortion of the image. For this purpose, the table reference coordinate generation unit <NUM> updates the coordinates (x, y) to which the geometric correction table is referred (S110). The exposure time difference adjusting unit <NUM> calculates reference coordinates (x', y') of the left image based on the exposure time difference between the left and right images (S120). Details of this portion will be described later based on <FIG>. Next, the geometric correction table right <NUM> for correction of the right image is accessed with coordinates (x, y), and the geometric correction table left <NUM> for correction of the left image is accessed with coordinates (x', y'). Using the results of accessing the respective geometric correction tables, the geometric correction (right) unit <NUM> and the geometric correction (left) unit <NUM> perform image correction processing as illustrated in <FIG> (S130). This correction processing is performed on the entire image (S140).

Next, using the geometrically corrected left and right images, the parallax calculation unit <NUM> generates parallax data (S150). The road surface estimation unit <NUM> creates a road surface parameter (α, β) using the parallax data (S160). Using the parallax data, the object detection unit <NUM> detects an object on the road surface, calculates a position, a distance, and a size of the object, and detects the road obstacle (S170). Then, the vehicle control unit <NUM> generates a control instruction for controlling the brake, the suspension, and the like according to the position of the road obstacle so as not to cause an obstacle in traveling of the own vehicle (S180), and then determines the ending of the system (S190).

<FIG> illustrates a processing flow of the exposure time difference adjusting unit <NUM> (the processing flow of S120 in <FIG> described above). The left and right geometric correction tables are accessed in accordance with (x, y), and a difference s in the number of rasters is obtained from the difference between the correction coordinate values recorded in the left and right geometric correction tables as illustrated in <FIG> (S200). This s is indicated by a table indicating the deviation in exposure time (a difference in exposure timing for each pixel) with a time ratio in units of rasters. The exposure time difference r between the left and right cameras of the pixel of (x, y) is obtained (by the time difference calculation unit <NUM>) from the difference s (S210). When the exposure time difference per raster of the image sensor is δ, the exposure time difference r is expressed by the following Expression (<NUM>). <NUM>) <MAT>.

Subsequently, a distance w and a direction (u, v) in which the vehicle has moved during the period of the exposure time difference r are obtained based on vehicle information such as an own vehicle speed and a turning angle (S220). Next, the coordinates of the correction portion (x, y) of the left image are converted into (X, Y, Z) in a three-dimensional space using Expressions (<NUM>), (<NUM>), and (<NUM>) (S230). The (u, v, w) is added to the (X, Y, Z) and applied to Expressions (<NUM>) and (<NUM>) to obtain the position (x', y') on the image as follows (by the coordinate adjustment unit <NUM>) (S240). <NUM>) <MAT> (Math. <NUM>) <MAT>.

The processing of S130 of <FIG> is performed using this (x', y') as described above.

As described above, the image processing device <NUM> of the first embodiment includes the geometric correction unit (the geometric correction (right) unit <NUM>, the geometric correction (left) unit <NUM>) that corrects the distortion of the images captured by the plurality of rolling shutter-type image sensors <NUM> and <NUM>, the parallax calculation unit <NUM> that calculates the parallax using the corrected image corrected by the geometric correction unit, and the exposure time difference adjusting unit <NUM> that adjusts the geometric correction position of at least one image (left image) based on the difference between the geometric correction amount (geometric correction table right <NUM>) used for distortion correction of the image (right image) as a reference captured by the pixel sensor <NUM> and the geometric correction amount (geometric correction table left <NUM>) used for distortion correction of the other image (left image) captured by the pixel sensor <NUM> and the vehicle information (own vehicle speed, turning angle, etc.) regarding the position or movement of the vehicle.

In addition, a geometric correction table (the geometric correction table right <NUM> and the geometric correction table left <NUM>) as a geometric correction amount is included, and the exposure time difference adjusting unit <NUM> obtains a difference in exposure timing for each pixel between the images from a difference in the correction amounts of the geometric correction table right <NUM> of the image (right image) serving as a reference and the geometric correction table left <NUM> of the other image (left image), and adjusts the geometric correction position of at least one image (left image) based on the difference in exposure timing for each pixel and the vehicle information.

As described above, in the first embodiment, the function of adjusting the deviation of the exposure timings of the left and right images is realized by adjusting the reference position (geometric correction position) of the geometric correction table used for the geometric correction processing of correcting the lens distortion of the left and right images. Specifically, in the stereo camera device <NUM>, a difference between the exposure timings of pixels at the same position is calculated for the left and right images after correcting the geometric distortion of the lens, the image is corrected in accordance with the distance and direction in which the vehicle has moved based on the difference, and the exposure timings of the pixels of the left and right images are matched.

As a result, according to the first embodiment, for example, in the stereo camera device <NUM>, pixels on the same raster of the respective images (left and right images) become images with the same exposure timing, and thus, there is an effect that an error in parallax calculation of the stereo matching processing can be reduced. In addition, it is not necessary to exclusively provide the image conversion processing for adjusting the deviation of the exposure timing, and as a result, it is possible to reduce the amount of hardware and the calculation load.

<FIG> is a system configuration diagram of the stereo camera device <NUM> on which the image processing device <NUM> including the exposure time difference adjusting unit <NUM> that previously holds a time difference between exposure timings of pixels of left and right images as a table (exposure left table <NUM>) in the second embodiment is mounted. The difference from <FIG> is that the exposure time difference adjusting unit <NUM> holds in advance a raster difference (for each pixel) of the same coordinate portion of the geometric correction table right <NUM> and the geometric correction table left <NUM> as the exposure difference table <NUM>. The raster difference corresponds to s in Expression (<NUM>) described above.

The exposure time difference adjusting unit <NUM> of the image processing device <NUM> extracts the value of the exposure difference table <NUM> corresponding to the pixel to be adjusted (corresponding to the deviation amount of the exposure timing between the same coordinates of the left and right geometrically corrected images), and corrects the positional deviation of the image using the extracted value and the vehicle information regarding the position or movement of the vehicle, in other words, adjusts the geometric correction position of the left image.

As described above, the image processing device <NUM> of the second embodiment includes the exposure difference table <NUM> indicating the difference in the exposure timing for each pixel between the images captured by the respective image sensors <NUM> and <NUM>, and the exposure time difference adjusting unit <NUM> extracts the value of the exposure difference table <NUM> corresponding to the pixel to be adjusted, and adjusts the geometric correction position of at least one image (left image) based on the extracted value and the vehicle information.

As described above, in the first embodiment illustrated in <FIG>, s is calculated in each frame from the geometric correction table right <NUM> and the geometric correction table left <NUM>, but in the second embodiment illustrated in <FIG>, the calculation of s can be omitted by the exposure difference table <NUM> to reduce the amount of calculation.

As a result, according to the second embodiment, it is possible to reduce the calculation load (of the exposure time difference adjusting unit <NUM>) in the image processing device <NUM> in addition to obtaining the same operation and effect as those of the first embodiment described above.

In the stereo camera device, in addition to the image deviation due to the distortion of the optical lens, the optical axis deviation of the left and right camera images may occur due to secular change or temperature change. Since a parameter of the distortion of the optical lens is determined when the lens is processed, the correction (distortion correction) is static geometric correction. On the other hand, in the correction of the optical axis deviation (amount) accompanying secular change or temperature change, a parameter is determined when the stereo camera device is used, and thus, the correction is dynamic geometric correction.

<FIG> is a system configuration diagram of the stereo camera device <NUM> on which the image processing device <NUM> provided with a dynamic geometric correction unit <NUM> according to a third embodiment is mounted. It is assumed that the dynamic geometric correction unit <NUM> detects the amount of optical axis deviation of an image during operation due to secular change or temperature change by a method as disclosed in <CIT>, for example, and calculates the amount of longitudinal deviation (also referred to as vertical deviation) of the image. The amount of longitudinal deviation is sent to the geometric correction (left) unit <NUM>, and processing of deviating the image captured by the left camera <NUM> in the longitudinal direction by an amount indicated by the amount of longitudinal deviation is performed. Alternatively, the amount of longitudinal deviation is sent to the geometric correction (right) unit <NUM>, and processing of deviating the image captured by the right camera <NUM> in the longitudinal direction by an amount indicated by the amount of longitudinal deviation is performed. In addition, the exposure time difference adjusting unit <NUM> adds (or subtracts) the amount of longitudinal deviation to the value of the exposure difference table <NUM> and calculates a final deviation amount of raster. Using this result, the exposure time difference adjusting unit <NUM> calculates the reference coordinates of the geometric correction table left <NUM> in the same manner as in the first and second embodiments described above.

As described above, the image processing device <NUM> of the third embodiment includes the dynamic geometric correction unit <NUM> that detects the amount of optical axis deviation of the image in operation, and the exposure time difference adjusting unit <NUM> adjusts the geometric correction position of at least one image (left image) using the deviation (difference) of the exposure timing of the image and (the amount of longitudinal deviation of the image corresponding to) the amount of optical axis deviation detected by the dynamic geometric correction unit <NUM>.

As described above, in the third embodiment, even when the stereo camera device <NUM> has a deviation in the left and right camera images due to secular change or temperature change, the deviation of the exposure timing is corrected in consideration of the deviation, and the image is corrected based on the correction.

As a result, according to the third embodiment, it is possible to calculate more correct parallax data in addition to obtaining the same operational effects as those of the first and second embodiments described above.

<FIG> is a diagram illustrating an example of correcting the image of a road sign <NUM> as an object other than an object on the road surface. In order to correct the image of the object, a distance Z is required in addition to the (x, y) on the image. The distance Z of the object at the road surface height can be obtained by the above Expression (<NUM>). However, the distance Z cannot be obtained for an object other than the road surface height by this method. Therefore, in the fourth embodiment, a method using parallax data will be described.

<FIG> is a processing flowchart of the parallax calculation unit <NUM> that obtains the distance Z of an object by using parallax data.

First, the coordinates (x, y) of a portion to be corrected in the image are specified (S300). Thereafter, the location (x", y") of the (x, y) is searched for from the image one frame before (S310). Parallax data d of the coordinates of (x", y") is converted into a distance Z" by the following Expression (<NUM>) (S320). In Expression (<NUM>), f is a focal distance, B is a distance between the left and right cameras, and a is a size of one pixel of the image sensor. <NUM>) <MAT>.

Next, the distance Z is obtained by adding a distance traveled by the own vehicle in one frame period to the distance Z" (S330). From the (x, y) and the Z, (X, Y) can be calculated by modifying Expressions (<NUM>) and (<NUM>) (S340). Then, in order to obtain an image in which the exposure timings are matched on the left and right sides, correction coordinates of the image can be calculated by Expressions (<NUM>) and (<NUM>).

As described above, according to the method of the fourth embodiment, it is possible to also correct an object other than the object on the road surface to an image in which the exposure timings of the left and right images are matched.

As a result, according to the fourth embodiment, it is possible to calculate correct parallax data in a wider range in addition to obtaining the same operational effects as those of the first to third embodiments described above.

Note that, in the above-described embodiment, the stereo camera device <NUM> includes two (left and right) cameras, but the number of cameras constituting the stereo camera device <NUM> may be three or more.

In addition, in the above-described embodiment, the camera (the left camera <NUM> and the right camera <NUM>) including the image sensor constituting the stereo camera device <NUM> and the image processing device <NUM> are configured as separate bodies. However, the camera and a part (for example, the geometric correction (left) unit <NUM>, the geometric correction (right) unit <NUM>, the exposure time difference adjusting unit <NUM>, and the like) or the whole of the image processing device <NUM> may be integrated, in other words, a part (for example, the geometric correction (left) unit <NUM>, the geometric correction (right) unit <NUM>, the exposure time difference adjusting unit <NUM>, and the like) or the whole of the image processing device <NUM> may be incorporated in the camera.

Each of the above configurations, functions, processing units, processing means, and the like may be partially or entirely achieved by hardware by, for example, designing by an integrated circuit. Each of the above configurations, functions, and the like may be achieved by software by a processor interpreting and executing a program that achieves each function. Information such as a program, a table, and a file for achieving each function can be stored in a memory device such as a memory, a hard disk, or a solid-state drive (SSD), or a recording medium such as an integrated circuit (IC) card, a secure digital (SD) card, or a digital versatile disc (DVD).

Claim 1:
An image processing device comprising:
a geometric correction unit (<NUM>, <NUM>) configured to correct distortion of a first image and a second image captured by a first rolling shutter-type image sensor (<NUM>) and a second rolling shutter-type image sensor (<NUM>), respectively, mounted in a vehicle; and
a parallax calculation unit (<NUM>) configured to calculate a parallax using a first corrected image and a second corrected image that are corrected by the geometric correction unit (<NUM>, <NUM>); characterized by
an exposure time difference adjusting unit (<NUM>) configured to adjust a geometric correction position of at least one image based on a difference between a geometric correction amount used for distortion correction of a first image serving as a reference and a geometric correction amount used for distortion correction of a second image and based on vehicle information regarding a position or movement of the vehicle.