Patent Description:
Conventionally, in a technical field of a stereocamera, as a parallax calculation algorithm for calculating parallax for each pixel, a block matching method of a feature point, a semi-global-matching (SGM) propagation method, and the like, are known. In these methods, when finding a feature point from each of an image of a left side and an image of a right side, a cost is calculated for each parallax, and a parallax having a minimum cost that is found in a search space is obtained as an integer parallax. Further, a parallax d including a subpixel parallax estimated by a predetermined calculation method is calculated, and a distance corresponding to each pixel is calculated using an expression indicating a relationship between the parallax d and a distance Z (Z=BF/d). That is, the above mentioned methods can be said that a method of voting cost in a parallax space (note that B is a distance between cameras, and F is a focal length).

In the above mentioned conventional parallax space cost voting method, it is known that assuring distance resolution is difficult for a remote region having a small integer parallax (that is, having a large distance Z). Thus, in a remote region, variance of a calculation result of parallax is larger, and variance of a measured value of distance tends to be larger.

For example, in a case in which a distance measurement system is installed on a vehicle, as exemplified by autonomous driving technology, accuracy of distance measurement of a remote place tends to be required. To meet the requirement, by using LIDAR (Light Detection And Ranging, or Laser Imaging Detection And Ranging) not having high spatial resolution but having high distance resolution, a method of integrating a measured result that is measured by a stereocamera (having high spatial resolution but not having high distance resolution of a remote place) and a measured result that is measure by LIDAR is known (the above mentioned integration may also be referred to as "fusion"). By the fusion, a sensor having measurement performance exceeding performance limits of a stereocamera and LIDAR can be developed. That is, even when measuring a distance in a remote place, a range image having small variance of distance resolution and high spatial resolution may be able to output. For example, it is expected that the fusion enables highly accurate distance measurement, low variance of a measured distance value, separation of discontinuous surface, and improvement of robustness with respect to environment.

As an example of the above mentioned fusion, a method of correlating distance information obtained by LIDAR with a depth image captured by a stereocamera is known (see Non-Patent Document <NUM> for example). Non-Patent Document <NUM> discloses a fusion method for increasing density of a low-texture region in a parallax image by using distance information obtained from LIDAR.

This document discloses a method for sensor combination on a stereo and a Time-of-Flight camera system.

This document discloses a probabilistic approach to Time-of-Flight and stereo data fusion.

This document discloses a low resolution Time-Of-Flight (TOF) depth image camera based on Photonic Mixer Devices with two standard cameras in a stereo configuration.

<CIT> discloses an information processing apparatus, an information processing method and a program.

However, the technique disclosed in Non-Patent Document <NUM> does not necessarily perform integration by utilizing advantages of both distance measurement methods. Generally, in an integration method of related arts, after a stereocamera outputs a range image by performing the block matching, distance information measured by LIDAR is added to the range image. In this method, because error in the range image output from the stereocamera is large, improvement of accuracy was limited even if the distance information obtained from the LIDAR was added.

That is, in the technique of related art, integration of LIDAR, which does not have high spatial resolution but has high distance resolution, and a stereocamera, which has high spatial resolution but does not have high distance resolution of a remote place, is not sufficient.

The present disclosure is made to solve the above problem, and aims at providing a distance measurement method effectively utilizing advantages of both of these distance measurement methods.

According to one aspect of the present disclosure, an image processing method of generating a range image is provided. The method includes the steps as specified in claim <NUM>.

According to the present disclosure, a distance measurement method having improved accuracy of distance measurement, at least with respect to a remote region, can be provided.

In the following, as embodiments of the present disclosure, a distance measurement system, and a method of measuring a distance (distance measurement method) performed by the distance measurement system will be described.

First, distance measuring performance of LIDAR and a stereocamera will be described with reference to <FIG>. <FIG> is a graph illustrating a relationship between a distance Z and distance resolution, with respect to LIDAR and a stereocamera. Note that, if a value of the distance resolution is smaller, it is expressed as "the distance resolution is superior" or "accuracy of distance measurement is superior" (conversely, if a value of the distance resolution is large, it is expressed as "the distance resolution is poor" or "accuracy of distance measurement is low"). As illustrated in <FIG>, the distance resolution of LIDAR is almost constant regardless of a value of the distance Z, but the distance resolution of a stereocamera becomes sharply larger in accordance with increase of the distance Z. That is, with respect to a stereocamera, accuracy of distance measurement greatly degrades in a remote region.

<FIG> is a graph illustrating a relationship between a distance Z and spatial resolution with respect to LIDAR. Note that, if a value of the spatial resolution is smaller, it is expressed as "the spatial resolution is superior" or "accuracy of spatial measurement is superior" (conversely, if a value of the spatial resolution is large, it is expressed as "the spatial resolution is poor" or "accuracy of spatial measurement is low"). In <FIG>, spatial resolution is illustrated for each emitting interval of <NUM> degrees, <NUM> degrees, <NUM> degrees, <NUM> degrees, <NUM> degrees, and <NUM> degrees (emitting resolution). However, regardless of magnitude of the emitting resolution, the spatial resolution becomes larger in accordance with increase of the distance Z.

<FIG> is a graph illustrating a relationship between a distance Z and spatial resolution with respect to stereocamera. As spatial resolution of a stereocamera is superior because a stereocamera can resolve space for each pixel, the spatial resolution does not sharply increase in accordance with increase of the distance Z.

Based on the above result, the following points need to be improved.

A distance measurement system <NUM> according to embodiments of the present disclosure solves the above problems. Also, as will be described below, when a range image is obtained by using an algorithm called semi-global-matching (SGM), a problem that a boundary of an object is lost in the range image or a problem of dilation of an object region may occur. The distance measurement system <NUM> according to the embodiments of the present disclosure can suppress occurrence of the above problems.

An act of finding a corresponding point of a certain point is referred to as "matching", and a degree of matching is expressed as an evaluation value. The evaluation value expressing a degree of matching may sometimes be referred to as "cost", "resemblance", or "dissimilarity"). When dissimilarity is low or when a resemblance is high, it means that both points match more. Dissimilarity and a resemblance may be generally expressed as a "matching level".

Next, definitions of resolution will be described with reference to <FIG> is a diagram for explaining spatial resolution and distance resolution. Distance resolution is a capability (measure of accuracy) of distinguishing a difference of distance to objects. In a case in which distance resolution (in a Z-direction) is <NUM>, an object that is <NUM> distant from an origin and an object that is <NUM> distant from the origin cannot be distinguished and both of the objects are recognized as a same one (or it is determined that both of the objects are at the same distance).

Spatial resolution is a capability of distinguishing objects that are separate from each other on a two-dimensional space. In a case in which spatial resolution is <NUM>, two objects that are <NUM> distant from each other on an XY-plane cannot be distinguished, and the two objects are recognized as a single object. Spatial resolution may also be referred to as "angle resolution".

Measurement direction means a direction when a distance to an object is measured, and means a direction where the object is positioned. A pixel specified by a measurement direction includes not only the pixel but also surrounding pixels of the pixel.

A distance evaluation value with respect to distance information represents an evaluation value (cost) determined in accordance with uncertainty of distance to a pixel (or surrounding pixels) specified with an emitting direction of an electromagnetic wave. In the present embodiment, the distance cost may be referred to as LIDAR cost CLI(p, Z). A matching evaluation value represents a degree of matching measured by block matching. In the present embodiment, the matching evaluation value may be referred to as stereo matching cost CST(p, Z).

<FIG> is a diagram for explaining a distance Z that is obtained by using a general parallax space. A graph (a) of <FIG> is for illustrating a cost C(p, d) or a propagation cost Lr(p, d) by using a block matching and an SGM algorithm, and a horizontal axis of the graph (a) represents a shift amount (to be described below). The graph (a) of <FIG> is an example in which a search range is <NUM> pixels. A variable p represents a pixel under consideration (may also be referred to as a "target pixel"), and a variable d represents a shift amount (searched parallax) between a reference image and a comparison image. The smallest cost C(p, d) or the smallest propagation cost Lr(p, d) in the search range of <NUM>-pixel is adopted as a parallax (integer parallax) of the target pixel p.

A graph (b) of <FIG> illustrates a cost C(p, Z) or a propagation cost Lr(p, Z) in Z-space. Note that the distance Z is obtained from the parallax d in the graph (a) by using the following formula (<NUM>).

In the above formula (<NUM>), B represents a distance between optical axes of a left camera and a right camera for a stereocamera, and F represents focal length of the left and right cameras. As illustrated in the graph (b), in the Z-space, density of the distances Z's with respect to which the cost C(p, Z) or the propagation cost Lr(p, Z) is obtained is not constant. This is because a variable d is included in a denominator of the formula (<NUM>) used for calculating the distance Z. That is, the distance Z is inversely proportional to the variable d, and the distance Z greatly varies in a case in which a value of the variable d is close to <NUM>.

Accordingly, in a general block matching method, it is equivalent that coarse cost propagation is performed with respect to a remote region. Thus, it is difficult to perform highly accurate distance measurement with respect to a remote region.

<FIG> is a diagram illustrating an example of a conventional integration (fusion) method of distance information of a LIDAR <NUM> and a range image of a stereocamera <NUM>. In a general conventional integration method, after the stereocamera <NUM> outputs a range image by performing the block matching, the LIDAR <NUM> added distance information measured by the LIDAR <NUM> to the range image. In this method, as described above with reference to <FIG>, because error in the range image output from the stereocamera <NUM> is large, improvement of accuracy was limited even if the distance information obtained from the LIDAR <NUM> was added.

In the present embodiment, as illustrated in <FIG>, distance information measured by the LIDAR <NUM> is integrated before the stereocamera <NUM> outputs a range image generated by the block matching or the like. <FIG> is a diagram illustrating an example of an integration method of distance information of the LIDAR <NUM> and a range image of the stereocamera <NUM>. Before the stereocamera <NUM> outputs a range image, the stereocamera <NUM> integrates distance information measured by the LIDAR <NUM> with cost C(p, Z).

As illustrated in <FIG>, when integrating distance information, the stereocamera <NUM> calculates cost C(p, Z) in the Z-space. <FIG> illustrates cost C(p, Z) or propagation cost Lr(p, Z) in the Z-space. Z-space having uniform density of the distances is prepared in advance, and in the Z-space, LIDAR cost is added to the cost C(p, Z) calculated by the stereocamera <NUM>. As propagation of cost by the SGM algorithm is also performed in the Z-space, a distance Z having the smallest cost can be identified, and a range image of excellent distance resolution can be obtained. Further, as spatial resolution of a range image is fundamentally excellent, a high-quality and high-resolution range image can be obtained.

As described above, in the distance measurement system according to the present embodiment, because integration of distance information measured by LIDAR is performed in Z-space before a stereocamera outputs a range image generated by the block matching or the like, a high-quality and high-resolution range image can be obtained.

An example of application of the distance measurement system <NUM> will be described with reference to <FIG> is a diagram illustrating the distance measurement system <NUM> installed in an automobile <NUM> which is an example of a moving body (in the following, the automobile <NUM> may also be referred to as a "moving body <NUM>"). In <FIG>, the distance measurement system <NUM> is fitted to a central position of a windshield, inside the moving body <NUM>. The distance measurement system <NUM> includes a laser radar distance measurement unit <NUM> and a stereogram processing unit <NUM>. Both the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> are disposed such that a region ahead of the moving body <NUM> is a region of which a distance is measured. Note that the present embodiment describes a case in which the laser radar distance measurement unit <NUM> is disposed between two camera units (image capturing units, or may also be referred to as a capturing means) of a stereocamera in the stereogram processing unit <NUM>.

Laser radar may also be referred to as the above mentioned LIDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging). In the present embodiment, laser radar and LIDAR are not distinguished. Laser radar or LIDAR emits pulses of light in a range out of human vision, and measures time until the emitted light returns, to calculate a distance. When light is emitted to a certain direction and the light returns, the laser radar distance measurement unit <NUM> records the direction of the light and a measured distance as a point in a <NUM>-D map in which the laser radar distance measurement unit <NUM> is centered.

Although <FIG> illustrates a distance measurement system <NUM> in which the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> are unified, the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> may be separated.

<FIG> is a diagram illustrating an example of a distance measurement system <NUM> in which the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> are separated. In <FIG>, the laser radar distance measurement unit <NUM> is installed inside a front grille, and the stereogram processing unit <NUM> is disposed at the front side in a cabin (such as in a vicinity of a back side of a rear-view mirror). In the present embodiment, a configuration of a distance measurement system <NUM> is not limited to that illustrated in <FIG> or <FIG>, and any configuration may be adopted as long as distance information output from a laser radar distance measurement unit <NUM> can be integrated with distance information measured by a stereogram processing unit <NUM>.

<FIG> are exemplary diagrams illustrating a range to which the laser radar distance measurement unit <NUM> emits laser light. <FIG> is a top view of the moving body <NUM>, and <FIG> is a side view of the moving body <NUM>.

As illustrated in <FIG>, the laser radar distance measurement unit <NUM> emits laser light by scanning horizontally in a predetermined range ahead of an advancing direction of the moving body <NUM>. The laser light may be regarded as light, or may be regarded as electromagnetic wave.

Also, as illustrated in <FIG>, the laser radar distance measurement unit <NUM> emits laser light to a predetermined range ahead of an advancing direction of the moving body <NUM>. The laser radar distance measurement unit <NUM> can measure distance to a target located approximately up to hundreds of meters ahead, although a distance that the laser light can reach depends on power of the laser radar distance measurement unit <NUM>. With respect to a distance to a target closely located, a distance to a target located less than <NUM> meter ahead can be measured. However, as there is little need to measure distance to such a target, a range of distance that the laser radar distance measurement unit <NUM> can measure may be determined in advance.

The laser radar distance measurement unit <NUM> is configured to scan horizontally while changing a direction of laser light emitted in an elevation angle direction. Accordingly, the laser radar distance measurement unit <NUM> can emit light over a range from a location close to an installed position of the laser radar distance measurement unit <NUM> to a location away from the installed position of the laser radar distance measurement unit <NUM>.

<FIG> are exemplary diagrams illustrating a range to which the stereogram processing unit <NUM> can capture an image. A set of a reference image and a comparison image is referred to as a stereogram. <FIG> is a top view of the moving body <NUM>. The stereogram processing unit <NUM> includes two image capturing units (a right camera <NUM> and a left camera <NUM>) whose optical axes are directed ahead of an advancing direction of the moving body <NUM> (a set of the right camera <NUM> and the left camera <NUM> corresponds to the stereocamera mentioned above), and captures images of a predetermined range in the advancing direction. A part of a range irradiated with laser light overlaps with at least a part of a range captured by the stereocamera.

<FIG> illustrate images each captured by the right camera <NUM> and the left camera <NUM> respectively. In the present embodiment, the image captured by the right camera <NUM> is referred to as a reference image, and the image captured by the left camera <NUM> is referred to as a comparison image. The right camera <NUM> and the left camera <NUM> are disposed at the same horizontal level, and are spaced from each other at a predetermined distance. Thus, the reference image overlaps the comparison image, but an object in the reference image is located in the comparison image at a position shifted horizontally.

The stereogram processing unit <NUM> calculates a shift amount (which is parallax) of an object in the comparison image from the object in the reference image, to generate and output a range image. The stereogram processing unit <NUM> also associates distance information with a pixel of a stereogram.

In another embodiment, either the right camera <NUM> or the left camera <NUM> may be omitted. That is, a stereogram can be obtained by using a monocular camera. In the following, a method of generating a stereogram by using a monocular camera will be described with reference to <FIG>.

<FIG> illustrates an example in which the stereogram processing unit <NUM> has both a right camera <NUM> and a left camera <NUM>. First, a large number of sets of a reference image and a comparison image are prepared (by using the stereogram processing unit <NUM> in <FIG>), and learning of a comparison image corresponding to a reference image is performed using deep learning (alternatively, learning of a reference image corresponding to a comparison image may be performed). In the following description, a case in which learning of a comparison image corresponding to a reference image is performed will be described.

Each pixel value of a reference image is input to an input layer of a DNN (Deep Neural Network) <NUM>. The DNN <NUM> also includes an intermediate layer and an output layer. The intermediate layer is formed by combining at least one convolutional layer, at least one pooling layer, a neural network, and an encoder-decoder network, and the intermediate layer is expressed as a set of coefficients of a two-dimensional filter. The output layer outputs each pixel value of an estimated comparison image. Based on a difference between a pixel value output from the output layer and a pixel value of an actual comparison image, coefficients of the two-dimensional filter are adjusted using backpropagation. The adjustment of the coefficients of the two-dimensional filter using a sufficient number of sets of a reference image and a comparison image corresponds to the learning of the DNN <NUM>. Note that initial values of the two-dimensional filter may be obtained by using an autoencoder.

<FIG> illustrates an example of a set of a monocular camera processing unit <NUM> and the DNN <NUM>, which are installed on the moving body <NUM> after learning process has been completed. When the set of the monocular camera processing unit <NUM> and the DNN <NUM> starts working, the monocular camera processing unit <NUM> outputs a reference image only. The DNN <NUM> serves as an output unit for outputting a comparison image. When the reference is input to the DNN <NUM>, the DNN <NUM> outputs an estimated comparison image. Although the estimated comparison image obtained by the set of the monocular camera processing unit <NUM> and the DNN <NUM> is not equal to a comparison image captured by a left camera <NUM>, it is confirmed that the estimated comparison image has a quality enough to generate a range image. The monocular camera processing unit <NUM> performs block matching using the reference image and the estimated comparison image.

Therefore, in the present embodiment, a stereogram can be obtained by using either the right camera <NUM> or the left camera <NUM>. It is not necessary to have a stereocamera. In other words, a stereogram is necessary in the present embodiment, but a means or method for generating a stereogram is not limited to a specific one.

Next, Relationship between a position to which laser light is emitted by the laser radar distance measurement unit <NUM> and a position of a pixel of a reference image of a stereogram captured by the stereogram processing unit <NUM> will be described with reference to <FIG>. <FIG> is an exemplary diagram illustrating a relationship between a position to which laser light is emitted and a position of a pixel of a reference image of a stereogram.

A direction to which laser light is emitted by the laser radar distance measurement unit <NUM> (which may also be referred to as an "emitting direction (of laser)" in the present embodiment) can be correlated with a position of a pixel of a reference image in advance. In <FIG>, two objects O<NUM> and O<NUM> seen from a side view, and an example of a reference image on which the objects O<NUM> and O<NUM> are captured is illustrated in <FIG>. As the objects O<NUM> and O<NUM> are positioned on a same line passing through the object O<NUM> (or O<NUM>) and the laser radar distance measurement unit <NUM> or the right camera <NUM>, the object O<NUM> is layered on the object O<NUM> in the reference image.

Suppose a case in which a height h<NUM> of the object O<NUM> is double a height h<NUM> of the object O<NUM>, and a distance L<NUM> from the moving body <NUM> to the object O<NUM> is double a distance L<NUM> from the moving body <NUM> to the object O<NUM> because a height of the laser radar distance measurement unit <NUM> from a road surface is much smaller than a distance L<NUM> from the moving body <NUM> to an object O<NUM>. As a ratio of height (h<NUM>) to distance (L<NUM>) with respect to the object O<NUM> is the same as a ratio of height (h<NUM>) to distance (L<NUM>) with respect to the object O<NUM>, the objects O<NUM> and O<NUM> appear on the reference image in the same size. Also, because of a positional relationship among the objects O<NUM> and O<NUM> and the moving body <NUM>, the object O<NUM> is layered on the object O<NUM> in the reference image. Therefore, if laser light passed on the top end of the objects O<NUM> and O<NUM>, the laser light would appear on the top end of the objects O<NUM> and O<NUM> in the reference image captured by the stereogram processing unit <NUM> (note that the laser light does not actually appear on the reference image visually because the laser light is not visible light). As described above, as there is a one-to-one correspondence between a laser light emitting direction and a pixel position of a reference image, they can be correlated with each other in advance.

In <FIG>, emitting directions corresponding to pixels P1 to P4 in a reference image are illustrated. For example, a pixel of coordinates P1 (x1, y1) corresponds to an emitting direction of θ1 in a horizontal angle direction and Φ1 in an elevation angle direction, a pixel of coordinates P2 (x2, y2) corresponds to an emitting direction of θ2 in the horizontal angle direction and Φ2 in the elevation angle direction, a pixel of coordinates P3 (x3, y3) corresponds to an emitting direction of θ3 in the horizontal angle direction and Φ3 in the elevation angle direction, and a pixel of coordinates P4 (x4, y4) corresponds to an emitting direction of θ4 in the horizontal angle direction and Φ4 in the elevation angle direction.

Therefore, when the laser radar distance measurement unit <NUM> outputs an emitting direction and distance information, the stereogram processing unit <NUM> can correlate the measured distance information with a pixel.

<FIG> illustrates an example of a functional block diagram of the laser radar distance measurement unit <NUM>. The laser radar distance measurement unit <NUM> includes a signal processing unit <NUM>, an elevation angle direction scan drive unit <NUM>, a motor <NUM>, an elevation angle direction scan mirror <NUM>, a laser light detecting unit <NUM>, an amplifier <NUM>, an interval counter <NUM>, a laser output unit <NUM>, and a laser driver <NUM>.

Based on an instruction from the signal processing unit <NUM>, the elevation angle direction scan drive unit <NUM> actuates the motor <NUM> to rotate the elevation angle direction scan mirror <NUM> in the elevation angle direction. By the operation being performed, the elevation angle direction scan mirror <NUM> rotates in the elevation angle direction.

Further, based on an instruction from the signal processing unit <NUM>, the laser driver <NUM> is activated, and the laser output unit <NUM> emits laser light. At this time, information of a time (may also be referred to as "output timing") when the laser light is temporarily emitted is retained in the interval counter <NUM>. As the laser light emitted by the laser output unit <NUM> is output via the elevation angle direction scan mirror <NUM>, a predetermined range is irradiated with the laser light.

The output laser light is reflected by an object located in an emitting direction, and the reflected light is received by the laser light detecting unit <NUM> via the elevation angle direction scan mirror <NUM>. The laser light detecting unit <NUM> includes multiple photodetectors (PD's) which are arranged vertically. The laser light that has entered the laser light detecting unit <NUM> is received by one of the photodetectors, and is converted to an electrical signal.

The electrical signal generated by the laser light detecting unit <NUM> is amplified at the amplifier <NUM>, and is input to the interval counter <NUM>. Based on the output timing of the laser light emitted by the laser output unit <NUM> and a time (may also be referred to as reception timing) when the reflected laser light is received at the laser light detecting unit <NUM>, the interval counter <NUM> calculates a time interval.

The time interval calculated by the interval counter <NUM> is converted to distance information at the signal processing unit <NUM>, and the distance information is output to the stereogram processing unit <NUM>, with information indicating the emitting direction.

The signal processing unit <NUM> also includes a failure monitoring unit 601a. The failure monitoring unit 601a monitors whether or not failure has occurred in the laser radar distance measurement unit <NUM>. For example, in a case in which a time interval calculated by the interval counter <NUM> (or distance information calculated by the signal processing unit <NUM>) does not vary for a certain period of time, the failure monitoring unit 601a determines that failure has occurred. Alternatively, if a state of distance information being out of range specified by a specification has continued for a certain period of time, or if the signal processing unit <NUM> has reached a temperature more than a regulated temperature, the failure monitoring unit 601a may determine that failure has occurred. When failure has been detected, the laser radar distance measurement unit <NUM> sends, to the stereogram processing unit <NUM>, a notification that failure has occurred.

The stereogram processing unit <NUM> monitors an entirety of the laser radar distance measurement unit <NUM>. For example, the stereogram processing unit <NUM> detects that no response is received from the laser radar distance measurement unit <NUM>, that communication with the laser radar distance measurement unit <NUM> has failed, and that a predetermined magnitude of voltage is not entered from the laser radar distance measurement unit <NUM> (power-off).

<FIG> is a schematic diagram of the distance measurement system <NUM>. <FIG> also illustrates each function of the stereogram processing unit <NUM> as a block. As the distance measurement system <NUM> is a device for measuring distance, it can also be referred to as a distance measurement device. The distance measurement system <NUM> may also be referred to as other names, such as a distance measurement unit.

As illustrated in <FIG>, the stereogram processing unit <NUM> includes the right camera <NUM>, the left camera <NUM>, a distortion adjusting unit <NUM>, and a distance calculation unit <NUM>. The stereocamera is formed by the right camera <NUM> and the left camera <NUM>.

The distortion adjusting unit <NUM>, the distance calculation unit <NUM>, or a combination of the distortion adjusting unit <NUM> and the distance calculation unit <NUM> may be implemented by dedicated electronic circuitry. Alternatively, the distortion adjusting unit <NUM> and/or the distance calculation unit <NUM> may be implemented by software, by executing programs embodying the distortion adjusting unit <NUM> and/or the distance calculation unit <NUM> on a computer (central processing unit (CPU)). Thus, the stereogram processing unit <NUM> has a function of an information processing device (apparatus). Further, as the stereogram processing unit <NUM> performs image processing, the stereogram processing unit <NUM> may also be regarded as an image processing device (apparatus).

The distortion adjusting unit <NUM> applies a general distortion adjustment to a reference image and a comparison image. When the adjustment is applied to a reference image and a comparison image, the reference image and the comparison image are adjusted such that no differences other than parallax are contained with respect to each other. The adjustment of an image becomes available by performing calibration in advance. For example, before installation of the left camera <NUM> and the right camera <NUM>, an object for calibration (such as a checkered chart) is captured by the left camera <NUM> and the right camera <NUM>. By comparing the captured images, a geometrical conversion look-up table (LUT) for converting image data (data of the captured images) is generated, in order to minimize a difference of the captured images caused by hardware-level allowable error such as distortion of lens, deviation of an optical axis, a difference of focal length, and distortion of image capturing element. The distortion adjusting unit <NUM> performs adjustment of an image with reference to such a LUT.

The distance calculation unit <NUM> calculates parallax by applying a specific algorithm, such as the block matching algorithm or the SGM algorithm, to a reference image and a comparison image. Also, before the distance calculation unit <NUM> outputs a range image, the distance calculation unit <NUM> integrates (fuses) LIDAR cost CLI (p, Z) with stereo matching cost CST (p, Z) with respect to distance information output by the laser radar distance measurement unit <NUM>. A process regarding the integration performed by the distance calculation unit <NUM> is referred to as an "integration process". The stereo matching cost CST (p, Z) is an example of the matching evaluation value, and the LIDAR cost CLI (p, Z) is an example of the distance evaluation value.

The distance calculation unit <NUM> also includes a failure monitoring unit 14a. The failure monitoring unit 14a monitors whether or not failure has occurred in the stereogram processing unit <NUM>. For example, in a case in which pixel values of a reference image or a comparison image remain unchanged for a certain period of time, the failure monitoring unit 14a determines that failure has occurred. Alternatively, if a state of pixel values being out of range specified by a specification has continued for a certain period of time, or if the distance calculation unit <NUM> has reached a temperature more than a regulated temperature, the failure monitoring unit 14a may determine that failure has occurred. When failure has been detected, the stereogram processing unit <NUM> sends a notification that failure has occurred, to an electronic control unit (ECU) <NUM>. The stereogram processing unit <NUM> also sends a notification to the ECU <NUM> when a notification that failure has occurred has been received from the laser radar distance measurement unit <NUM>, or when detecting failure in the laser radar distance measurement unit <NUM>.

The ECU <NUM> monitors an entirety of the stereogram processing unit <NUM>. For example, the ECU <NUM> detects that no response is received from the stereogram processing unit <NUM>, that communication with the stereogram processing unit <NUM> has failed, and that a predetermined magnitude of voltage is not entered from the stereogram processing unit <NUM> (power-off).

In <FIG>, an example in which a range image and a reference image are sent out to the ECU <NUM> is described. The ECU <NUM> is a control unit for a moving body such as a vehicle. In a case in which the distance measurement system <NUM> is installed in a moving body, the distance measurement system <NUM> may be referred to as an on-board device. The ECU <NUM> performs various driver-assistance by using the range image and the reference image output by the distance measurement system <NUM>. The reference image is used for recognizing a preceding vehicle, a pedestrian, a lane marking, a state of a traffic signal, and the like, by applying various pattern matchings.

Functions of the driver-assistance differ depending on vehicles. An example of the functions of the driver-assistance includes an alarming function or a braking function. In the alarming function or braking function, when a horizontal position of an object in the reference image is overlapped with a width of a moving body in which the ECU <NUM> is installed, an alarm is output or braking is performed in accordance with a time to collision (TTC) calculated based on a distance and relative velocity. Further, if it is difficult to stop the moving body until collision occurs, a steering operation is performed to avoid collision.

The ECU <NUM> also performs control of a space between the moving body and a preceding vehicle while the moving body is moving, such that the space is changed in accordance with speed of the moving body. The ECU <NUM> stops the moving body when the preceding vehicle stops, and the ECU <NUM> also starts the moving body when the preceding vehicle starts moving. In a case in which the ECU <NUM> is configured to recognize a lane marking, the ECU <NUM> can perform lane keeping control in which the moving body is steered such that the moving body runs in a middle of a lane, or the ECU <NUM> can perform lane departure avoidance control in which, when the moving body starts to deviate from a current lane, a driving direction of the moving body is changed such that the moving body remains in the current lane.

Further, when the moving body is started, if an obstacle is present in a driving direction of the moving body, the ECU <NUM> can prevent an abrupt starting of the moving body. For example, if an obstacle is found in a driving direction which is determined by a position of a gearshift, and if an amount of operation of a gas pedal is large, the ECU <NUM> can relieve damage by limiting engine power or warning a driver.

The ECU <NUM> is connected to a display device <NUM>. Examples of the display device <NUM> include a flat panel display (such as an LCD or an organic EL display) fitted to a center console or a dashboard. The display device <NUM> may also be a head-up display (HUD). When failure occurred in the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM>, the ECU <NUM> displays information on the display device <NUM> indicating that the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM> has failed. Displayed examples will be described in a third embodiment.

Note that the configuration illustrated in <FIG> is merely an example. For example, the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> may be integrated. Alternatively, the ECU <NUM> may have a function of the stereogram processing unit <NUM>.

A calculation method of integer parallax using the block matching algorithm will be described with reference to <FIG> is a diagram illustrating an example of calculating SAD (Sum of Absolute Difference) as a cost of a pixel p = (Px3, Py5) of which a cost is to be calculated, with respect to a reference image <NUM> captured by the right camera <NUM> and a comparison image <NUM> captured by the left camera <NUM>. Note that, in the present embodiment, a pixel of which a cost is to be calculated may be referred to as a "pixel of interest". In the example of <FIG>, the pixel p = (Px3, Py5) is a pixel of interest. Also, a mathematical expression of SAD will be described below.

Because the reference image <NUM> and the comparison image <NUM> have been captured from different locations, an object corresponding to the pixel p = (Px3, Py5) in the reference image <NUM> is different from an object corresponding to the pixel p = (Px3, Py5) in the comparison image <NUM>, although coordinates of the two pixels are the same. The pixel p = (Px3, Py5) in the comparison image <NUM> corresponds to an object shifted in a horizontal direction from a location of an object corresponding to the pixel p = (Px3, Py5) in the reference image <NUM>. Thus, a difference between luma of the pixel of interest p = (Px3, Py5) in the reference image <NUM> and luma of the pixel of interest p = (Px3, Py5) in the comparison image <NUM>, which is the SAD when a block size is 1x1 pixel, becomes large.

Next, a pixel of interest in the comparison image <NUM> of which the SAD is to be calculated is changed (shifted) in a right direction by one pixel. That is, the SAD when parallax is assumed to be <NUM> (shift amount d=<NUM>) is calculated. Specifically, a difference between luma of the pixel of interest p = (Px3+<NUM>, Py5) in the comparison image <NUM> and luma of the pixel of interest p = (Px3, Py5) in the reference image <NUM> is calculated. In the example of <FIG>, the SAD also becomes large when d is <NUM>.

Subsequently, the shift amount d is changed gradually (such as d = <NUM>, <NUM>,. ), and the SAD is calculated for each value of the shift amount d. In the example of <FIG>, when the parallax is assumed to be <NUM> (when d=<NUM>), an object corresponding to the pixel p = (Px3, Py5) in the reference image <NUM> coincides with an object corresponding to the pixel p = (Px3+<NUM>, Py5) in the comparison image <NUM>. Accordingly, the SAD when d=<NUM> becomes smaller than SAD of other cases (the cases in which d is not <NUM>).

<FIG> illustrates an example of a calculation result of SAD with respect to a certain pixel of interest (p = (Px3, Py5)) by varying the shift amount d. The SAD is an example of cost C (p, d). Regarding the pixel (p = (Px3, Py5)), as the SAD becomes smallest when d=<NUM>, the parallax (d) is determined as <NUM>.

<FIG> illustrates an example of a calculation result of SAD with respect to another pixel of interest (p = (Px4, Py6)) by varying the shift amount d. In the example of <FIG>, because a variation of the SAD is small in accordance with change of the shift amount d, the distance calculation unit <NUM> cannot determine the parallax. As described here, because there may exist a pixel whose parallax cannot be detected only by using the block matching, the distance calculation unit <NUM> performs an energy calculation processing (SGM algorithm) to make parallax apparent.

When the SAD for each shift amount d (cost C (p, d)) as illustrated in <FIG> has been calculated, it is preferable that fractional part of parallax is calculated. Examples of method for obtaining fractional part of parallax include a high-degree (sixth-degree) polynomial fitting, a high-degree (fourth-degree) polynomial fitting, and a parabola fitting.

The SAD is expressed as the following mathematical expression: <MAT>.

As can be seen from the above expression, the SAD is obtained by calculating an absolute value of a luma difference for each pixel and calculating a sum of the absolute values. The SAD becomes smaller as pixels resemble each other.

Further, measures other than the SAD, such as SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), or ZNCC (Zero-mean Normalized Cross Correlation), may be used for block matching.

The SSD is expressed as the following mathematical expression: <MAT>.

The SSD is obtained by calculating a square of a luma difference for each pixel and calculating a sum of the absolute values. The SSD becomes smaller as pixels resemble each other.

The NCC is expressed as the following mathematical expression: <MAT>.

A numerator of the NCC represents a sum of a scalar product of lumas of pixels. The scalar product becomes larger as pixels resemble each other. An expression in a denominator of the NCC is for normalizing the numerator, and the denominator becomes larger as pixels resemble each other. A maximum of the NCC is <NUM>, and a minimum of the NCC is <NUM>.

The ZNCC is expressed as the following mathematical expression: <MAT> <MAT>.

The ZNCC corresponds to a normalized cross correlation after subtracting a mean value. The subtracting operation corresponds to a removal of DC (Direct Current) component of a signal, and the NCC is effective for comparing images each having different brightness. In the above expression of the ZNCC, M represents the number of pixels in a horizontal direction, and N represents the number of pixels in a vertical direction.

Other than the above mentioned measures, ZSAD (Zero-mean Sum of Absolute Difference) or ZSSD (Zero-mean Sum of Squared Difference) may be used. The ZSAD corresponds to a sum of absolute difference after subtracting a mean value. The ZSSD corresponds to a sum of squared difference after subtracting a mean value.

The distance calculation unit <NUM> calculates a propagation cost Lr by using an algorithm called SGM, and calculates an energy cost S(p, d) of a pixel of interest p by using the propagation cost Lr. Note that the SGM algorithm is a form of a dense matching algorithm.

First, a process for calculating propagation cost (may also be referred to as "propagation cost function") Lr by using the SGM algorithm is described. <FIG> is a schematic diagram illustrating the process for calculating propagation cost Lr by using the SGM algorithm.

<FIG> illustrates a case in which propagation costs Lr of four directions are calculated with respect to a pixel of interest <NUM>. Specifically, with respect to the pixel <NUM>, a propagation cost L<NUM> in a direction of an arrow <NUM>, a propagation cost L<NUM> in a direction of an arrow <NUM>, a propagation cost L<NUM> in a direction of an arrow <NUM>, and a propagation cost L<NUM> in a direction of an arrow <NUM> are calculated. Note that directions (r) of the propagation cost to be calculated with respect to the pixel <NUM> are not limited to the above mentioned four directions. For example, propagation costs of eight directions or propagation costs of two directions may be calculated.

As illustrated in <FIG>, the propagation cost L<NUM> in the direction of the arrow <NUM> can be obtained based on the following formula (<NUM>).

Note that variable p and d in the above formula (<NUM>) respectively represent coordinates of the pixel p <NUM> and a parallax. Also, in the above formula (<NUM>), (p-<NUM>) represents coordinates of a pixel located left of the pixel p <NUM> by one pixel, and (p+<NUM>) represents coordinates of a pixel located right of the pixel p <NUM> by one pixel (hereinafter, the pixel located left of the pixel p <NUM> by n pixels is referred to as a "pixel (p-n)", and the pixel located right of the pixel p <NUM> by n pixels is referred to as a "pixel (p+n)"). Further, P<NUM> and P<NUM> are predetermined constants. As described here, the propagation cost L<NUM>(p, d) can be calculated based on the cost C(p, d) of the pixel <NUM>, and propagation costs of the pixel (p-<NUM>) corresponding to different parallax (such as d-<NUM>, d, or d+<NUM>). That is, the propagation cost in the direction of the arrow <NUM> is calculated sequentially from left to right. Note that a propagation interval of cost, when the propagation cost is to be calculated from left to right, is not limited to one pixel. That is, the propagation cost L<NUM>(p, d) may be calculated by using propagation costs of a pixel (p-a) ("a" is natural number) corresponding to different parallax.

Similarly, the propagation cost L<NUM> in the direction of the arrow <NUM> is calculated sequentially from top to bottom. Also, the propagation cost L<NUM> in the direction of the arrow <NUM> is calculated sequentially from right to left, and the propagation cost L<NUM> in the direction of the arrow <NUM> is calculated sequentially from bottom to top.

Next, a process for calculating an energy cost S(p, d) of a pixel of interest p by using the propagation cost Lr will be described.

The energy cost S(p, d) of each pixel is calculated in accordance with the following formula (<NUM>), based on the propagation costs of various directions.

Thus, in the example of <FIG>, S(p, d) can be obtained by calculating S(p, d) = L<NUM>(p, d) + L<NUM>(p, d) + L<NUM>(p, d) + L<NUM>(p, d).

Next, a process for calculating a distance of each pixel performed by the distance measurement system <NUM> will be described with reference to <FIG> is an exemplary flowchart illustrating a process of the distance measurement system <NUM>.

At step S1, the laser radar distance measurement unit <NUM> acquires distance information. In parallel with step S1, in the stereogram processing unit <NUM>, the right camera <NUM> captures a reference image and the left camera <NUM> captures a comparison image (step S2). The distortion adjusting unit <NUM> applies a distortion adjustment to each of the reference image and the comparison image such that no differences other than parallax are contained (step S3). Subsequently, the stereogram processing unit <NUM> calculates stereo matching cost CST(p, Z) (step S4).

Steps S2 to S4 may be executed synchronously or asynchronously with step S1. In a case in which steps S2 to S4 are executed asynchronously with Step S1, the stereogram processing unit <NUM> may use the latest distance information obtained from the laser radar distance measurement unit <NUM>.

<FIG> is a diagram illustrating an example of the stereo matching cost CST(p, Z). <FIG> illustrates a state in which cost in a parallax space (which is a coordinate space in which a horizontal axis corresponds to a parallax and a vertical axis corresponds to cost) obtained by stereo matching is converted to the stereo matching cost CST(p, Z) in Z-space (which is a coordinate space in which a horizontal axis corresponds to the distance Z and a vertical axis corresponds to cost). If cost in a parallax space obtained by stereo matching is simply converted to the stereo matching cost in Z-space, the stereo matching cost CST(p, Z) at regular intervals cannot be obtained. Thus, the distance calculation unit <NUM> interpolates the cost obtained by stereo matching. In <FIG>, circles represent costs obtained by stereo matching, and squares represent costs obtained by interpolation. Any interpolation appropriate to curve fitting may be used, such as parabola fitting, a high-degree polynomial interpolation, and a spline interpolation. In the example of <FIG>, the stereo matching cost CST(p, Z) is calculated at every <NUM> meters. Note that a variable offset illustrated in <FIG> represents a fraction to obtain the distance Z every <NUM> meters (by converting d into Z).

At step S5, the distance calculation unit <NUM> calculates the cost C(p, Z) by fusing (integrating) LIDAR cost CLI(p, Z) with the stereo matching cost CST(p, Z). The fusion is performed in accordance with the following formula (<NUM>). <MAT> where.

First, the LIDAR cost CLI(p, Z) will be described with reference to <FIG> and <FIG>. <FIG> is a diagram illustrating an example of the LIDAR cost CLI(p, Z), and <FIG> is a diagram for explaining the LIDAR cost CLI(p, Z) supplementarily. Also, a function in the formula (<NUM>) described below represents an example of the LIDAR cost CLI(p, Z). <MAT> where.

In order to integrate distance information obtained from the laser radar distance measurement unit <NUM> with the stereo matching cost CST(p, Z), the inventors of the present application have defined the LIDAR cost CLI(p, Z) anew. As illustrated in <FIG>, the LIDAR cost CLI(p, Z) takes the minimum when p is equal to r<NUM> (a pixel corresponding to a center of emitted laser light), and the LIDAR cost CLI(p, Z) becomes larger as p becomes apart from r<NUM>. As is apparent from the formula (<NUM>), in a case in which A is <NUM>, the LIDAR cost CLI(p, Z) of a pixel r<NUM> is <NUM> (the LIDAR cost CLI(p, Z) when p = r<NUM> is <NUM>). Conversely, the LIDAR cost CLI(p, Z) of a pixel distant from r<NUM> is <NUM>.

Each rectangle illustrated in <FIG> represents a pixel, and a pixel r<NUM> is illustrated in the center of a group of pixels in <FIG>. Also, a horizontal resolution and a vertical resolution of a pulse of laser light are expressed in pixels. A horizontal range in which a pulse of laser light is spread horizontally is referred to as a horizontal resolution, and a vertical range in which a pulse of laser light is spread vertically is referred to as a vertical resolution. In the example of <FIG>, a horizontal resolution is <NUM> pixels and a vertical resolution is <NUM> pixels. A pulse of laser light spreads wider as the laser light travels further. However, an area (number of pixels) captured by a stereocamera is substantially constant regardless of distance (the reason will be described below). Thus, regardless of the distance Z with respect to the pixel r<NUM> measured by the laser radar distance measurement unit <NUM>, the LIDAR cost CLI(p, Z) may be calculated with respect to pixels (p) within a range determined by the horizontal resolution and the vertical resolution.

When the LIDAR cost CLI(p, Z) has been calculated in accordance with the formula (<NUM>), the distance calculation unit <NUM> performs a voting of the LIDAR cost CLI(p, Z) to the stereo matching cost CST(p, Z). <FIG> is an exemplary diagram illustrating a voting of the LIDAR cost CLI(p, Z) to the stereo matching cost CST(p, Z). As the distance Z of the pixel r<NUM> corresponding to an emitting direction of laser has been obtained by the laser radar distance measurement unit <NUM>, a voting of the LIDAR cost CLI(p, Z) is performed to the stereo matching cost CST(p, Z) specified by the pixel r<NUM> and the distance Z. Note that the voting in the present embodiment means an action of adding a value. The LIDAR cost CLI(p, Z) with respect to the pixel r<NUM> is <NUM>-A, and if A is <NUM>, the LIDAR cost is <NUM>.

To each pixel p included in a region specified by the pixel r<NUM>, the horizontal resolution and the vertical resolution (which is a rectangular region illustrated in <FIG> having a width of the horizontal resolution and a height of the vertical resolution, in the center of which the pixel r<NUM> is positioned), the distance calculation unit <NUM> performs the above mentioned voting (adding the LIDAR cost CLI(p, Z)). Magnitude of the LIDAR cost CLI(p, Z) added to the stereo matching cost CST(p, Z) corresponding to each pixel is a value calculated in accordance with the function of the formula (<NUM>), by using a length between the pixel r<NUM> and the pixel p.

As described above, in the present embodiment, the stereo matching cost CST(p, Z) and the LIDAR cost CLI(p, Z) can be fused on the Z-space. Also, cost of the pixel r<NUM> corresponding to an emitting direction of laser becomes the smallest, and cost of surrounding pixels of the pixel r<NUM> becomes larger.

Note that a shape of a graph of the LIDAR cost CLI(p, Z) illustrated in <FIG> is merely an example. For example, the shape may be of rectangular shape, such that cost in a predetermined range from the pixel r<NUM> takes minimal value. Alternatively, the shape may be of reverse triangle shape, similar to <FIG>. Further, according to the formula (<NUM>) described above, cost is calculated by adding the stereo matching cost CST(p, Z) and the LIDAR cost CLI(p, Z). However, cost may be calculated by subtracting the LIDAR cost CLI(p, Z) from the stereo matching cost CST(p, Z). Further, the LIDAR cost CLI(p, Z) may be a negative value.

Next, regarding a process at step S5, supplemental explanation will be made with reference to <FIG>, <FIG>. In the following, degradation of accuracy of distance information, with respect to pixels surrounding a pixel corresponding to a center of emitted laser light, will be described. Even for the pixels within a horizontal resolution range and a vertical resolution range, accuracy of the distance information with respect to the pixels degrades if the pixels are apart from the pixel corresponding to a center of emitted laser.

<FIG> are exemplary diagrams illustrating the number of pixels on an object surface (note that an object surface means an area that an object occupies in image data). <FIG> represents the number of pixels of a certain object, and <FIG> represents a relation between a size of the object and the number of pixels of an object surface.

Suppose a case in which an object having a width of N [m] and a height of M [m] is located at a point distant from the distance measurement system <NUM> by Z [m]. By using a focal length f, the number of horizontal pixels X and the number of vertical pixels Y occupied by an image of the object in image data (captured by the stereocamera) can be calculated based on the following formulas. <MAT> <MAT> where pt is pixel pitch in the above formulas.

<FIG> is an exemplary diagram illustrating the number of pixels (xL and yL) in image data (captured by the stereocamera) that are occupied by emitted laser light, in a case in which laser light is emitted on an object located at a point distant from the distance measurement system <NUM> by Z [m]. Let a width of the laser light emitted on the object be A, and let a height of the laser light emitted on the object be B. Also, let a horizontal resolution of laser light be θx [deg], and let a vertical resolution of laser light be θy [deg]. The width A and the height B of the laser light are expressed as the following formulas. <MAT> <MAT>.

Further, let the number of pixels in a horizontal direction in image data corresponding to an area irradiated with the laser light be xL, and let the number of pixels in a vertical direction in image data corresponding to the area irradiated with the laser light be yL. The number of pixels xL and yL are expressed as the following formulas. <MAT> <MAT>.

As is apparent from the above formulas, an area of a surface of an object irradiated with laser light (may also be referred to as an "irradiated surface") becomes larger as the distance Z increases, but the number of pixels occupied by the irradiated surface remains constant, regardless of magnitude of the distance Z (note that the above description is only applicable when a light source of laser light is a point source. If a light source has a certain size, the number of pixels occupied by the irradiated surface becomes larger in a short range (in a case in which the distance Z is small), and the number of pixels converges to a certain value when the distance Z becomes larger).

Based on the above formulas, a condition (of the distance Z) that an object surface covers an irradiated surface when laser light is emitted to an object is obtained. <FIG> is an exemplary diagram illustrating a state in which an object surface covers an irradiated surface.

The condition in which an object surface covers an irradiated surface can be expressed as "X > xL and Y > yL". The expressions can also be transformed as in the following. <MAT> and <MAT>.

Thus, the condition of Z in which an object surface covers an irradiated surface can be expressed as "N/(2tan(θx/<NUM>)) > Z and M/(2tan(θy/<NUM>)) > Z".

When dimensions of an object are "N=M=<NUM>, and θx=θy=<NUM> [deg]", such as a vehicle, <NUM> > Z.

Also, when a dimension of an object is "N=<NUM>, M=<NUM>, θx=<NUM> [deg], and θy=<NUM> [deg]", <NUM> > Z.

Therefore, in a case in which an object (on which laser light is emitted) is in a close distance (when the distance Z is small), an object surface covers an irradiated surface. Conversely, in a case in which an object is distant (when the distance Z is large), an irradiated surface becomes larger than an object surface.

<FIG> is a diagram illustrating a relation between an irradiated surface and an object surface state when an object is in a close distance. <FIG> is a diagram illustrating a relation between an irradiated surface and an object surface state when an object is distant. In <FIG>, similar to <FIG>, an object surface is larger than an irradiated surface. As a size of an irradiated surface is only xL pixels in width and only yL pixels in height, it is uncertain at which point laser light is reflected on an object surface (XY in <FIG>). It means that reliability of distance information measured by the laser radar distance measurement unit <NUM> degrades as a point at which laser light is reflected is close to a periphery of an irradiated surface. Thus, as described above with reference to <FIG>, it is effective to configure the LIDAR cost CLI(p, Z) to become larger as p becomes close to a periphery of an irradiated surface.

<FIG> indicates that an irradiated surface becomes larger than an object surface when an object is distant. In such a case, as laser light is not necessarily reflected on a surface of an object (XY plane illustrated in <FIG>), laser light may travel deeper than a location of the object. This case may also be referred to as "multi-pulse". In a case in which multi-pulse occurs, multiple objects may be detected. Details of multi-pulse case will be described below.

Next, at step S6, the distance calculation unit <NUM> calculates propagation cost Lr(p, Z). The propagation cost Lr(p, Z) is expressed as follows (formula (<NUM>)).

The first term of the propagation cost Lr(p, Z) is the cost C(p, Z) obtained by fusing LIDAR cost CLI(p, Z) with the stereo matching cost CST(p, Z). The second term of the propagation cost Lr(p, Z) is cost obtained by performing the SGM algorithm in the Z-space. The propagation cost Lr(p, Z) is calculated based on the first term and the second term.

However, in the present embodiment, the propagation cost obtained by performing the SGM algorithm in the Z-space is not necessarily required. That is, cost calculation by the SGM algorithm may not be performed.

At step S7, the distance calculation unit <NUM> determines whether the propagation cost Lr(p, Z) has been calculated in all pixels. Steps S5 and S6 are repeatedly executed until the propagation cost Lr(p, Z) has been calculated in all pixels.

After the propagation cost Lr(p, Z) has been calculated in all pixels, the distance calculation unit <NUM> calculates energy cost function S(p, Z) at step S8 (S(p, Z) may also be referred to as "energy cost S(p, Z)").

<FIG> is an example of a diagram illustrating a method of calculating the energy cost S(p, Z). It is considered that cost of a pixel is related to cost of surrounding pixels. Thus, when obtaining the propagation cost Lr(p, Z) of a certain pixel (may also be referred to as a "target pixel"), by adding the propagation cost Lr(p, Z) of surrounding pixels of the target pixel, the propagation cost Lr(p, Z) of the target pixel is calculated more accurately.

<FIG> illustrates a case in which the propagation cost Lr(p, Z) of eight surrounding pixels of the target pixel is added. That is, the energy cost of the target pixel S(p, Z) is expressed as the following formula (<NUM>).

<FIG> is a schematic diagram illustrating a method of calculating the energy cost S(p, Z). As expressed in the formula (<NUM>), the propagation cost Lr(p, Z) of eight surrounding pixels of the target pixel is added in the Z-space. By performing this calculation, the energy cost S(p, Z) of the target pixel is obtained.

The above mentioned calculation of adding the propagation cost Lr(p, Z) of eight surrounding pixels is merely an example. The number of the propagation cost Lr(p, Z) terms (in the formula (<NUM>)) to be added (such as the propagation cost of four pixels, five pixels, or <NUM> pixels) may be determined based on a calculation workload and accuracy of calculated distance. Alternatively, the addition of the propagation cost Lr(p, Z) may not be performed.

At step S9, the distance calculation unit <NUM> determines distance that minimizes the energy cost S(p, Z) (the distance is denoted by Z<NUM>). The distance Z<NUM> is determined as a distance of the target pixel.

<FIG> is a diagram illustrating an example of a method of obtaining the distance Z<NUM> that minimizes the energy cost S(p, Z). Distance Z (Z<NUM>) that minimizes the energy cost S(p, Z) is assumed to be most probable distance of the target pixel.

Further, in order to calculate fractional part of distance Z, a high-degree (sixth-degree) polynomial fitting, a high-degree (fourth-degree) polynomial fitting, a parabola fitting, and the like, may be used.

After the distance Z<NUM> (distance minimizing the energy cost S(p, Z)) is determined for all pixels, the process in <FIG> terminates.

Next, a process performed when multi-pulse has occurred, which has been described above with reference to <FIG>, will be described. <FIG> is an exemplary diagram illustrating the process performed when multi-pulse has occurred. A diagram (a) of <FIG> is a top view of a range in which laser light emitted from the laser radar distance measurement unit <NUM> is spread, and a diagram (b) of <FIG> is an exemplary diagram illustrating a relationship between a light reception level and a distance Z.

As illustrated in the diagram (a) of <FIG>, in a case in which an irradiated surface becomes larger than an object surface, the laser radar distance measurement unit <NUM> receives light reflected from multiple objects (objects O<NUM> and O<NUM> in the example of <FIG>). When a relationship between a power level of light received by the laser radar distance measurement unit <NUM> (may also be referred to as a "light reception level") and a distance Z is illustrated as a graph, as illustrated in the diagram (b) of <FIG>, the light reception level becomes larger at points of distances Z<NUM> and Z<NUM> where the objects O<NUM> and O<NUM> are placed. Accordingly, the laser radar distance measurement unit <NUM> can detect that multiple objects are present. As the laser radar distance measurement unit <NUM> generally determines that an object is present at a location where the light reception level is larger than a threshold, the laser radar distance measurement unit <NUM> can detect distances of two objects each placed at a different distance, in a case illustrated in the diagram (b) of <FIG>.

When distances of two objects each placed at a different distance has been detected with a pulse of light, the distance calculation unit <NUM> performs integration of the stereo matching cost CST(p, Z) and the LIDAR cost CLI(p, Z) with respect to two distances of the respective two objects. That is, to the stereo matching cost CST(p, Z) of a pixel r<NUM> and surrounding pixels corresponding to an emitting direction of laser light with respect to a distance Z<NUM>, the LIDAR cost CLI(p, Z) is added. Also, to the stereo matching cost CST(p, Z) with respect to a distance Z<NUM>, the LIDAR cost CLI(p, Z) is added.

In a conventional technique, it is difficult for one pixel to integrate cost with respect to two distances (Z<NUM> and Z<NUM>). However, in the present embodiment, because integration is performed on the Z-space, cost with respect to two distances can be appropriately integrated. If the laser radar distance measurement unit <NUM> detects distances of multiple objects, it means that the detected distance information is ambiguous. Thus, when calculating the LIDAR cost CLI(p, Z) in a case in which distances of multiple objects are detected, the LIDAR cost CLI(p, Z) may be adjusted such that the LIDAR cost CLI(p, Z) becomes larger.

A method of measuring distance performed by the laser radar distance measurement unit <NUM> described above with reference to <FIG> utilizes Time-of-Flight (TOF) principle. However, other methods such as Fast-Chirp Modulation (FCM) or Frequency Modulated Continuous Wave (FMCW) can be used. In the FCM and the FMCW, distance is obtained by converting a frequency of a beat signal caused by a slight frequency difference between a transmitting wave and a receiving wave.

A method of measuring distance by the FCM will be described with reference to <FIG>, <FIG>, <FIG>. Models of a transmitting wave, receiving wave, and a reflecting wave are illustrated in <FIG>. As illustrated in <FIG>, a transmitting wave emitted by a millimeter wave transceiver (MMW transceiver) <NUM> is reflected by an object <NUM>, and a reflecting wave (part of the transmitting wave being reflected by the object <NUM>) is received by the MMW transceiver <NUM> as a receiving wave. Let a distance between the MMW transceiver <NUM> and the object <NUM> be R.

<FIG> are diagrams for explaining frequencies of the transmitting wave and the receiving wave. In the FCM, a frequency of a signal (transmitting wave) is gradually caused to be increased as time passes. <FIG> illustrates a state in which frequencies of the transmitting wave <NUM> and the receiving wave <NUM> are increasing gradually. A signal in which a frequency varies with time is referred to as a chirp. As illustrated in <FIG>, frequencies of the transmitting wave <NUM> and the receiving wave <NUM> are increasing as time passes, while amplitudes of the transmitting wave <NUM> and the receiving wave <NUM> are constant. Note that the receiving wave <NUM> is observed with a delay Δt, which corresponds to a time until the transmitting wave <NUM> reflected by the object <NUM> is returned. Thus, because a frequency of the receiving wave <NUM> is slightly different from a frequency of the transmitting wave <NUM>, a beat signal is generated.

<FIG> is a graph illustrating changes of frequency components with time, with respect to the transmitting wave and the receiving wave. The receiving wave <NUM> is observed after a time Δt has passed from a time when an emission of the transmitting wave <NUM> was started (hereinafter, the time Δt may be referred to as a "delay"). Note that the frequency of the transmitting wave <NUM> increases at a constant rate with time. Thus, in a case in which the delay Δt is constant, a frequency difference Δf between the transmitting wave <NUM> and the receiving wave <NUM> is also constant. Therefore, if the frequency difference Δf is observed, the delay Δt can be obtained by using the frequency difference Δf. Further, if the delay Δt is obtained, a distance to an object can be calculated by using the delay Δt. Note that a period when a frequency of a signal (transmitting wave) is gradually caused to be increased is denoted as T, and a difference between a maximum frequency of the transmitting wave <NUM> and a minimum frequency of the transmitting wave <NUM> is F.

As there is a frequency difference between the transmitting wave <NUM> and the receiving wave <NUM>, a beat signal is generated when the receiving wave <NUM> is superimposed on the transmitting wave <NUM>. A beat signal corresponds to an envelope waveform of a superimposed signal, and if Δf is constant, a frequency of the envelope waveform is also constant. Also, it is known that a frequency of a beat signal is equal to Δf.

When Fourier transform (preferably fast Fourier transform) is applied to a beat signal, a frequency spectrum having a peak at a frequency of the beat signal is obtained. Accordingly, a frequency Δf can be detected by applying Fourier transform to a beat signal. <FIG> is a schematic diagram illustrating a frequency spectrum obtained by applying Fourier transform.

Next, a method of obtaining the distance R from Δf will be described. The distance R and Δt (the delay) satisfy a relationship expressed by the following formula (<NUM>). <MAT> where C represents velocity of light traversed in the air.

Next, as is apparent from <FIG>, Δf and Δt have a relationship of "Δt:Δf = T:F", which can be transformed into the following formula (<NUM>).

By substituting the formula (<NUM>) into the formula (<NUM>), the following formula (<NUM>) is obtained. The formula (<NUM>) can be transformed into a formula (<NUM>) below. <MAT> <MAT>.

Therefore, by substituting a frequency of a beat signal obtained by Fourier transform, for Δf in the formula (<NUM>), the distance R can be obtained.

The FMCW is a method of performing the FCM continuously, and a principle of the FMCW is the same as that in the FCM. <FIG> is a graph illustrating changes of frequencies with time, with respect to a transmitting wave <NUM> and a receiving wave <NUM> used in the FMCW, and illustrating a beat waveform <NUM> generated in the FMCW. In the FMCW, the frequency of the transmitting wave <NUM> or the receiving wave <NUM> changes relatively slowly, and the change occurs repeatedly. It is said that the FCM is superior in recognition capability of relative speed and recognition capability of multi-target.

Next, a method of detecting a direction of an object by using the FCM will be described with reference to <FIG> are diagrams illustrating a direction of an object. A positional relationship between the MMW transceiver <NUM> and an object <NUM> seen from above is illustrated in <FIG>. As a receiving wave (part of a transmitting wave being reflected by the object <NUM>) is regarded as a plane wave when the object <NUM> is apart from the MMW transceiver <NUM>, the FCM detects a direction where the object <NUM> is located by estimating an incoming direction of the receiving wave (plane wave). In <FIG>, let an advancing direction of a moving body in which the MMW transceiver <NUM> is installed be <NUM> degrees. Also, suppose a case in which the object <NUM> is located at θ degrees from the advancing direction. In this case, an incoming direction of the receiving wave is θ.

<FIG> is a diagram illustrating a method of estimating the incoming direction θ of the receiving wave. Θ is detected by using an array antenna. <FIG> illustrates a case in which N number of receiving antennas <NUM> are arranged in a line with each spaced at intervals of d. As described above, the receiving wave arriving at the receiving antennas <NUM> is regarded as a plane wave if the object <NUM> is apart from the receiving antennas <NUM>. Path differences of receiving waves received by the respective receiving antennas <NUM> can be expressed as multiples of (d × sin Θ) (in other words, when a path difference of two receiving waves received by two adjacent receiving antennas <NUM> is denoted as "<NUM>", the path difference <NUM> can be expressed as "<NUM> = d × sin Θ"). The path difference <NUM> can be calculated by using a delay r (a difference of time when adjacent two receiving antennas <NUM> receive the receiving waves). Specifically, <NUM> and r have a relationship of "r = <NUM>/C", where C represents velocity of light traversed in the air. Thus, if the delay r of the receiving waves received by the adjacent receiving antennas <NUM> is detected, Θ can be estimated by using the following formula (<NUM>).

In the following, effect of the distance measurement system <NUM> according to the present embodiment will be described with reference to experimental results.

A reference image used in a first experiment is illustrated in <FIG>. A picture (a) is a reference image, and (b) is an enlarged view of a central region of the picture (a). A moving body <NUM>, a person <NUM>, and a chart (such as a road sign indicating a driving direction) <NUM> appear in a vicinity of a center of the reference image. Actual distances to these objects (from the distance measurement system <NUM>) are as follows:.

<FIG> are overhead view maps generated by the first experiment. <FIG> is an overhead view map generated based on distance information measured by the distance measurement system <NUM> according to the present embodiment, and <FIG> is a comparative example, which is an overhead view map generated based on distance information obtained by block matching. In the present embodiment (<FIG>), because accuracy of distance measurement and resolution performance improve, the moving body <NUM> can be separated from the chart <NUM> (or person <NUM>). Also, a gate located <NUM> ahead, and a shed located <NUM> ahead are separated. However, the chart <NUM> and the person <NUM> positioned at the same distance cannot be separated, which is a further task.

Regarding the overhead view map, supplemental description will be made with reference to <FIG> is a diagram an example of a method of generating an overhead view map. <FIG> represents a range image. In a case in which a distance of pixel coordinates (x, y) is "z", a parallax d(x, y) of the pixel coordinates (x, y) is expressed as "d(x, y) = B×F/Z".

If coordinates of a center of the range image are (x<NUM>, y<NUM>), coordinates of an actual space (X, Y, Z) are expressed as the following expressions: <MAT> <MAT> <MAT>.

For each pixel in the range image, the distance calculation unit <NUM> performs calculations using the above expressions. As the overhead view map in <FIG> is a two-dimensional map, by calculating coordinates (X, Z) corresponding to each pixel in the range image, and by adding a value to a mesh space corresponding to an XZ-coordinate space, the overhead view map like <FIG> is obtained. Also, by calculating coordinates (X, Y, Z) corresponding to each pixel in the range image, a three-dimensional map is obtained.

A reference image used in a second experiment is illustrated in <FIG>. A picture (a) of <FIG> is a reference image, and (b) of <FIG> is an enlarged view of a central region of the picture (a). Moving bodies <NUM> and <NUM> appear in a vicinity of a center of the reference image. Actual distances to these objects (from the distance measurement system <NUM>) are as follows:.

<FIG> illustrates an overhead view map generated by the first experiment and an overhead view map according to a comparative example. A picture (b) of <FIG> is an overhead view map generated based on distance information measured by the distance measurement system <NUM> according to the present embodiment, and a picture (a) of <FIG> is the overhead view map according to the comparative example, which is generated based on distance information obtained by block matching. In the present embodiment, because accuracy of distance measurement and resolution performance improve, the moving bodies <NUM> and <NUM> can be clearly separated. Also, a gate located <NUM> ahead, and a shed located <NUM> ahead are separated.

<FIG> is a reference image, which is an image of a headlight captured at night. A picture (a) of <FIG> is an entire reference image and a region processed by the distance measurement system <NUM>, and (b) of <FIG> is an enlarged view of a headlight portion of a moving body <NUM>.

<FIG> illustrates an overhead view map generated by the first experiment and an overhead view map according to a comparative example. A picture (b) of <FIG> is an overhead view map generated based on distance information measured by the distance measurement system <NUM> according to the present embodiment, and a picture (a) of <FIG> is the overhead view map according to the comparative example, which is generated based on distance information obtained by block matching. In the overhead view map generated based on distance information obtained by the block matching, locations of objects are unclear. On the other hand, the distance measurement system <NUM> according to the present embodiment can separately detect three-dimensional objects (moving bodies <NUM> and <NUM>, a shed <NUM>, and a gate <NUM>) positioned at a remote location <NUM> meters or more distant.

A result of an experiment in which a distance of a chart was measured will be described. In this experiment, a distance of a chart <NUM> away and a distance of a chart <NUM> away were measured. <FIG> is a reference image of a chart. Although colored range images have been actually obtained in this experiment, in the present specification, instead of illustrating the range images, a summary of an evaluation result of accuracy of measured distance will be described in the following table (Table <NUM>). In this experiment, distances to a surface of the chart have been measured. As a result of the measurement, an average, a variance, and a standard deviation of the distances are illustrated in the following table.

Results of detecting distances to the chart <NUM> by the present embodiment, the SGM algorithm, and the block matching are illustrated in Table <NUM>. In both cases in which the chart <NUM> is <NUM> away, and in which the chart <NUM> is <NUM> away, it is found that the distance measurement system <NUM> according to the present embodiment attains improved accuracy with respect to the average, the variance, and the standard deviation.

<FIG> are diagrams for explaining effect for suppressing dilation caused by the SGM. <FIG> illustrates a reference image, <FIG> illustrates a range image obtained by the distance measurement system <NUM> according to the present embodiment, and <FIG> illustrates a range image obtained by the SGM algorithm. In <FIG>, because of dilation, a part of pixels between the posts of a chart, which should correspond to a road surface, are represented as if a distance to the road surface were equal to a distance to a surface of the chart. However, in <FIG> (the present embodiment), pixels between posts of a chart are representing a distance to the road surface beyond the chart.

In the experiment of <FIG>, a multi-layer laser radar distance measurement unit <NUM>, which scans laser light in multiple vertical layers, is used.

As described above, in the distance measurement system <NUM> according to the present embodiment, because integration of distance information measured by LIDAR is performed in Z-space before a stereocamera outputs a range image generated by the block matching or the like, a high-quality and high-resolution range image can be obtained.

For example, in conventional techniques such as that disclosed in <CIT>, distance information measured by LIDAR is integrated with a parallax image obtained by block matching on a parallax space. However, in the present embodiment, integration is performed in a distance space.

In a method of performing integration in a parallax space, despite distance resolution of LIDAR being excellent and accuracy of measured distance being secured, distance resolution with integration degrades especially in a remote location. Further, sub-pixel estimation is performed by using distance information of coarse distance resolution. Thus, accuracy of measured distance cannot be secured, and improving accuracy is not expected.

In the method of performing integration in a distance space, as in the present embodiment, integration of LIDAR with a cost curve is realized while securing high accuracy of distance measured by LIDAR; accordingly, a high-quality and high-resolution range image can be obtained.

Accordingly, in the present embodiment, even in a distance range in which a parallax is close to <NUM>, such as a remote location approximately <NUM> away, and in which measurement by a stereocamera is difficult, a range image having accurate distance information and having small variance of distance information with respect to an object surface can be generated. Also, when reconstructing three-dimensional space, a wide area of three-dimensional space can be reconstructed.

That is, as compared to a conventional stereocamera, accuracy of distance measurement improves, and variance of distance to an object surface becomes smaller. Also, as ability of separately detecting objects improves, ability of detecting an object at a remote location improves, and accuracy of distance measurement of an object at a remote location improves. Further, because of integration according to the present embodiment, even for a pixel having unreliable cost, such as due to a repetitive pattern or a low-texture region, a more accurate distance can be measured. Further, even at night, more accurate distance can be measured. Further, it is also effective for suppressing dilation caused by the SGM. Also, as compared to conventional LIDAR, the present embodiment improves spatial resolution.

In a distance measurement system <NUM> which will be described in a second embodiment, integration of stereo matching cost CST (p, Z) with LIDAR cost CLI (p, Z), considering ambiguity of a distance measured by LIDAR, is performed.

As described in the first embodiment, even within a range of a horizontal resolution and a vertical resolution of laser light, the LIDAR cost CLI(p, Z) becomes larger in a peripheral part. To solve the problem, in the first embodiment, cost is adjusted such that cost in a peripheral part becomes larger. The second embodiment solves the problem by considering ambiguity of distance information measured by the laser radar distance measurement unit <NUM>.

That is, the LIDAR cost CLI(p, Z) in the first embodiment is calculated based on only accuracy in an xy-plane (may also be referred to as spatial component of LIDAR cost (or spatial component cost)), but the second embodiment considers accuracy in the Z-direction (may also be referred to as distance component of LIDAR cost (or distance component cost)). In the present embodiment, LIDAR cost CLI(p, Z) is defined as a function described in the following formula (<NUM>).

In the following, a method of calculating the LIDAR cost CLI(p, Z) considering distance component cost CLD(p, Z) will be described.

An example of the spatial component cost may be defined as the above described formula (<NUM>). The distance component cost is defined, for example, as the following formula (<NUM>). <MAT> where.

<FIG> is a diagram schematically illustrating the distance component cost CLD(p, Z) of the LIDAR cost. As described, the distance component cost CLD(p, Z) takes the minimum at a point equal to a distance measured by LIDAR (may also be referred to as a "LIDAR distance"), and the distance component cost CLD(p, Z) becomes larger as a deviation from a LIDAR distance becomes larger. LIDAR distance may deviate from an actual value depending on an effect of a reflection rate of an object, or depending on a point where laser light is emitted. If LIDAR distance deviates from an actual value, an object surface forming a single surface may be detected as separate objects (this phenomenon may be referred to as "splitting"). By considering such ambiguity of LIDAR distance and including the ambiguity as distance component of the LIDAR cost, as described in the formula (<NUM>), occurrence of splitting can be reduced.

Note that a shape of a graph of the distance component cost CLD(p, Z) in <FIG> is merely an example. For example, the shape may be of rectangular shape, such that distance component cost in a predetermined range from the LIDAR distance γ takes a minimal value. Alternatively, the shape may be of reverse triangle shape, similar to <FIG>.

<FIG> is a diagram illustrating how the spatial component cost CLI(p, Z) and the distance component cost CLD(p, Z) of the LIDAR cost is integrated with the stereo matching cost CST(p, Z).

As described above, the distance measurement system <NUM> according to the second embodiment exhibits an effect for preventing an object surface from splitting, in addition to the effects described in the first embodiment.

In a third embodiment, a distance measurement system <NUM> capable of continuing a process even when failure occurs, and capable of displaying a message (information) indicating that failure has occurred, will be described. As described above with reference to <FIG> and <FIG>, the ECU <NUM> can detect failure in the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM>.

However, if communication failure, such that the ECU <NUM> cannot communicate with the stereogram processing unit <NUM>, occurs, the ECU <NUM> cannot acquire distance information from the laser radar distance measurement unit <NUM> via the stereogram processing unit <NUM>. In such a case, the ECU <NUM> may be configured to acquire an emitting direction and distance information from the laser radar distance measurement unit <NUM> directly.

Accordingly, if either the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM> is operating normally, the ECU <NUM> can continue driver-assistance by using a range image or distance information, though fusion is not performed.

<FIG> is an exemplary flowchart illustrating a process of the ECU <NUM> when failure has occurred in the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM>. A process illustrated in <FIG> is repeatedly executed while a moving body is running.

First, the ECU <NUM> determines if failure is detected in both the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> (S101). The failure may be detected by the laser radar distance measurement unit <NUM>, the stereogram processing unit <NUM>, or the ECU <NUM>.

If the determination at step S101 is positive (S101: YES), the ECU <NUM> displays information on the display device <NUM>, indicating that the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> have failed (S102). An example of the information displayed on the display device <NUM> will be illustrated in <FIG>.

If the determination at step S101 is negative (S101: NO), the ECU <NUM> determines if failure is detected in the laser radar distance measurement unit <NUM> (S103). The failure may be detected by the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM>.

If the determination at step S103 is positive (S103: YES), the ECU <NUM> performs driver-assistance by only a range image from the stereogram processing unit <NUM> (S104). As the ECU <NUM> has been performing driver-assistance by using a range image, a process of the driver-assistance is not changed.

Next, the ECU <NUM> displays information on the display device <NUM>, indicating that the laser radar distance measurement unit <NUM> has failed (S105). An example of the information displayed on the display device <NUM> will be illustrated in <FIG>.

Next, the ECU <NUM> determines if failure is detected in the stereogram processing unit <NUM> (S103). The failure may be detected by the stereogram processing unit <NUM> or the ECU <NUM>.

If the determination at step S106 is positive (S106: YES), the ECU <NUM> performs driver-assistance by only distance information from the laser radar distance measurement unit <NUM> (S107). That is, the ECU <NUM> starts driver-assistance based on a location of an object (emitting direction) and a distance, not based on a range image.

Next, the ECU <NUM> displays information on the display device <NUM>, indicating that the stereogram processing unit <NUM> has failed (S108). An example of the information displayed on the display device <NUM> will be illustrated in <FIG>.

Accordingly, when the laser radar distance measurement unit <NUM> fails, or when the stereogram processing unit <NUM> fails, the ECU <NUM> can continue driver-assistance.

<FIG> are the examples displayed on the display device <NUM> when failure occurs in the laser radar distance measurement unit <NUM> or the stereogram processing unit <NUM>.

The information illustrated in <FIG> is displayed on the display device <NUM> when both the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> have failed. In <FIG>, a message "Warning Radar and camera sensor have failed. Terminate Driver-assistance" is displayed. Based on the message, an occupant in a moving body can recognize that driver-assistance is no longer performed because the laser radar distance measurement unit <NUM> and the stereogram processing unit <NUM> are in failure.

The information illustrated in <FIG> is displayed on the display device <NUM> when the laser radar distance measurement unit <NUM> has failed. In <FIG>, a message "Warning Radar has failed. Continue Driver-assistance using camera sensor. " is displayed. Based on the message, an occupant in a moving body can recognize that the laser radar distance measurement unit <NUM> is in failure but that driver-assistance is continued.

The information illustrated in <FIG> is displayed on the display device <NUM> when the stereogram processing unit <NUM> has failed. In <FIG>, a message "Warning Camera has failed. Continue Driver-assistance using radar. " is displayed. Based on the message, an occupant in a moving body can recognize that the stereogram processing unit <NUM> is in failure but that driver-assistance is continued.

In a fourth embodiment, addition of LIDAR cost CLI (p, Z) to stereo matching cost CST (p, Z) with LIDAR cost CLI (p, Z) is performed partially. In the first embodiment, as described with reference to <FIG> and <FIG>, LIDAR cost CLI(p, Z) is added to the stereo matching cost CST(p, Z) of the pixel r<NUM> corresponding to the emitting direction of laser light and the stereo matching cost CST(p, Z) of all surrounding pixels of the pixel r<NUM>.

In the fourth embodiment, as illustrated in <FIG>, voting (addition) of LIDAR cost CLI(p, Z) is not necessarily performed to all of the stereo matching cost CST(p, Z) of the pixel ro corresponding to the emitting direction of laser light and the surrounding pixels of the pixel r<NUM>. The addition is performed in accordance with statuses of the LIDAR cost CLI(p, Z)and the stereo matching cost CST(p, Z).

<FIG> is a diagram schematically illustrating image data and locations where laser light is emitted (hereinafter, the location may be referred to as an "irradiated location"). Two irradiated locations <NUM> and <NUM> are illustrated in <FIG>. Laser light is spread on part of pixels in the image data. Stereo matching cost CST(p, Z) of the irradiated location <NUM> is almost flat, and it is difficult to identify an extreme value. Conversely, stereo matching cost CST(p, Z) of the irradiated location <NUM> is convex downward (that is, an extreme value is clearly present). Thus, it is difficult to find an extreme value of the stereo matching cost CST(p, Z) of the irradiated location <NUM>, but it is easy to find an extreme value of the stereo matching cost CST(p, Z) of the irradiated location <NUM>.

<FIG> and <FIG> are diagrams illustrating integration of a light reception level associated with distance information with stereo matching cost CST(p, Z). <FIG> illustrates an example of integration with respect to the irradiated location <NUM>, and <FIG> illustrates an example of integration with respect to the irradiated location <NUM>. In <FIG>, a minimum value of the light reception level associated with distance information is clear. Thus, as a result of integrating a light level with stereo matching cost CST(p, Z), a minimum value can be seen clearly.

In a case such as <FIG>, integration is effective when multi-pulse occurs in a light receiving level (when multiple pulse-like peaks appear on a graph of a light receiving level). In <FIG>, two local minimum values are present on a light receiving level curve. Such a light receiving level curve is obtained when laser light is reflected by multiple objects. As a result of integrating a light level with stereo matching cost CST(p, Z) in <FIG>, a minimum value can be seen clearly.

Thus, the stereogram processing unit <NUM> performs integration of a light reception level with stereo matching cost CST(p, Z) in accordance with the following rules. Note that the integration is performed, similar to the first embodiment, with respect to a pixel r<NUM> corresponding to an emitting direction of laser light and surrounding pixels of the pixel r<NUM>.

In other words, when stereo matching cost CST(p, Z) does not vary in accordance with change of a distance Z and multi-pulse occurs in a light receiving level, integration can be omitted because an effect of integration cannot be obtained sufficiently. Also, when a minimum value of stereo matching cost CST(p, Z) is clearly determined, and a minimum value of a light receiving level is clearly determined, integration can be omitted because a location of an object can already be determined without performing integration (however, in this case, integration may be performed for confirmation).

Whether or not stereo matching cost CST(p, Z) is flat can be determined, for example, by comparing a difference of a maximum value and a minimum value with a threshold. If the difference is sufficiently large, a minimum value is clearly determined. Whether or not multi-pulse occurs in a light receiving level can be determined by, for example, determining if the number of local minimum values is more than one.

In the above description regarding <FIG> and <FIG>, the expression of "integrating a light reception level of LIDAR with stereo matching cost" is used. This is equivalent to integrate LIDAR cost CLI(p, Z) with stereo matching cost CST(p, Z).

In a first example useful for understanding but not pertaining to the invention as claimed, a range image provision system including a server apparatus will be described. The server apparatus performs at least a part of the process performed in the above described distance measurement system <NUM>.

<FIG> is a schematic diagram of a range image provision system <NUM>. As illustrated in <FIG>, a distance measurement system <NUM> installed in a moving body <NUM> communicates with a server apparatus <NUM> via a network N (the server apparatus <NUM> may also be referred to as a "server <NUM>"). To the server <NUM>, the distance measurement system <NUM> transmits distance information, an emitting direction, a reference image, and a comparison image. When receiving the above mentioned information, the server <NUM> generates a range image by performing the process described in the first, second, third, or fourth embodiment, and returns the range image to the distance measurement system <NUM>.

<FIG> is an example of a functional block diagram of the range image provision system <NUM>. Functions of the laser radar distance measurement unit <NUM> are the same as described in the first to fourth embodiments. The laser radar distance measurement unit <NUM> transmits an emitting direction of laser light and distance information to a communication device <NUM>. A stereogram processing unit <NUM> in the first example does not need to have the distance calculation unit <NUM>, and the distortion adjusting unit <NUM> transmits a reference image and a comparison image to the communication device <NUM>. The communication device <NUM> transmits the reference image and the comparison image to the server <NUM>.

The server <NUM> includes a communication device <NUM> and a distance calculation unit <NUM>. The server <NUM> performs integration of stereo matching cost CST (p, Z) with LIDAR cost CLI (p, Z), to generated a range image (high-density / high-resolution 3D range image). The communication device <NUM> in the server <NUM> transmits the range image to the moving body <NUM>.

The moving body <NUM> transmits the range image and the reference image to the ECU <NUM>. Accordingly, the ECU <NUM> can perform driver-assistance, similar to the first to fourth embodiments.

As described above, as the moving body <NUM> generates a range image by communicating with the server <NUM>, cost of the distance measurement system <NUM> can be reduced.

The server <NUM> may also transmit the range image to another moving body (which is other than the moving body from which the server <NUM> receives information). For example, when a moving body <NUM> positioned at a head of a vehicle line in a traffic jam transmits, to the server <NUM>, distance information, an emitting direction, a reference image, and a comparison image, the server <NUM> transmits the generated range image to following moving bodies. Accordingly, the following moving bodies can recognize a state of the moving body <NUM> positioned at the head of the vehicle line.

A best mode for practicing the present invention has been described above using embodiments. However, the present invention is not limited to the above described embodiments. Various variations and replacements may be applied within the scope of the present invention which is defined in the appended claims.

Examples of a moving body in which the distance measurement system <NUM> is installed include a vehicle and an automobile, but the distance measurement system <NUM> is applicable to various types of moving bodies. For example, it is effective to apply the distance measurement system <NUM> to a moving body travelling autonomously in some cases, such as an aircraft, a drone, a ship, or a robot.

Laser light used in the present invention is not limited to a specific one. Any type of light having a wavelength appropriate for measuring distance may be used. Visible light, infrared radiation, or ultraviolet radiation (in a range not affecting a human body) may be used. Light may be regarded as an electromagnetic wave.

In the present embodiment, as an example of a distance measurement method having distance resolution, LIDAR is introduced. However, other methods such as methods using millimeter wave or ultrasound (sonar) may be used. A method of actively measuring distance, such as the above mentioned examples, is referred to as an active distance measurement.

In the present embodiment, a case for using a stereocamera having two camera units is described. However, a stereocamera used in the present invention may have more than two camera units. Also, the camera units may be arranged apart from each other in a horizontal direction, or may be arranged apart from each other in a vertical direction. Further, the camera units may capture light other than visible light, such as near infrared radiation or infrared radiation. The camera units may capture light via polarization filters.

Claim 1:
An image processing method of generating a range image, the method comprising:
detecting a distance to an object as a measured distance, by a laser radar distance measurement unit (<NUM>);
performing, by an image processing unit (<NUM>), matching of a stereogram by shifting a reference image, to calculate matching cost values of a plurality of pixels, wherein each of the matching cost values corresponds to a respective shift amount of a pixel between the reference image and a comparison image,
converting, by the image processing unit (<NUM>), the matching cost values each corresponding to the respective shift amounts into a set of matching cost values each corresponding to a corresponding distance, wherein interpolation is used to obtain matching cost values for a uniform density of corresponding distances;
performing, by the image processing unit (<NUM>), integration of the distance with the matching cost value of a pixel in the stereogram corresponding to an emitting direction of a laser of the laser radar distance measurement unit (<NUM>), among a plurality of pixels in the stereogram each including a corresponding matching cost value,
wherein the integration includes integrating a distance evaluation value related to the measured distance, with a matching cost value corresponding to the measured distance among the matching cost values of said pixel, wherein the distance evaluation value is expressed as a first function of a length between said pixel and a surrounding pixel of said pixel, and wherein, in integrating the distance evaluation value, a value of the first function is added to a matching cost value corresponding to the measured distance among matching cost values of the surrounding pixel; and
generating, after the integration, a range image for measuring a distance to an object, based on a result of the integration for a plurality of pixels.