Patent ID: 12243262

DESCRIPTION OF EMBODIMENTS

An apparatus that can estimate the distance to an object of complex shape appropriately with relatively small memory capacity will now be described with reference to the attached drawings. The apparatus for estimating distance first extracts a reference feature map from a reference image generated by a reference camera for taking a picture of an object from a predetermined reference position, and extracts a source feature map from each of one or more source images respectively generated by one or more source camera for taking a picture of the object from a position different from the reference position. The reference feature map represents features corresponding to respective pixels included in the reference image whereas the source feature map represents features of respective pixels included in the source image. The apparatus then projects the source feature map onto hypothetical planes to generate a cost volume in which coordinates on the hypothetical planes are associated with features. The hypothetical planes are hypothetically disposed by transforming the features in the reference feature map corresponding to respective pixels included in the reference image so that the features in the reference feature map correspond to respective pixels of an image corresponding to an image surface of the reference camera for the case that the image surface is moved in the direction of an optical axis of the reference camera. The apparatus then sets sampling points on a ray in the cost volume extending from the reference position in a direction corresponding to one of the pixels included in the reference image; interpolates, for each of the sampling points, a feature corresponding to the sampling point, using features associated with coordinates near the sampling point and on some of the hypothetical planes disposed near the sampling point in the cost volume; and inputs features corresponding to the interpolated sampling points into a classifier to calculate occupancy probabilities corresponding to the respective sampling points. The classifier is trained to output the occupancy probabilities each depending on a feature corresponding to coordinates on one of the hypothetical planes and indicating how likely the coordinates are inside the object. The apparatus then adds up products of the occupancy probabilities corresponding to the respective sampling points and the distances from the reference position to the corresponding sampling points to estimate the distance from the reference position to a surface of the object.

FIG.1schematically illustrates the configuration of a vehicle equipped with the apparatus for estimating distance.

The vehicle1includes cameras2for taking a picture of the surroundings, and an electronic control unit (ECU)3, which is an example of the apparatus for estimating distance. The cameras2are connected to the ECU3via an in-vehicle network conforming to a standard, such as a controller area network, so that they can communicate with each other.

The cameras2are an example of imaging units for generating images of the surroundings of the vehicle1. The cameras2each include a two-dimensional detector constructed from an array of optoelectronic transducers, such as CCD or C-MOS, having sensitivity to visible light and a focusing optical system that forms an image of a target region on the two-dimensional detector. The cameras2include a left camera2-1and a right camera2-2. For example, the left camera2-1is disposed in a front and upper left area in the interior of the vehicle and oriented forward whereas the right camera2-2is disposed in a front and upper right area in the interior of the vehicle and oriented forward. Since they are disposed at different positions in the vehicle1, the left camera2-1and the right camera2-2can take pictures of the same object from different viewpoints. The cameras2of the present embodiment include two cameras, i.e., the left camera2-1and the right camera2-2; however the cameras2may include three or more cameras disposed at different positions. The cameras2take pictures of the surroundings of the vehicle1through a windshield every predetermined capturing period (e.g., 1/30 to 1/10 seconds), and output images representing the surroundings.

The ECU3estimates the distance from a reference position to an object represented in images generated by the cameras2. Additionally, the ECU3predicts a future position of the object, based on the estimated distance from the reference position to the object, and controls a travel mechanism (not shown) of the vehicle1lest a future distance between the vehicle1and the object fall below a predetermined distance threshold.

FIG.2schematically illustrates the hardware of the ECU3. The ECU3includes a communication interface31, a memory32, and a processor33.

The communication interface31, which is an example of a communication unit, includes a communication interface circuit for connecting the ECU3to the in-vehicle network. The communication interface31provides received data for the processor33, and outputs data provided from the processor33to an external device.

The memory32, which is an example of a storage unit, includes volatile and nonvolatile semiconductor memories. The memory32contains various types of data used for processing by the processor33, e.g., the locations of the cameras2, the directions of optical axes of their focusing optical systems, and their focal lengths. The memory32also contains a set of parameters for defining a neural network that operates as a classifier for extracting a feature map from an image (e.g., the number of layers, layer configuration, kernels, and weighting factors). The memory32further contains a cost volume generated with feature maps as well as a set of parameters for defining a neural network that operates as a classifier for outputting occupancy probabilities corresponding to coordinates included in the cost volume, based on features corresponding to the coordinates. Additionally, the memory32contains various application programs, such as a distance estimation program to execute a distance estimation process.

The processor33, which is an example of a control unit, includes one or more processors and a peripheral circuit thereof. The processor33may further include another operating circuit, such as a logic-arithmetic unit, an arithmetic unit, or a graphics processing unit.

FIG.3is a functional block diagram of the processor33included in the ECU3.

As its functional blocks, the processor33of the ECU3includes an extraction unit331, a generation unit332, a setting unit333, an interpolation unit334, a calculation unit335, and an estimation unit336. These units included in the processor33are functional modules implemented by a computer program stored in the memory32and executed by the processor33. The computer program for achieving the functions of the units of the processor33may be provided in a form recorded on a computer-readable and portable medium, such as a semiconductor memory, a magnetic medium, or an optical medium. Alternatively, the units included in the processor33may be implemented in the ECU3as separate integrated circuits, microprocessors, or firmware.

The extraction unit331extracts a reference feature map from a reference image generated by a reference camera. The reference feature map represents features corresponding to respective pixels included in the reference image. The extraction unit331also extracts source feature maps from one or more source images respectively generated by one or more source cameras. Each source feature map represents features of respective pixels included in a corresponding source image.

FIG.4is a diagram for explaining extraction of feature maps.

The left camera2-1and the right camera2-2mounted on the vehicle1take a picture of an object OBJ, and respectively output a reference image PRand a source image PSthat represent the object OBJ. In the description of the present embodiment, the left camera2-1is assumed to be a reference camera for generating the reference image PR, and the right camera2-2is assumed to be a source camera for generating the source image PS; however, these cameras may be interchanged. In the case that the cameras2include three or more cameras, one of the cameras is a reference camera and the other cameras are first, second, and other source cameras. The location of the left camera2-1corresponds to the reference position. The right camera2-2is disposed at a position different from the reference position.

The extraction unit331inputs the reference image PRand the source image PSinto a classifier C1to extract a reference feature map FMRrepresenting features corresponding to respective pixels included in the reference image PRand a source feature map FMSrepresenting features corresponding to respective pixels included in the source image PS. The reference feature map FMRand the source feature map FMSare depth maps that have the same length and width as the reference image PRand the source image PSand whose pixels indicate estimated distances to an object represented in respective pixels of the reference image PRand the source image PS. The classifier C1may be, for example, a convolutional neural network (CNN) including convolution layers connected in series from the input toward the output, such as Multi-Scale Deep Network. A CNN that has been trained in accordance with a predetermined training technique, such as backpropagation, using images having pixels associated with depths as training data operates as the classifier C1to extract features of respective pixels from an image.

The reference feature map FMRand the source feature map FMSmay be segmentation maps in which the pixels of the reference image PRand the source image PSare classified into classes such as “road,” “human,” and “vehicle.” To output such feature maps, the classifier C1may be, for example, a CNN such as SegNet.

FIG.5is a diagram for explaining estimation of distance with the feature maps.

The generation unit332transforms the features in the reference feature map FMRcorresponding to respective pixels included in the reference image PRso that the features in the reference feature map correspond to respective pixels of an image corresponding to an image surface of the left camera2-1for the case that the image surface is moved in the direction of an optical axis of the left camera2-1, thereby hypothetically disposing hypothetical planes HP1to HP4between the location of the left camera2-1, which is the viewpoint of the reference image PR, and the object OBJ. The hypothetical planes HP1to HP4are orthogonal to the optical axis of the left camera2-1and disposed at different distances from the location of the left camera2-1. On each of the hypothetical planes HP1to HP4, the features in the reference feature map FMRcorresponding to respective pixels included in the reference image PRare arranged in an area reduced or enlarged according to the distance from the location of the left camera2-1.

The generation unit332projects the source feature map FMSonto each of the hypothetical planes to generate a cost volume. The cost volume includes coordinates on the hypothetical planes. These coordinates are associated with features depending on the difference between the features in the reference feature map FMRand those in the source feature map FMS. The present embodiment illustrates an example in which four hypothetical planes are disposed; however, the number of hypothetical planes is not limited thereto.

The generation unit332executes homography transformation of the source feature map FMSonto the positions of the hypothetical planes HP1to HP4to project the source feature map FMSonto the hypothetical planes HP1to HP4. The generation unit332generates a cost volume CV including features depending on the features in the source feature map FMSprojected onto the hypothetical planes HP1to HP4. In the case that there are multiple source images and corresponding source feature maps, each coordinate included in the cost volume CV is associated with features depending on the respective source feature maps.

The setting unit333sets sampling points p1, p2, and p3on a ray in the cost volume CV (sampling ray SR) extending from the location of the left camera2-1in a direction corresponding to a target pixel T, which is one of the pixels included in the reference image PR.

The setting unit333sets sampling points densely near one of the hypothetical planes such that the depth indicated by a feature associated with coordinates on the hypothetical plane near the sampling ray SR is closest to the depth at which the hypothetical plane is disposed.

The setting unit333may set sampling points on the sampling ray SR at regular intervals or at random intervals.

For each of the sampling points p1, p2, and p3, the interpolation unit334interpolates a feature corresponding to the sampling point, using features associated with coordinates near the sampling point and on some of the hypothetical planes disposed near the sampling point in the cost volume CV.

The following describes interpolation of a feature corresponding to the sampling point p1as an example; however, features of the other sampling points can be interpolated similarly. The interpolation unit334first identifies hypothetical planes close to the sampling point p1. The sampling point p1is set at a position whose lateral and vertical coordinates are i1and j1, respectively, on a plane at a depth k1parallel to the reference feature map FMR, and denoted by p1(i1, j1, k1). The interpolation unit334identifies a hypothetical plane having a maximum depth of not more than k1, and a hypothetical plane having a minimum depth of not less than k1.

On the identified hypothetical planes, the interpolation unit334identifies coordinates close to the sampling point p1(i1, j1, k1). The identified coordinates are, for example, ones whose lateral coordinate is a maximum of not more than ii and whose vertical coordinate is a maximum of not more than j1, or ones whose lateral coordinate is a maximum of not more than i1and whose vertical coordinate is a minimum of not less than j1.

The interpolation unit334interpolates the feature corresponding to the sampling point p1(i1, j1, k1), for example, by trilinear interpolation, using features associated with the coordinates on the hypothetical planes close to the sampling point p1(i1, j1, k1).

The calculation unit335inputs features corresponding to the interpolated sampling points into a classifier C2to calculate occupancy probabilities corresponding to the respective sampling points. An occupancy probability is a probability that coordinates included in the cost volume CV are inside the object OBJ. The classifier C2is trained to output the occupancy probabilities each depending on a feature corresponding to coordinates on one of the hypothetical planes disposed in the cost volume CV. Training of the classifier C2will be described below.

The classifier C2can be configured by a fully-connected neural network including fully-connected layers in which all input values are connected to all output values, such as a multi-layered perceptron.

The calculation unit335may weight occupancy probabilities corresponding to the respective sampling points so that an occupancy probability of a sampling point is weightier as the space between a pair of sampling points adjacent to the sampling point (bin size) is wider. This weighting enables the ECU3to appropriately process occupancy probabilities corresponding to unevenly set sampling points.

Using bin size biobtained by the following Expression (1), the calculation unit335weights the occupancy probability corresponding to one of the sampling points piset in ascending order of distance from the position of the left camera2-1, in accordance with the following Expression (2).

bi=di+1+di2-di+di-12(1)qi=ef⁡(pi)⁢bi∑jef⁡(pj)⁢bj(2)

In Expression (1), didenotes the distance from the location of the left camera2-1to the sampling point pi. In Expression (2), f(pi) denotes the occupancy probability outputted by the classifier C2and corresponding to the sampling point pi.

In the case that the sampling points are set at regular intervals, a softmax function may be applied to the occupancy probabilities outputted by the classifier C2to adjust them so that the sum of the occupancy probabilities of the sampling points equals 1.

The estimation unit336adds up products of the occupancy probabilities corresponding to the respective sampling points and the distances from the location of the left camera2-1to the corresponding sampling points, and outputs the added value as an estimated distance from the location of the left camera2-1to a surface of the object OBJ represented in the target pixel T.

The classifier C2is trained in accordance with a predetermined training technique, such as backpropagation, using a training reference image and a training source image as training data. The training reference image includes training pixels associated with true values of the depth of a represented object; the training source image is generated by taking a picture of the object from a viewpoint different from that of the training reference image.

A training source feature map extracted from the training source image is projected onto hypothetical planes disposed parallel to the training reference image to generate a training cost volume.

Training sampling points are set on a training sampling ray passing through the viewpoint of the training reference image and a training pixel included in the training reference image. In the present specification, the terms “training sampling point,” “training sampling ray,” and “training pixel” refer to a sampling point, a sampling ray, and a target pixel used for training the classifier C2, respectively.

The training sampling points are preferably set on the training sampling ray more densely at a location closer to a depth associated with the training pixel. For example, first, a predetermined number of initial training sampling points are set as the training sampling points at a point corresponding to the depth associated with the training pixel and uniformly between predetermined nearest and farthest planes. Additionally, in each bin defined by adjacent initial training sampling points, hierarchical training sampling points are set as the training sampling points; the number of these sampling points is a predetermined number multiplied by the possibility that the bin includes a surface of the object.

At the training stage, the classifier C2determines occupancy probabilities of the training sampling points set on the training sampling ray. At the training stage, the classifier C2further adds up products of the distances from the viewpoint of the training reference image to the respective training sampling points and the occupancy probabilities of the corresponding training sampling points to estimate the distance (depth) from the viewpoint of the training reference image to the training pixel.

The classifier C2is trained so as to reduce the difference between occupancy probabilities (from 0 to 1) depending on input of features at the training sampling points and occupancy states calculated from the depth associated with the training pixel (true value) and the coordinates of the training sampling points. When 0, an occupancy state indicates that a training sampling point is closer to the viewpoint than the surface of the object indicated by the depth (i.e., outside the object); when 1, it indicates that a training sampling point is farther from the viewpoint than the surface of the object (i.e., outside the object). The following Expression (3) is an error function preferably used for training the classifier C2.
=λdepthdepth+λoccocc(3)

In Expression (3), Ldepthdenotes the error between the estimated depth and the depth associated with the training pixel (true value); Loccdenotes the error between the estimated occupancy probabilities and the occupancy probabilities of the training sampling points (true values), as expressed by Expression (4) below. According to Expression (3), the classifier is trained so as to reduce the difference between the estimated depth and the depth associated with the training pixel (true value) as well as the difference between the estimated occupancy probabilities and the occupancy states at the training sampling points (true values). λdepthand λoccare hyperparameters for controlling the effect of training appropriately. For example, (λdepth, λocc) are set at (1e−3, 1), and then L is multiplied by 1e5, which leads to numerical stability.

ℒocc=1Ns⁢∑iCE⁡(s⁡(pi),γσ⁡(f⁡(pi)))(4)

In Expression (4), Nsis the number of training sampling points, and CE is a cross entropy function. s(pi) denotes the occupancy state at the training sampling point pi, and is obtained by dividing the absolute value of the difference between the depth of the training sampling point and the depth associated with the training pixel (true value) by a hyperparameter for controlling the range of the occupancy state and then subtracting the obtained value from 1 (the minimum is 0). s(pi) approaches 1 as the depth of the training sampling point piapproaches the true depth; it approaches 0 as the depth deviates from the true depth.

In Expression (4), f(pi) denotes the occupancy probability of the training sampling point pioutputted by the classifier C2; σ( ) is a sigmoid function; and γ is a trainable scalar value to adjust the scale difference between Locc(occupancy loss) and Ldepth(depth loss).

At estimating the depth of the training object at a training pixel in training of the classifier C2, the coordinates values of the training sampling points may be modified with values set for each training pixel. For example, at inferring the depth of a training pixel (x, y, z), the coordinates (xi, yi, zi) of a training sampling point set on a training sampling ray passing through the training pixel are modified with values (xa, ya, za) set for the training pixel (x, y, z) as follows: (xi+xa, yi+ya, zi+za). Modification of the coordinate values of the training sampling points to be inputted into the classifier C2results in prevention of overfitting of the classifier C2.

FIG.6is a flowchart of a distance estimation process. The ECU3executes the distance estimation process in response to input of a reference image PRand one or more source images PS.

The extraction unit331of the ECU3extracts a reference feature map FMRfrom the reference image PRand extracts a source feature map FMSfrom each of the one or more source images PS(step S1).

The generation unit332of the ECU3then projects the source feature map FMSonto hypothetically disposed hypothetical planes to generate a cost volume CV (step S2).

The setting unit333of the ECU3then sets sampling points on a ray extending from the location of the left camera2-1in a direction corresponding to one of pixels included in the reference image PR(step S3).

For each of the sampling points, the interpolation unit334of the ECU3then interpolates a feature corresponding to the sampling point, using features associated with coordinates near the sampling point and on some of the hypothetical planes disposed near the sampling point in the cost volume CV (step S4).

The calculation unit335of the ECU3then inputs features corresponding to the interpolated sampling points into the classifier C2to calculate occupancy probabilities corresponding to the respective sampling points (step S5).

The estimation unit336of the ECU3then adds up products of the occupancy probabilities corresponding to the respective sampling points and the distances from the location of the left camera2-1to the corresponding sampling points to estimate the distance from the location of the left camera2-1to a surface of an object (step S6), and terminates the distance estimation process.

By executing the distance estimation process in this way, the ECU3processes space including an object as a neural network without using voxels corresponding to the object. Thus the ECU3can estimate the distance to an object of complex shape appropriately with relatively small memory capacity.

The ECU3executes the distance estimation process at different times, and estimates the distances to a surface of an object at the respective times. The ECU3identifies the positions of the vehicle1at the respective times, based on positioning signals received at different times by a global navigation satellite system (GNSS) receiver (not shown) mounted on the vehicle1. The ECU3estimates the positions of the object at the respective times, based on the identified positions of the vehicle1, the estimated distances to the surface of the object, the mounted positions of the cameras2, the directions of their focusing optical systems, and their focal lengths. From the positions of the object at these times, the ECU3calculates the moving speed of the object during an interval therebetween, and predicts the position of the object in the future later than the last of these times. The ECU3generates a trajectory of the vehicle1lest a future distance between the vehicle1and the object fall below a predetermined distance threshold, and outputs a control signal to a travel mechanism (not shown) of the vehicle1. The travel mechanism includes, for example, an engine or a motor for accelerating the vehicle1, brakes for decelerating the vehicle1, and a steering mechanism for steering the vehicle1.

The travel control of the vehicle1described above is an example of usage of the distance to an object estimated by the distance estimation process of the present disclosure; the estimated distance can also be used for other processes. The apparatus for estimating distance need not be mounted on a vehicle, and may be used for estimating the distance to an object other than objects around a vehicle.

Note that those skilled in the art can apply various changes, substitutions, and modifications without departing from the spirit and scope of the present disclosure.