Patent ID: 12260577

DETAILED DESCRIPTION

Multiple embodiments are described in the present disclosure, but the description is exemplary rather than limiting, and there may be more embodiments and implementation solutions within the scope of the embodiments described in the present disclosure. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with or in place of any other feature or element of any other embodiment.

When describing representative embodiments, the specification may present methods and/or processes as a specific sequence of steps. However, to the extent that the method or process does not depend on the specific order of steps described in the present disclosure, the method or process should not be limited to the specific order of steps described. As understood by those of ordinary skills in the art, other orders of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be interpreted as limitation to the claims. In addition, the claims for the method and/or process should not be limited to the steps performed in the written order, and those of skilled in the art may readily understand that these orders may vary and still remain within the essence and scope of the embodiments of the present disclosure.

Unless otherwise defined, technical terms or scientific terms used in the embodiments shall have common meanings as construed by those of ordinary skills in the art to which the present disclosure pertains. The terms “first”, “second” and the like used in the embodiments of the present disclosure do not represent any order, quantity, or importance, but are merely used to distinguish different components. The terms “include”, “contain” or the like mean that elements or articles appearing before such terms cover elements or articles listed after the terms and their equivalents without excluding other elements or articles. The terms “connect”, “link” or the like are not limited to physical or mechanical connection, but may include electrical connections, whether direct or indirect.

Depth estimation is used to obtain distance information, by relating each pixel in the image to a camera device. An image including distance information is a depth image.

The pixel coordinates of the same object in two images are different, the pixel coordinates of the object closer to the camera are different, and the pixel coordinates of the same object farther away from the camera are smaller. Parallax is the difference in pixel coordinates of points in the same world coordinate system in different images. The parallax between different images can be converted into the distance from the object to the camera according to the camera parameters, that is, the distance is the depth.

Taking an image in a stereo image (in a pair of images which includes left image and right image) as a reference image (e.g., the left image), a size of the left image is the size of the reference image, and the image whose element value is the disparity value is a disparity map. Disparity estimation is a process of obtaining a disparity value of corresponding pixels between the left image and the right image, that is, the stereo matching process.

An autoencoder is an Artificial Neural Network (ANN) used in semi-supervised learning and unsupervised learning, the autoencoder takes input information as a learning target, and performs representation learning on the input information. The autoencoder includes an encoder and a decoder. According to a learning paradigm, autoencoders can be divided into contractive autoencoders, regularized autoencoders, and variational autoencoders (VAE). Depending on the type of construction, the autoencoder can be a neural network with a feedforward structure or a recursive structure.

The depth estimation method provided in the embodiment is applied to a scenario of automatic driving. It is understood that the depth estimation method provided in the embodiment is not limited to be applied in such scenario.

Referring toFIG.1, a vehicle1includes a depth estimation system20. The depth estimation system20is arranged in an interior compartment behind a windshield10of the vehicle100. The depth estimation system20includes a camera device201, a distance acquisition device202, and a processor203. The processor203is coupled with the camera device201and the distance acquisition device202.

It can be understood that the camera device201, the distance acquisition device202, and the processor203can be arranged in other positions on the vehicle100, and the camera device201can capture images of the view in front of the vehicle100, the distance acquisition device202can detect the distance between the object in front of the vehicle100and the vehicle100. For example, the camera device201and the distance acquisition device202may be located in the metal grille or front bumper of the vehicle100. Further, althoughFIG.1shows only one distance acquisition device202, the vehicle100may have a number of distance acquisition devices202in different directions (e. g., a side, a front, a rear of the vehicle100, etc.). The distance acquisition devices202may also be arranged at positions such as windshields, door panels, bumpers, or metal grilles.

In one embodiment, the camera device201on the vehicle100can capture images of scenes in front of and both sides of the vehicle100. As illustrated inFIG.1, in a horizontal coverage area110(shown by dashed lines) that can be detected by the camera device201, there are two objects, a vehicle120and a vehicle130. The camera device201can capture images of the vehicle120and the vehicle130in front of the vehicle100.

In some embodiments, the camera device201may be a binocular camera or a monocular camera.

In some embodiments, the camera device201may be implemented as an automobile data recorder. The automobile data recorder is used to record information such as images and sounds during travelling of the vehicle100. After the vehicle100is installed with the automobile data recorder, the automobile data recorder can record the images and sounds of the entire travelling process of the vehicle100, thereby providing effective evidence for traffic accidents. For example, in addition to the above functions, the functions provided by the automobile data recorder may also include functions such as global positioning system (GPS), driving track capture, remote monitoring, electronic dog, navigation, etc., which is not specifically limited.

The distance acquisition device202may be used to detect objects in front of and on both sides of the vehicle100, and obtain the distance between an object and the distance acquisition device202. As shown inFIG.1, the distance acquisition device202on the vehicle100may obtain the distance between the vehicle120and the distance acquisition device202, and the distance between the vehicle130and the distance acquisition device202. The distance acquisition device202may be an infrared sensor, a Lidar, a Radar, or the like.

For example, the distance acquisition device202is a radar, the radar utilizes radio frequency (RF) waves to determine the distance, direction, speed and/or height of objects in front of the vehicle. In detail, a radar includes a transmitter and a receiver, the transmitter transmits radar waves (i.e., RF signals), and the RF waves will be reflected when encountering objects on the propagation path. RF waves reflected by the object return a small fraction of energy to the receiver. As illustrated inFIG.1, the radar is configured to transmit the RF signals through the windshield in the horizontal coverage area140and to receive RF signals reflected by any object within the horizontal coverage area140, so as to receive a 3D point cloud image of any object.

In one embodiment, the horizontal coverage area110and the horizontal coverage area140may completely or partially overlap.

In some embodiments, the camera device201may capture images of the scene within the horizontal coverage area110in every time period. Likewise, the radar may capture the three-dimensional point cloud images of the scene within the horizontal coverage area140in every time period. The time period of the camera device201and the radar for capturing the images may be the same or different. The images and three-dimensional point cloud images captured by each camera device201may be marked with a timestamp. When the time period of the camera device201and the radar for capturing the images are different, the timestamps can be used to simultaneously or nearly simultaneously select the captured image and the 3D point cloud image for further processing (e.g., image fusion).

The three-dimensional point cloud, also known as a laser point cloud (PCD) or point cloud, can be a series of point sets for reflecting spatial distribution and surface features of target obtained by obtaining a three-dimensional spatial coordinates of each sampling point on the surface of the object under the same spatial reference system, using the laser. Compared with the image, the three-dimensional point cloud includes more three-dimensional space information, that is, the distance information between the object and the distance acquisition device202is included.

For example, as illustrated inFIG.1, at time TO, the camera device201may capture the images of the vehicle120and the vehicle130. At the same time (time TO), the distance acquisition device202can also obtain the three-dimensional point cloud image within the horizontal coverage area140, that is, the distance information between the vehicle120and the distance acquisition device202, and the distance information between the vehicle130and the distance acquisition device202are obtained at the time TO.

In one embodiment, the processor203may recognize depth of objects in the captured scene based on the images of the scene captured by the camera device201at the same time and the distance information of the same scene captured by the distance acquisition device202. The objects may be other vehicles, pedestrians, road signs, or obstacles.

It can be understood that the structure illustrated in the embodiment does not constitute a specific limitation on the depth estimation system. In other embodiments, the depth estimation system may include more or fewer components than shown, or some components may be combined, or some components may be split, or may have a different arrangement of components.

FIG.2illustrates a flowchart of an embodiment of a method for estimating depth, the method for estimating depth can be applied in the depth estimating system20. The method is provided by way of example, as there are a variety of ways to carry out the method. Each block shown inFIG.2represents one or more processes, methods, or subroutines carried out in the example method. Furthermore, the illustrated order of blocks is by example only and the order of the blocks can be changed. Additional blocks may be added or fewer blocks may be utilized, without departing from this disclosure. The example method can begin at block11.

At block11, obtaining a first image.

In one embodiment, the depth estimation system20may obtain the first image captured by the camera device. For example, the camera device can be a monocular camera, the monocular camera can record a video, and the depth estimation system can cut out an image frame from the video as the first image. The monocular camera can also capture an image, and the captured image is taken as the first image.

At block12, inputting the first image into a pre-trained depth estimation model, and obtaining a first depth image.

In some embodiments, the depth estimation model may include an autoencoder (AE) and an image conversion module. After the depth estimation system inputs the first image into the depth estimation model, the autoencoder processes the first image and outputs a disparity map corresponding to the first image. The image conversion module then converts the disparity map to the first depth image.

In other embodiments, the depth estimation model may omit the image conversion module. The depth estimation model processes the first image, and outputs a disparity map corresponding to the first image. The depth estimation system then converts the disparity map to the first depth image.

The training method of the depth estimation model is described in detail below.

FIG.3illustrates a flowchart of an embodiment of a method for training a depth estimating model. The method is provided by way of example, as there are a variety of ways to carry out the method. Each block shown inFIG.3represents one or more processes, methods, or subroutines carried out in the example method. Furthermore, the illustrated order of blocks is by example only and the order of the blocks can be changed. Additional blocks may be added or fewer blocks may be utilized, without departing from this disclosure. The example method can begin at block31.

At block31, obtaining a first image pair from a training data set.

In one embodiment, the first image pair includes a first left image and a first right image.

It can be understood that the image pair includes two images of the same scene captured by the camera device at the same time, which includes the left image and the right image. The sizes and the number of pixels of the left and right images are the same.

In one embodiment, the training data set may be a data set of images captured by a binocular camera when the vehicle is travelling. The images captured by the binocular camera include image pairs of the same scene captured by two cameras at the same time.

At block32, inputting the first left image into the depth estimation model to be trained, and obtaining a disparity map.

It can be understood that the depth estimation model to be trained can be an initialization model, the parameters of the initialization model can be preset according to actual requirement.

At block33, adding the first left image and the disparity map, and obtaining a second right image.

In one embodiment, the second right image is the right image predicted by the depth estimation model. The sizes and the number of pixels of the first and second right images are the same.

At block34, calculating a mean square error and cosine similarity of pixel values of all corresponding pixels in the first right image and the second right image.

The corresponding pixels are two pixels in the two images that have a corresponding position relationship. For example, the first right image includes a first pixel, the second right image includes a second pixel corresponding to the first pixel, and the position of the first pixel in the first right image is the same as that of the second pixel in the second right image.

It can be understood that the pixels in the image have pixel values of three color channels of red (R), green (G), and blue (B).

In one embodiment, the mean square error of pixel values of all corresponding pixels in the first right image and the second right image is calculated according to the following formula (1):
MSE=1/nΣi=1n(⅓Σk=13(yik−y′ik))2(formula (1)).

In formula (1), n is the number of pixels in the first right image or the second right image, yikis the pixel value of the k-th color channel of the i-th pixel in the first right image, and y′ikis the pixel value of the k-th color channel of the i-th pixel in the second right image.

In one embodiment, the mean square error can measure the difference of pixel values between the two corresponding pixels, and by minimizing the mean square error, the difference of pixel values between the two corresponding pixels can be minimized. The smaller the mean square error is, the higher the prediction accuracy of the depth estimation model is. When the mean square error is 0, it indicates that the pixel values of the two corresponding pixels are the same, that is, the predicted value of the depth estimation model is the same as the real value.

In one embodiment, the cosine similarity cos(A) of pixel values of all corresponding pixels in the first right image and the second right image is calculated according to the following formula (2):
cos(θ)=1−Σi=1n=Σk=13yik·y′ik/√{square root over (Σk=13yik2)}·√{square root over (Σk=13y′ik2)}  (formula (2)).

In one embodiment, the cosine similarity can measure the color difference between two corresponding pixels in the color space. The smaller the angle θ between the RGB values of the two corresponding pixels is, the smaller the cosine similarity is, which indicates that the colors of the two pixels are more similar. When the angle θ between the RGB values of two corresponding pixels is 0, the cosine similarity cos(θ)=0, which indicates that the two pixels have a same color.

At block35, calculating mean values of the mean square error and the cosine similarity of the pixel values of all corresponding pixels in the first right image and the second right image, and obtaining a first mean value of the mean square error and a second mean value of the cosine similarity.

In one embodiment, each pixel in the first right image and the second right image is polled in turn, and the mean square error of the pixel values of each corresponding pixel in the first right image and the second right image can be sequentially calculated by formula (1) until a number of mean square errors of the pixel values of all corresponding pixels is obtained. The first mean value can be obtained by calculating the mean value of the number of mean square errors.

Similarly, each pixel in the first right image or the second right image is polled in turn, and the cosine similarity of the pixel values of each corresponding pixel in the first right image and the second right image can be sequentially calculated by formula (2), until a number of cosine similarities of the pixel values of all corresponding pixels is obtained. The second mean value can be obtained by calculating the mean value of the number of cosine similarities.

At block36, adding the first mean value and the second mean value, and obtaining a loss value of the depth estimation model.

In one embodiment, when the loss value of the depth estimation model is 0, the depth estimation model converges.

At block37, updating parameters of the depth estimation model by using a back-propagation algorithm according to the loss value.

In one embodiment, the parameters of the depth estimation model are updated by using the back-propagation algorithm, so that the loss between the actual value and the predicted value can be reduced.

At block38, cyclically executing blocks31to37, and iteratively training the depth estimation model until training of the first image pairs in the training data set is completed or the depth estimation model converges.

In some embodiments, the training of the depth estimation model is completed when training of the first image pairs in the training data set is completed. At this time, the parameters of the depth estimation model with the smallest loss value are selected as the final model parameters.

In other embodiments, during the training process of the depth estimation model, when the depth estimation model converges, the training is completed. At this time, the parameters of the converged depth estimation model are used as the final model parameters.

It can be understood that the calculation of loss value of the depth estimation model combines the mean square error and cosine similarity, which can not only improve the prediction accuracy of the depth estimation model, but also improve the color-sensitivity of the depth estimation model, allowing color differences between individual pixels even in low-texture areas to be distinguished. Obtaining depth images by using the depth estimation model can improve the accuracy of depth information.

FIG.4illustrates the electronic device40in one embodiment. The electronic device40includes, but is not limited to, a processor42, a storage device41, and a computer program.FIG.4illustrates only one example of the electronic device40. Other examples can include more or fewer components than as illustrated or can have a different configuration of the various components in other embodiments.

In one embodiment, the processor42and the processor203can be the same. The electronic device40can further include a camera device and a distance acquisition device.

The processor42can be a central processing unit (CPU), a microprocessor, or other data processor chip that performs functions in the electronic device40.

In one embodiment, the storage device41can include various types of non-transitory computer-readable storage mediums. For example, the storage device41can be an internal storage system, such as a flash memory, a random access memory (RAM) for the temporary storage of information, and/or a read-only memory (ROM) for permanent storage of information. The storage device41can also be an external storage system, such as a hard disk, a storage card, or a data storage medium.

The storage device41stores instructions, and the processor42executes the computer program stored in the storage device41for implementing the method for training a depth estimating model and the method for estimating depth provided in the embodiments of the present disclosure. The computer program includes a training program, an estimating program, and instructions.

Upon execution of the instructions stores in the storage device41, the processor42is configured to:obtain a first image; andinput the first image into a pre-trained depth estimation model, and obtain a first depth image.

Upon execution of the instructions stores in the storage device41, the processor10is further configured to:obtain a first image pair from a training data set;input the first left image into the depth estimation model to be trained, and obtain a disparity map;add the first left image and the disparity map and obtain a second right image;calculate a mean square error and cosine similarity of pixel values of all corresponding pixels in the first right image and the second right image;calculate mean values of the mean square error and the cosine similarity of the pixel values of all corresponding pixels in the first right image and the second right image, and obtain a first mean value of the mean square error and a second mean value of the cosine similarity;add the first mean value and the second mean value, and obtain a loss value of the depth estimation model;update each parameter of the depth estimation model by using a back-propagation algorithm according to the loss value; anditeratively train the depth estimation model until training of the first image pairs in the training data set is completed or the depth estimation model converges.

It is believed that the present embodiments and their advantages will be understood from the foregoing description, and it will be apparent that various changes may be made thereto without departing from the spirit and scope of the disclosure or sacrificing all of its material advantages, the examples hereinbefore described merely being embodiments of the present disclosure.