Virtually boosted training

A method for training a machine learning model includes receiving real data comprising a real element in a real environment. The training also includes annotating the real element with a first annotation based on predicted attributes of the real element. The first annotation having a first format. The training further includes converting the first format of the first annotation to a second format corresponding to a ground truth annotation of the real element. The training still further includes adjusting parameters of the machine learning model to minimize a difference between values of the ground truth annotation of the real element and the converted first annotation.

BACKGROUND

Field

Certain aspects of the present disclosure generally relate to machine learning and, more particularly, to improving systems and methods for object detection using an artificial neural network.

Background

A machine learning model (e.g., artificial neural network (ANN)) may be trained to detect objects in frames generated from information captured by one or more sensors, such as a light detection and ranging (LIDAR) sensor or a red-green-blue (RGB) camera. The sensors may be coupled to, or in communication with, a device, such as a robotic device, or a vehicle, such as an autonomous vehicle. The detected objects may be identified, localized, and/or tracked. Object detection may be used in various applications, such as route planning and collision avoidance.

For object detection, the machine learning model may be trained using simulated data, such as virtual images of virtual elements in a virtual environment. During training, the machine learning model localizes an object with a three-dimensional (3D) bounding box. A prediction error may be calculated from a loss between the predicted 3D bounding box and a ground truth 3D bounding box. The machine learning model may be updated based on the prediction error.

To improve training, it is desirable to train the artificial neural networks with both simulated data and real world data. The real world data may be annotated with two-dimensional (2D) bounding boxes provided by a human annotator. Still, because the artificial neural network predicts object locations with 3D bounding boxes, it is difficult to determine a prediction error for predictions based on real world images.

Aspects of the present disclosure are directed to improving the training of a machine learning model for object detection.

SUMMARY

In one aspect of the present disclosure, a method for training a machine learning model is disclosed. The method includes receiving real data comprising a real element in a real environment. The method also includes annotating the real element with a first annotation based on predicted attributes of the real element. The first annotation is in a first format. The method further includes converting the first annotation from the first format to a second format corresponding to a ground truth annotation of the real element. The method still further includes adjusting parameters of the machine learning model to minimize a difference between values of the ground truth annotation of the real element and the converted first annotation.

In another aspect of the present disclosure, a non-transitory computer-readable medium with non-transitory program code recorded thereon is disclosed. The program code is for training a machine learning model. The program code is executed by a processor and includes program code to receive real data comprising a real element in a real environment. The program code also includes program code to annotate the real element with a first annotation based on predicted attributes of the real element. The first annotation is in a first format. The program code further includes program code to convert the first annotation from the first format to a second format corresponding to a ground truth annotation of the real element. The program code still further includes program code to adjust parameters of the machine learning model to minimize a difference between values of the ground truth annotation of the real element and the converted first annotation.

Another aspect of the present disclosure is directed to an apparatus for training a machine learning model. The apparatus having a memory and one or more processors coupled to the memory. The processor(s) is configured to receive real data comprising a real element in a real environment. The processor(s) is also configured to annotate the real element with a first annotation based on predicted attributes of the real element. The first annotation is in a first format. The processor(s) is further configured to convert the first annotation from the first format to a second format corresponding to a ground truth annotation of the real element. The processor(s) still further configured to adjust parameters of the machine learning model to minimize a difference between values of the ground truth annotation of the real element and the converted first annotation.

DETAILED DESCRIPTION

Real world objects, such as cars, trees, and people, have nine degrees of freedom, three degrees for position (e.g., x, y, z), three degrees for volume (e.g., width, length, and height), and three degrees in rotation (e.g., pitch, yaw, roll angles). Conventional object localization systems localize objects in a two-dimensional (2D) space. These conventional systems are limited to providing a 2D translation (e.g., height and width) and an object's scale.

To improve applications that use object localization, such as autonomous driving, robotics, and augmented reality, it is desirable to determine an object's volume, rotation, and relationship to other objects in a frame. The three-dimensional (3D) translation (e.g., location of the object within the frame), the 3D volume (e.g., width, length, and height) and 3D rotation (e.g., (x, y, z) coordinates) of the object may be referred to as a nine-dimensional (9D) pose of an object. Predicting the 9D pose of an object improves real world interactions with the object. For example, accurate predictions of an object's 9D pose improves route planning and collision avoidance applications. Predicting the 9D pose of an object may be referred to as 9D object localization. Aspects of the present disclosure are not limited to predicting the 9D pose of an object. The object localization system may determine additional attributes of the object.

For object localization, such as 9D object pose localization, a machine learning model may be trained to localize an object in a frame generated from information provided by one or more sensors coupled to, or in communication with, a device, such as an autonomous vehicle. At training time, the machine learning model is provided with a set of training data T={T1, T2, T3, . . . , Tn}. Each training instance Timay be simulated data or real world data. The simulated data may include photo-realistic images of objects (e.g., virtual images of objects). The real world data include real world images, such as RGB images, of objects. In the present disclosure, the machine learning model may be referred to as a model. The real world data may be referred to as real data. Finally, the simulated data may be referred to as virtual data.

During training, the machine learning model receives the simulated data, detects a simulated object, and predicts the simulated object's 9D pose (e.g., translation, volume, and rotation). Each object's predicted 9D pose may be identified with a 3D bounding box. A prediction error may be calculated from a loss between the predicted 3D bounding box and a 3D ground truth bounding box. The 3D ground truth bounding box refers to a known 3D bounding box that is generated as part of the simulation. The prediction error (e.g., loss) may be backpropagated to update parameters of the model. That is, parameters of the model are adjusted to minimize the prediction error.

To improve an accuracy of a model, it is desirable to use real world data, such as real world images of objects, during training. In conventional systems, objects in real world images may be annotated with 3D bounding boxes by a human annotator to generate 3D ground truth bounding boxes. Given a limited time to annotate images, the 3D ground truth bounding boxes annotated (e.g., labeled) by the human annotator may be inaccurate. Inaccurate 3D ground truth bounding boxes may lead to training errors, which may lead to errors in the final model. Furthermore, due to the intricacies of accurately determining an object's volume and rotation, an amount of training time is increased when 3D ground truth bounding boxes are provided by the human annotator.

To improve the accuracy and speed of training, objects in real world images may be annotated with 2D bounding boxes by the human annotator to generate 2D ground truth bounding boxes. Still, conventional 3D object detection models are not trained using 2D ground truth bounding boxes because conventional 3D object detection models cannot determine a prediction error between a predicted 3D bounding box and a 2D ground truth bounding box. That is, a 3D prediction error cannot be calculated when the real world image does not include a 3D ground truth bounding box. Additionally, a 2D prediction error cannot be calculated when the predicted 3D bounding box cannot be compared to the 2D ground truth bounding box.

As discussed herein, 3D ground truth bounding boxes are used to train a model for 3D object detection. The 3D ground truth bounding boxes may be generated by annotating (e.g., labeling) objects in each frame of a sequence of frame. Due to the vast amount of data used for training, the process of annotating objects in each each frame may be time intensive. Furthermore, the use of human annotators is expensive and prone to errors.

To reduce the time, costs, and errors associated with using annotated real world data to train the model, simulated data may be used as an alternative to the real world data. In the simulated data, the data and labels are machine generated. However, when the model is trained with only simulated data, the resulting model may be an overfitted model. An overfitted model generally refers to a model that has been trained too well, such that the performance of the model is reduced when the model is presented with new data, such as real world data. The models trained with only simulated data may not be suitable for real world scenarios.

There is a need to train a model with both simulated data and real world data. For example, the simulated data may be used to train the model to determine the 3D position, size, and orientation of an object. The real world data may be used to train the model to recognize patterns in real world images. Additionally, the real world data may prevent overfitting based on the simulated data. Performance of the trained model may be improved when the model is trained with the simulated data and the real world data.

According to aspects of the present disclosure, the model is trained in an iterative manner using simulated (e.g., synthetic) data associated with machine generated 3D bounding boxes and real world data associated with human annotated 2D bounding boxes. In one configuration, during the simulated data iteration, a 3D prediction error is used to update the model. Additionally, during the real world data iteration, the model generates a 3D response to a 2D image. For example, the 3D response may be a 3D bounding box. The 3D response is transformed to a 2D response that is compatible with the provided 2D annotation. For the real world data, the model is updated with a 2D prediction error.

To train the model with both the simulated data and real world data, aspects of the present disclosure are directed to using a translator function. The translator function may bridge a domain gap between the model's predictions and the annotated real world data. In one configuration, the translator function transforms parameters of a predicted 3D bounding box to a 2D bounding box that is comparable with a 2D ground truth bounding box.

FIG. 1illustrates an example of training a model using simulated data according to aspects of the present disclosure. As shown inFIG. 1, during training, at block100, the model may receive a simulated data frame108that includes a simulated image of a road102and a car104. Simulated data refers to virtual data generated by a device, such as a computer. The simulated data may be photo-realistic.

At block105, based on initial training parameters (e.g., weights and biases), the model detects one or more objects in the simulated data frame108. The model may be trained to detect one or more specific types of objects or areas of interest, such as cars, pedestrians, buildings, and bikes. Alternatively, the model may be trained to detect all objects (e.g., 3D objects). Each detected object may be localized with a 3D bounding box112. For example, the model may detect the car104in the simulated data frame108. In this example, the translation, volume, and rotation of the 3D bounding box112are based on a predicted (e.g., estimated) 9D pose of a corresponding object (e.g., car104).

At block110, the 3D bounding box112is compared with a 3D ground truth bounding box114. In one configuration, the 3D ground truth bounding box114is generated as part of the simulation data. Because the car104is a simulated object, the simulation data includes data corresponding to the car's1043D pose. Thus, as shown in block110, the 3D ground truth bounding box114has a better fit around the car104in comparison to a fit of the 3D bounding box112.

In the present example, the model may have overestimated the car's1043D pose (as illustrated in the 3D bounding box112). A difference between coordinates of the 3D bounding box112and the 3D ground truth bounding box114may be used to determine a prediction error. As an example, one of the differences may be a difference116between two similar coordinates, such as the upper right (x, y, z) coordinates, of the 3D bounding box112and the 3D ground truth bounding box114. Parameters of the model may be updated to reduce subsequent prediction errors.

The difference116between the two coordinates shown inFIG. 1is used as an example. Aspects of the present disclosure are not limited to only calculating a difference between two points of the 3D bounding box112and the 3D ground truth bounding box114. Furthermore, aspects of the present disclosure are not limited to detecting one object in a frame. Multiple objects may be detected in each frame.

To improve training, the model should also be trained on real world images of objects, such that the model is trained on both real world images of objects and simulated images of objects. In conventional systems, a human annotates a real world image of an object with a 3D bounding box. That is, a human draws the 3D bounding box around the real world image of the object to generate the 3D ground truth bounding box. The 3D ground truth bounding box is intended for comparison with a predicted 3D bounding box to determine a prediction error.

As previously discussed, human annotated 3D ground truth bounding boxes are prone to error as a human may not be as precise as a machine (e.g., computer) that generates 3D ground truth bounding boxes based on 3D pose information provided in simulated images of objects. Errors in the human annotated 3D ground truth bounding boxes may be propagated through the model, leading to inaccuracies in the final trained model. Furthermore, a process for a human to annotate real world images with 3D ground truth bounding boxes is time consuming.

In some cases, a human may annotate real world images of objects with 2D ground truth bounding boxes. That is, a human may draw the 2D ground truth bounding box around an object. The human annotated 2D ground truth bounding box may be more accurate in comparison to the human annotated 3D ground truth bounding box. Thus, the accuracy of the model may be improved by using human annotated 2D ground truth bounding boxes. Furthermore, a time period for drawing the 2D ground truth bounding box around the object is less than a time period for drawing the 3D ground truth bounding box around the object. Therefore, the training time may be decreased by using human annotated 2D ground truth bounding boxes

Still, conventional models do not use 2D ground truth bounding boxes because a predicted 3D bounding box cannot be compared to a 2D ground truth bounding box to calculate a prediction error. For example, a 3D bounding box includes twenty-four coordinates (e.g., (x, y, z) coordinates for each corner). A 2D bounding box includes eight coordinates (e.g., (x, y) coordinates for each corner). As such, the model cannot determine a prediction error due to the different number of coordinates for the predicted 3D bounding box and the 2D ground truth bounding box.

FIG. 2illustrates an example of detecting an object in a real world image. As shown inFIG. 2, during training, at block200, the model receives a real world data frame208that includes a real world image of a road202and a car204. Real world data refers to an image captured by an image capturing device, such as an RGB camera, in a real environment.

At block205, based on initial parameters, the model detects one or more objects in the real world data frame208. Each detected object may be localized with a 3D bounding box212based on the predicted 9D pose of each corresponding object. For example, the model may detect the car204in the real world data frame208. In this example, a 3D bounding box212is drawn around the predicted 9D pose of the car204.

At block210, the 3D bounding box212is compared with a 2D ground truth bounding box214. As previously discussed, the 2D ground truth bounding box214may be drawn by a human annotator. In the present example, the model has overestimated one or more attributes corresponding to the car's2049D pose. That is, the height, length, and width of the 3D bounding box212are greater than the height, length, and width of the car204.

To improve the accuracy of object detection applications, such as a robotic device's interactions with real world objects, a predicted bounding box should be substantially similar to a ground truth bounding box of the detected object. In this example, the 3D bounding box212generated based on predicted attributes of the car204, should be substantially similar to the 9D pose of the car204. To improve the model's accuracy, the model's parameters may be updated to reduce a prediction error.

Still, in this example, the model cannot determine a difference between coordinates of the 3D bounding box212and the 2D ground truth bounding box214. That is, because the number of coordinates differs between the 3D bounding box212and the 2D ground truth bounding box214, the prediction error is not determined. Thus, as shown in the current example, conventional systems may not determine a 3D prediction error by using a 2D ground truth bounding box214in a real world image of an object.

According to aspects of the present disclosure, the model generates a 3D bounding box for objects in an image. The 3D bounding box may be generated based on a function F( ). The function F( ) may be represented as:
F(x)=y,(1)
where x is a training image, such as a real world image or a simulated image, and y is a predicted 3D bounding box. Aspects of the present disclosure are directed to error minimization for a model. The error function E( ) of the model is as follows:
E(x,y*)=∥F(x)−y*∥.(2)

That is, for a training image x and a ground truth bounding box y*, the error function minimizes an error between the predicted 3D bounding box y and the ground truth bounding box y*. The error is not limited to a specific loss, such as L1 loss or L2 loss. In one configuration, the error is the difference in dimensions between the predicted 3D bounding box y and the ground truth bounding box y*. For example, a distance between corresponding corner points of the predicted 3D bounding box y and the ground truth bounding box y* may be used to determine the difference in dimensions.

When the ground truth bounding box y* is a 3D bounding box, the error may be minimized based on EQUATION 2. However, due to the difference in dimensions between the predicted 3D bounding box y and a 2D ground truth bounding box y*, the model cannot calculate the difference in dimensions between the predicted 3D bounding box y and the ground truth bounding box y*. Therefore, the model may not be updated to minimize the error between the predicted 3D bounding box y and a 2D ground truth bounding box y*.

To minimize the error between the 3D bounding box y and the 2D ground truth bounding box y*, the 3D bounding box y may be transformed into a 2D bounding box. In one configuration, a transformation function GO is used to transform the 3D bounding box y generated by F(x) to a format that is compatible with a 2D ground truth bounding box y*. When a transformation is necessitated to transform a predicted annotation (e.g., label) to a format that is compatible with a ground truth annotation, the error function ET( ) is as follows:
ET(x,y*)=∥G(F(x))−y*∥.(3)

FIG. 3illustrates an example of a transformation of a 3D bounding box to a 2D bounding box according to aspects of the present disclosure. As shown inFIG. 3, during training, at block300, the model receives a real world data frame308that includes a real world image of a road302and a car304. At block305, based on initial training, the model detects one or more objects in the real world data frame308. For example, the model may detect the car304in the real world data frame308.

Each detected object may be localized with a 3D bounding box312. In this example, the 3D bounding box312is determined based on the predicted 9D pose of the car304. Furthermore, the model may have prior knowledge that the real world data frame308is annotated with a 2D ground truth bounding box316. Therefore, as shown in block310, a transform function is used to transform the 3D bounding box312to a 2D bounding box314. In one configuration, when using a transform function, the 3D bounding box312is not drawn around an object, rather, the dimensions are used by the transform function to transform the 2D bounding box314. The 3D bounding box312ofFIG. 3is provided for illustrative purposes.

At block315, the 2D bounding box314is compared with a 2D ground truth bounding box316. Because the 2D bounding box314and the 2D ground truth bounding box316have similar dimensions, coordinates of the 2D bounding box314may be compared with coordinates of the 2D ground truth bounding box316. The error function for the model minimizes the difference318between the 2D bounding box314and the 2D ground truth bounding box316. That is, parameters of the model may be updated to minimize the aforementioned error.

In one configuration, the transformation function transforms a 3D bounding box to a 2D bounding box. In this configuration, the transformation function GO is as follows:

In EQUATION 4, a 2D corner point (x′, y′) of a 2D bounding box is calculated from each corner point (x, y, z) of the 3D bounding box. In one configuration, T is a 3×1 translation vector, R is a 3×3 rotation matrix,

(xyz)
are coordinates to one corner point of the 3D bounding box, and

(fx0cx0fycy001)
is the camera projection matrix. For the camera projection matrix, fxand fyare the focal length for either image axis, and cxand cyrepresent the principal point of the image. The values for the camera projection matrix are intrinsic to the 2D sensor. The values for T, R, and

(xyz)
may be obtained from the 3D bounding box (e.g., F(x)). The transformation function calculates (x′/z′, y′/z′) coordinates of a 2D bounding box from each corner point (x, y, z) coordinate of the 3D bounding box. After transforming all (x, y, z) coordinates of the 3D bounding box, the transform function performs a min and a max operation on all eight transformed points to obtain the final four points of the 2D bounding box.

The transform function is not limited to transforming a 3D bounding box to a 2D bounding box. According to aspects of the present disclosure, the transform function may be used for other types of perception discrepancies. For example, a model may be trained to generate a parametric representation of a road. The representation may be a spline, a polynomial, or another type of parametric representation. For example, given multiple control points, the model may determine the spline of the road. In this example, the parametric representation determines a curve of the road.

When training the model with simulated data, the model may determine a parametric representation of a simulated road for a given simulated data frame. The device that generated the simulated data may also calculate a ground truth parametric representation of the simulated road. For example, the model may predict a spline of the road and the predicted spline may be compared to a ground truth spline to determine a prediction error. An error function, such as the error function of EQUATION 2, may be used to minimize the prediction error between the predicted spline and the ground truth spline.

FIG. 4Aillustrates an example of determining a parametric representation for an element in a simulated data frame408according to aspects of the present disclosure. As shown inFIG. 4A, during training, at block400, the model receives a simulated data frame408that includes a simulated image of a road402. At block405, based on initial training, the model determines a parametric parameter for one or more elements in the simulated data frame408.

For example, the model may be trained to determine the parametric parameter of the road402. Of course, aspects of the present disclosure are not limited to determining the parametric parameter of the road402. The model may determine the parametric parameter of other elements in a frame. As shown inFIG. 4A, the model identifies multiple points406on the road402to predict a spline412. The predicted spline412represents a predicted curve of the road402.

At block410, the predicted spline412is compared with a ground truth spline414. In one configuration, the ground truth spline414is generated as part of the simulation data. Because the road402is a simulated object, the simulation data provides accurate data of the road's402parametric representation to generate the ground truth spline414. Thus, as shown inFIG. 4A, the ground truth spline414is a more accurate representation of the road's402curve in comparison to the predicted spline412.

In the present example, a difference between coordinates of the predicted spline412and the ground truth spline414may be used to determine a prediction error. An error function, such as the error function of EQUATION 2, may be based on the minimized difference between coordinates of the predicted spline412and the ground truth spline414. In this example, for EQUATION 2, x is the simulated data frame408, y is the predicted spline412(e.g., the output of the function F(x)), and y* is the ground truth spline414.

The simulated data may include information used to generate an accurate ground truth parametric representation. Additionally, or alternatively, the simulated data may include information that identifies specific pixels that correspond to each element in a frame. For example, for simulated data that includes a road, the simulated data may identify specific pixels that correspond to the road and other specific pixels that correspond to other elements, such as trees.

FIG. 4Billustrates an example of determining a parametric representation for an element in a simulated data frame428according to aspects of the present disclosure. As shown inFIG. 4B, during training, at block450, the model receives a simulated data frame428that includes a simulated image of a road422. At block455, based on initial training, the model determines a parametric parameter for one or more elements in the simulated data frame428.

For example, the model may be trained to determine the parametric parameter of the road422. Of course, aspects of the present disclosure are not limited to determining the parametric parameter of the road422. The model may determine the parametric parameter of other elements in a frame. As shown inFIG. 4B, the model identifies multiple points426on the road422to predict a spline432. The predicted spline432represents a predicted curve of the road422.

In the present example, the simulated data does not include a ground truth spline. Rather, the simulated data has distinguished pixels corresponding to the road422from other pixels in the frame428. Therefore, the format of the predicted spline432does not match a format of the ground truth representation424of the road422. Thus, the predicted spline432is transformed to a format that corresponds to the format of the ground truth representation424of the road422. In this example, the predicted spline432is transformed to a transformed spline430. The transformed spline430is for illustrative purposes and is not meant to limit aspects of the present disclosure to the format shown inFIG. 4B.

At block460, the transformed spline430is compared with the ground truth representation424of the road422. In the present example, a difference between coordinates of the transformed spline430and the ground truth representation424of the road422may be used to determine a prediction error. An error function, such as the error function of EQUATION 3, may be based on the minimized difference between coordinates of the transformed spline430and the ground truth representation424of the road422. In this example, for EQUATION 3, x is the simulated data frame428, y is the predicted spline432(e.g., the output of the function F(x)), and y* is the ground truth representation424of the road422. Additionally, the function GO transforms the predicted spline432to the transformed spline430.

For real world data, a ground truth parametric representation is provided by a human annotator. Real world data does not include information for a human annotator to provide an accurate ground truth parametric representation. Still, the human annotator may generate substantially accurate ground truth parametric representations by performing various calculations using the real world data. Nonetheless, the amount of time needed to generate substantially accurate ground truth parametric representations by the human annotator may cause an increase in training time. To reduce training time, the human annotator may be given a limited time to annotate real world data with ground truth parametric representations.

Given the limited time to annotate real world data, the ground truth parametric representations provided by the human annotator may be inaccurate. However, given the limited time, the human annotator may be able to accurately identify various features in the real world data. In one configuration, the human annotator distinguishes pixels corresponding to one element from pixels corresponding to other elements in the real world data. For example, when determining a ground truth parametric representation of a road, the human annotator identifies pixels corresponding to the road.

FIG. 5illustrates an example of determining a parametric representation for an element in a real world data frame508according to aspects of the present disclosure. As shown inFIG. 5, during training, at block500, the model receives the real world data frame508that includes a real world image of a road502. The real world data frame508may comprise multiple pixels. Each pixel may be a cell in a grid of cells.

At block505, based on a given task, such as annotating the road502, a human annotator distinguishes pixels of the road502from other pixels in the real world data frame508. For example, as shown in block505, the human annotator highlights the pixels corresponding to the road502. The highlighted pixels may be used as the ground truth representation504of the road502.

At block510, based on initial training parameters, the model identifies multiple points506on the road502to predict a spline512. The predicted spline512represents a predicted curve of the road502. In most cases, the predicted spline512is compared to a ground truth spline to determine a prediction error. (SeeFIG. 4A). However, in some cases, the ground truth spline is not available. For example, as previously discussed, when faced with a limited time, the human annotator may not provide accurate ground truth splines. Rather, as shown in block505, the human annotator distinguishes pixels of the road502from pixels of other elements, such as trees and background.

To mitigate the difference between the predicted spline512and the distinguished pixels of the road502, a translator function may be used to convert the predicted spline512to a format that can be compared against the ground truth representation504of the road502. In one configuration, the transform function transforms the predicted spline512to a transformed spline514that is compared with the ground truth representation504. The predicted error may be determined based on a comparison of the translated spline514and the ground truth representation504.

At block515, the transformed spline514is compared with a ground truth representation504. In the present example, a difference between coordinates of the translated spline514and the ground truth representation504may be used to determine a prediction error. An error function, such as the error function of EQUATION 2, may minimize the difference between coordinates of the translated spline514and the ground truth representation504. In this example, for EQUATION 3, x is the real world data frame508, y is the translated spline514(e.g., the output of the function f(x)), and y* is the ground truth representation504. Additionally, the function GO transforms the predicted spline512to the transformed spline514.

FIG. 6is a diagram illustrating an example of a hardware implementation for an object localization system600according to aspects of the present disclosure. The object localization system600may be a component of a vehicle, a robotic device, or other device. For example, as shown inFIG. 6, the object localization system600is a component of a car628. Of course, aspects of the present disclosure are not limited to the object localization system600being a component of the car628, as other devices, such as a bus, boat, drone, or robot, are also contemplated for using the object localization system600.

The object localization system600may be implemented with a bus architecture, represented generally by a bus630. The bus630may include any number of interconnecting buses and bridges depending on the specific application of the object localization system600and the overall design constraints. The bus630links together various circuits including one or more processors and/or hardware modules, represented by a processor620, a communication module622, a location module618, a sensor module602, a locomotion module626, a planning module624, and a computer-readable medium614. The bus630may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.

The object localization system600includes a transceiver616coupled to the processor620, the sensor module602, an artificial neural network module608, the communication module622, the location module618, the sensor module602, the locomotion module626, the planning module624, and the computer-readable medium614. The transceiver616is coupled to antennae632. The transceiver616communicates with various other devices over a transmission medium. For example, the transceiver616may receive commands via transmissions from a user or a remote device. As another example, the transceiver616may transmit driving statistics and information from the artificial neural network module608to a server (not shown).

The object localization system600includes the processor620coupled to the computer-readable medium614. The processor620performs processing, including the execution of software stored on the computer-readable medium614providing functionality according to the disclosure. The software, when executed by the processor620, causes the object localization system600to perform the various functions described for a particular device, such as the car628, or any of the modules602,608,614,616,618,620,622,624,626. The computer-readable medium614may also be used for storing data that is manipulated by the processor620when executing the software.

The sensor module602may be used to obtain measurements via different sensors, such as a first sensor606and a second sensor604. The first sensor606may be a vision sensor, such as a stereoscopic camera or an RGB camera, for capturing 2D images. The second sensor604may be ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the aforementioned sensors as other types of sensors, such as, for example, thermal, sonar, and/or lasers are also contemplated for either of the sensors604,606. The measurements of the first sensor606and the second sensor604may be processed by one or more of the processor620, the sensor module602, the artificial neural network module608, the communication module622, the location module618, the locomotion module626, the planning module624, and the computer-readable medium614.

As previously discussed, the measurements from the first sensor606may be used to capture 2D images. Furthermore, the measurements from the second sensor604may be used for depth measurements. In one configuration, the data captured by the first sensor606and the second sensor604may be transmitted to an external device via the transceiver616. The first sensor606and the second sensor604may be coupled to the car628or may be in communication with the car628.

The location module618may be used to determine a location of the car628. For example, the location module618may use a global positioning system (GPS) to determine the location of the car628. The communication module622may be used to facilitate communications via the transceiver616. For example, the communication module622may be configured to provide communication capabilities via different wireless protocols, such as WiFi, long term evolution (LTE), 3G, etc. The communication module622may also be used to communicate with other components of the car628that are not modules of the object localization system600.

The locomotion module626may be used to facilitate locomotion of the car628. As an example, the locomotion module626may control movement of the wheels. As another example, the locomotion module626may be in communication with a power source of the car628, such as an engine or batteries. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels and are contemplated for any other type of component for providing locomotion, such as propellers, treads, fins, and/or jet engines.

The object localization system600also includes a planning module624for planning a route or controlling the locomotion of the car628, via the locomotion module626, based on the analysis performed by the artificial neural network608. In one configuration, the planning module624overrides the user input when the user input is expected (e.g., predicted) to cause a collision. The modules may be software modules running in the processor620, resident/stored in the computer-readable medium614, one or more hardware modules coupled to the processor620, or some combination thereof.

The artificial neural network608may be in communication with the sensor module602, the transceiver616, the processor620, the communication module622, the location module618, the sensor module602, the locomotion module626, the planning module624, and the computer-readable medium614. In one configuration, the artificial neural network608receives sensor data from the sensor module602. The sensor module602may receive the sensor data from the first sensor606and the second sensor604. According to aspects of the disclosure, the sensor module602may filter the data to remove noise, encode the data, decode the data, merge the data, extract frames, or perform other functions. In an alternate configuration, the artificial neural network608may receive sensor data directly from the first sensor606and the second sensor604.

As shown inFIG. 6, the artificial neural network608(e.g., machine learning module) may include an extractor610and an annotator612. The extractor610and the annotator612may be components of a same or different convolutional neural network (CNN), such as a deep CNN. The artificial neural network608is not limited to a CNN and may be another type of artificial neural network, such as a support vector machine (SVM). The extractor610receives a data stream from the first sensor606and the second sensor604. The data stream may be data combined from the first sensor606and the second sensor604. For example, the data stream may be a 2D RGB image from the first sensor606that is merged with LIDAR data points from the second sensor604. In another configuration, the data stream is a separate stream from each sensor604,606. The data stream may include multiple frames, such as image frames.

The extractor610extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the extractor610may be trained to extract features of 3D objects. As another example, the extractor610may be trained to extract features of different terrains, such as roads, sidewalks, buildings, and background. That is, the exactor610identifies areas of attention based on the training. The artificial neural network608may include one or more extractors610. For example, one extractor610may be configured to detect 3D objects and another extractor610may be configured to segment different elements of the data, such as roads, sidewalks, buildings, and background.

The annotator612receives the extracted features from the extractor610to annotate the area of interest. The annotator612may be a classification segment of the CNN, as opposed to a human annotator as discussed herein before. As previously discussed, the annotator612may be configured to draw a 3D bounding box around an area of interest, such as an object. In another configuration, the annotator612determines a parametric representation of an area of interest, such as a road, or traffic lane. The artificial neural network608may output the annotated data from the annotator612to one or more of the sensor module602, the transceiver616, the processor620, the communication module622, the location module618, the locomotion module626, the planning module624, and the computer-readable medium614. For example, the annotated data may output to the planning module624for route planning, collision avoidance, or other planning functionality.

Aspects of the present disclosure are directed to improving the training of the model that includes an extractor and annotator.FIG. 7illustrates a flow diagram for training a model700according to aspects of the present disclosure. In one configuration, training data704,710may be stored at a data source, such as a server. As shown inFIG. 7, the real training data704may be distinguished from the simulated training data710. The different training data704,710may be stored on separate servers, distinguished via meta data, or some other type of distinction. During training, a set of samples702are selected from one of the sources of training data704,710. The set of samples702includes the input data x, such as the simulated data and the real world data. Additionally, the set of samples702includes ground truth labels y* corresponding to the input data x.

The model700may be initialized with a set of parameters w. The parameters may be used by layers of the model700, such as layer1, layer2, and layer3, of the model700to set weights and biases. The extractor and annotator ofFIG. 6may be different layers of the model700. During training, the model700receives input data x to transform the input data x to an output y. As shown in EQUATION 1, the model700may be based on a function F( ). The output y may be parameters of an annotated element, such as a 3D bounding box or a parametric representation.

During training, if the source of a sample702was the real training data704, the output y of the model700is input to a transform function706(G(F(x))). As discussed above, the transform function706may transform the output y to a format that corresponds to a format of the ground truth label y*. For example, if the ground truth label y* is a 2D bounding box, the transform function706transforms the 3D bounding box to a 2D bounding box. That is, the transform function706bridges a domain gap between the output y and ground truth labels that are in a different format from the output y. In some cases, such as when the ground truth label y* is in the same format as the output y, the transform function706is not used.

The output of the transform function706is received at a loss function708. Depending on whether the output y was transformed, the output of the transform function706may be the transformed output y or non-transformed output y. The loss function708compares the transformed output y or non-transformed output y to the ground truth label y*. The error is the difference (e.g., loss) between the transformed output y or non-transformed output y and the ground truth label y*. The error is output from the loss function708to the model700. The error is backpropagated through the model700to update the parameters. As shown in EQUATIONS2and3, an error function E( ) and ET( ) minimizes the error determined by the loss function708. The training may be performed during an offline phase of the model.

FIG. 8illustrates a method800for training a machine learning model according to an aspect of the present disclosure. At block802, the machine learning model receives real data comprising a real element in a real environment. The real data may include frames captured by a sensor, such as an RGB camera, LIDAR, RADAR, etc. At block804, the machine learning model annotates the real element with a first annotation based on predicted attributes of the real element. The first annotation may be in a first format, such as a three-dimensional bounding box. For example, based on the predicted attributes, such as the dimensions of the real element, a three-dimensional bounding box may be placed around the real element. In another configuration, the first format is a parametric representation.

At block806, the machine learning model converts the first annotation from the first format to a second format corresponding to a ground truth annotation of the real element. The second format may be a two-dimensional bounding box or distinguished pixels. For example, the three-dimensional bounding box may be converted to a two-dimensional bounding box. In another example, the parametric representation may be converted to distinguished pixels. At block808, the machine learning model adjusts parameters of the machine learning model to minimize a difference between values of the ground truth annotation of the real element and the converted first annotation. In one configuration, the ground truth annotation of the real element is annotated by a human annotator.

In an optional configuration, at block810, the machine learning model receives simulated data comprising a simulated element in a simulated environment. For example, the simulated data may be simulated by virtual environment that is generated by a computer. In another optional configuration, at block812, the machine learning model annotates the real element with a second annotation based on predicted attributes of the simulated element. The second annotation may be in the first format. The second annotation may be annotated by the computer that generated the simulated data. For example, the second annotation may be a three-dimensional bounding box.

In still another optional configuration, at block814, the machine learning model adjusts the parameters to minimize a difference between values of a ground truth annotation of the simulated element and the second annotation. The parameters may include weights of the machine learning model. After adjusting the parameters of the machine learning model (see blocks808and814), in an optional configuration, at block816, the adjusted machine learning model plans a route for a robotic device. The robotic device may be a vehicle, such as a drone or a car.

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a specially configured processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A specially configured processor may be a microprocessor, but in the alternative, the processor may be a commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The processor may be responsible for managing the bus and general processing, including the execution of software stored on the machine-readable media. Software shall be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Machine-readable media may include, by way of example, random access memory (RAM), flash memory, read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable Read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable media may be embodied in a computer-program product. The computer-program product may comprise packaging materials.