Automated large scale data generation for autonomous vehicle
By: Cong Jie Cj
Tags: Self Driving Cars, Perception, Data, Scalability, Neural Networks

Over one year ago we built our pure vision neural network-based autonomous truck system to drive from Seoul to Busan fully autonomously. Our perception system improves according to the increasing amount of good quality ground truth data. It is necessary to build a scalable data acquisition pipeline loop from collecting raw driving data from our trucks, ground truth data annotation, to training a neural network. We developed in-house ML-powered annotation tools with humans in the loop to acquire ground truth data to fuel our perception system. It is a tedious and expensive (in terms of time and human resource) process as it requires proper training for our annotators to have consistent annotation and annotated data quality checks. As our perception system takes advantage of both spatial and temporal information, annotating long sequence of video frames become even more expensive. Scaling up to million miles of driving video data collected from our trucks becomes infeasible.

Outsource data annotation too does not scale

Figure 1: Manual data annotation by human

Another alternative for data annotation is to outsource the job to a third party company. Many self-driving car companies opt for this approach but it comes with a very high price tag. One of the very well known third party companies in the industry is scale ai. Some autonomous vehicle open-dataset such as the Lyft Level 5 and nuScenes are curated by the third party service. While third-party services usually rely on ML-powered with the human in the loop data annotation, turning raw driving video data into a useful ground truth is very costly. Take an example of a single frame extracted from our raw driving video data as shown in figure 1., annotating the image consists of 5 lane lines and 12 vehicles with the single image on-demand pricing (as of the date of posting the annotation cost 0.08/Image + 0.08/Annotation) would cost USD 1.44 for this particular video frame. Extrapolating this to an hour driving video of 20 FPS would cost USD 103,680! Note that, this is just an example, our perception system does not rely on 2D image plane ground truth for learning. These approaches whether by in-house manual annotation or outsource annotation simply do not scale!

Automatic data annotation

With the goal of deployment to a large number of trucks in mind, being able to automate this process, is one of the keys to scalability. Since then, with a very small team of engineers, we revamped our data acquisition pipeline to get rid of the expensive process by developing sensor fusion algorithms to automate the job. Unlike most self-driving car sensor suites which are equipped with multiple depth-sensing sensors (i.e., LiDAR), with the absence of LiDAR in our system, generating absolute scale ground truth data in 3D becomes very challenging. Instead of relying on additional hardware to perceive depth (which can increase the cost to our system), we took a more challenging approach by solving this problem with maths and algorithms. With a monocular camera as our main forward sensing sensor and other built-in sensors measurements across space and time obtained from our trucks, our algorithm is able to perceive absolute scale depth for both static scene and dynamic moving objects in 3D as shown in figure 2. from our generated data.

Figure 2: Automated data annotation without human

Today, while our trucks are being used to do the logistic shipping from warehouse to warehouse, we are able to turn the raw video data automatically into map and useful ground truth data of many highway routes in South Korea under different traffic, weather, and lighting conditions. This in turn enables us to leverage our perception system to the next level.

In Mars Auto, we solve many challenging problems to build an autonomous truck system. Join us on our journey to build the future!