A Feast of Sensor Data: Feeding Self-driving Algorithms
Tags: Self Driving Cars, Data Labeling, Annotation Tools, Autonomous Cars, Computer Vision

10 minutes read

Training the algorithms of AI-based systems for autonomous or highly automated driving requires enormous volumes of data to be captured and processed. The algorithms must be able to master numerous challenges so that self-driving cars can detect all essential details of their environment, make the right decisions and take people safely to their destination. Warning! Number crunching coming — get your calculators ready.

Every Data Annotation Project is Large Scale in Autonomous Driving

Why does training require this much data? AI-based systems enable quick progress, but this progress slows down after a certain point. More and more data is needed to improve the model performance. Moreover, we have to make sure that self-driving systems can reliably handle rare events. Rare event data has a significant impact on the performance of the trained algorithm. Bringing sophisticated AI-based driving functions to the road safely therefore requires a growing amount of ever higher-quality data.

Learning progress over time

In general, AI functions in production systems must cross a very high reliability threshold before being used in real systems. In automated driving in particular, the associated safety risks are extremely high. The tragic accidents at Uber and Tesla are admonishing reminders of this. As a consequence, registration authorities require ever stricter validation measures for high-quality driving functions to ensure their correct operation. Since these validation measures are often based on driven kilometers, they require large volumes of data.

Threshold for usable models

The following calculation example will give you an impression of the data volumes that have to be processed to validate a combined radar and camera system (sensor system).

Sensor Set-up & Method

For this example, the correct functioning of the sensor system must be demonstrated over a distance of 300,000 km with a predefined mix of driving scenarios. The sensor system provides an object list that describes the vehicle’s environment.

To check if the sensor system object list is correct, a comparison list, called ground truth, is required. This list is generated using a reference sensor set consisting of a camera and lidar sensor. The reference sensor set is installed in the vehicle as a rooftop box in addition to the sensors to be checked. The data stream of the reference sensor set is transferred to the ground truth object list by means of an annotation process. By systematically comparing the two object lists, deviations are detected and corrective measures can be derived.

Development of a validation project

Estimating the Labeling Effort

The estimation of the annotation effort, which is still primarily done by humans, is based on the following assumptions regarding the parameters of the reference sensor setup.

Lidar frequency: 10 frames/second

Achieved average speed: 45 km/h

Average number of objects (vehicles, pedestrians, motorcycles) per frame: 15 objects/frame

Example Calculation of a Validation Project

Travel time in seconds: 300,000 km / 45 km/h = 6,666 h = 23,997,600 s Number of frames: 23,997,600 s * 10 f/s = 239,976,000 f Number of objects: 239,976,000 f * 15 o/f = 3,599,640,000 o

This results in an annotation effort of almost 3.6 billion objects.

State-of-the-art systems achieve a throughput of approx. 4 seconds per object in a 3D space. The time required is measured end-to-end, i.e., across all annotation, review, and correction steps, from import to export.

Annotated objects in a 3D space

Time in seconds: 3,599,640,000 o * 4 s/o = 14,398,560,000 s Work estimate: 28,797,120,000 s = 3,999,600 h = 499,950 8hr days Project duration: 3 years (220 workdays each) Number of labelers needed: 499,950 d / 660 wd = 757.5 labelers

Assuming a project duration of 3 years (at 220 working days per year), an annotation team with an average size of approx. 750 people would be required.

It’s a lot!

The challenge lies in providing a data annotation system that is capable of continuously providing 750 people with sufficient work and enabling an average processing time of 4 s per object for the specified target quality — across all employees and skill levels. The average processing time per object must be further reduced, before the costs go through the roof. Annotation automation is a key ingredient. Learn more about it in our video Why is automated labeling essential for autonomous driving?

What else needs to be done?