How might we accelerate advent of autonomous driving?
By: Petr Meissner
Tags: Long Tail, Autonomous Vehicles, Future, Self Driving Cars, Edge Cases

The race of approaches

Waymo started to operate the first public fully autonomous raid-hailing service. Yet the operation is still geo-fenced to the Phoenix area, heavily relies on precise 3D maps with resolution in centimeters and only 5-10 % of rides are fully autonomous without the need for a human operator. This is the result of a 3 years testing with a closed group of passengers and a fleet of 600 cars.

Meanwhile Tesla announced that their autonomous driving architecture is full-self-driving ready and already started rolling it out to first beta customers. (Let’s not confuse readiness with capability though.)

They nicely represent the two distinct approaches, so the bets were made.

Precision & redundancy vs. abundant data

The two approaches have their own advantages and drawbacks.

Precision & redundancy: Reliable fail-safe limited by area, costs and growth speed :

Waymo counts on reliable lidars providing a full (yet sparse) 3D scene of all surroundings. They rely on accurate 3D maps that needs to be collected in any area they want to operate in. That data is supplemented by further image recognition data from cameras, radars and ultrasonic sensors. While it gives plenty of redundant data to rely on, it also limits how fast they can grow to new areas, thus limiting their growth.

: Waymo counts on reliable lidars providing a full (yet sparse) 3D scene of all surroundings. They rely on accurate 3D maps that needs to be collected in any area they want to operate in. That data is supplemented by further image recognition data from cameras, radars and ultrasonic sensors. While it gives plenty of redundant data to rely on, it also limits how fast they can grow to new areas, thus limiting their growth. Abundant data limited by a need for driver’s baby-sitting:

Tesla ditched lidars as unnecessary and relies heavily on image processing from vehicle cameras to reconstruct all the scene, supported by radars and ultrasonics.

While it was probably the only affordable approach for Tesla to benefit from their huge fleet (1000× more cars, 200× miles (2B) than Waymo) and drivers teaching the system how to handle risky situations, it also means that no Tesla can drive itself alone without a baby-sitting of a human driver who continues to be responsible for accidents. Nobody knows how long it will take them to get to the point where it will be more reliable than human drivers in any situation, even the hard ones.

What will be the winning approach that will reach cities at scale faster? More data to train autonomous systems? Or reliable redundant fail-safe methods limited by 3D maps?

Challenges, plus tricks of the rest of the pack

What are some challenges to be overcome identified by the creators of autonomous vehicles?

WAYMO principal scientist Drago Anguelov emphasized:

a long-tail testing challenge

testing challenge behavioral corner cases with not enough training data

WAYMO director of engineering Sasha Arnoud claims that:

when you are 90 % done you have 90 % to go

long tail of very rare situations of diverse situations including objects states like flashing police cars

Tesla’s AI director Andrej Karpathy emphasizes:

lots of edge-cases across countries in many variations

in many variations plus weird situations of what a car can encounter on roads

Mobileye (Intel, Fiat Chrysler, BMW) already makes profit from delivering driver-assistance systems (ADAS) to major car makers and creating 3D maps from what their sensors see. Their CEO Amnon Shashua claims they are able to reconstruct a 3D scene from even a single camera. They were actually delivering a first generation of Tesla’s autopilot till 2016 but parted their ways because they insist on their partners not having their own autonomous R&D.

Aurora (VW & Hyundai) CEO Chris Urmson in this interview and talk (from previous job at Google) wants:

Scene recognition of car going through a construction zone (from Chris Urmson’s talk, gif)

a magic wand to solve perception forecasting what will happen in next seconds

Aurora co-founder Sterling Anderson wants to make autonomy broadly available to all who share the vision and considers tech challenges:

forecasting intent and future behavior of all actors and how they’ll respond to actions of your own

of all actors and how they’ll respond to actions of your own long-tail of development

Voyage (Fiat Chrysler) CEO Oliver Cameron thinks to partner with industry is the right way to go (MB, NVIDIA, Uber ATG) and describes:

hard edge-cases like moving foliage/bushes

like moving foliage/bushes risk of misidentified or missed objects

Their inspiring approach is to focus on retirement communities. While it might be surprising at first, it’s quite a match with calm, distant areas to serve, people with special needs or without an ability to drive and less demands on the speed of transportation. He claims that seniors are not afraid of this technology, because they perceive it rather a more accessible taxi than some high-tech oddity.

APTIV/Motional’s (Hyundai) CEO Karl Iagnemma & director Oscar Beijbom, present challenges:

completeness and biases of data

not enough data from rare crashes

Cruise (GM, Honda) CTO Kyle Vogt in his presentation and interview mentions we need to:

do thousands of little things in handling every variation of each situation , context and intent (acknowledges humans for being excellent in edge-cases)

, context and intent (acknowledges humans for being excellent in edge-cases) maximize diversity of data, large datasets , mix of maneuvers

of data, , mix of maneuvers focus on disengaging as a signal

as a signal price of datasets and learning : collecting and more importantly labeling 1 M miles costs 1 $B. It takes weeks to train and test a new version.

: collecting and more importantly labeling 1 M miles costs 1 $B. It takes weeks to train and test a new version. benchmarking and estimating performance and categorizing safety risks

What a car might encounter on road (from Andrej Karpathy’s workshop video)

Comma.ai founder George Hotz considers a challenge:

interaction between actors (counterfactuals)

(counterfactuals) getting real human behavior

prediction of occluded areas

Optimus Ride co-founder Sertac Karaman sees challenging to:

simulate and predict human behavior (what will people do at a specific intersection hundreds of times?)

(what will people do at a specific intersection hundreds of times?) typical accidents

Lyft level 5’s Head of Research Peter Ondrúška published about a need to:

learn from human driving and experiences

and experiences he sees an advantage in acquiring data to feed algorithms with detailed real-world driving scenarios at scale through a rideshare network

Stanford professor, Policies and Safety, Chris Gerdes asks for sharing information about edge-case scenarios to learn collectively like in Aviation Safety Information Analysis and Sharing (ASIAS) system: voluntarily sharing safety information across the industry with anonymized assessment/benchmarking of how they perform in comparison with others.

Will data decide?

What repeats across is the need to train for a very long tail of edge-cases (especially for rare critical events) and predicting real human behavior and interactions. How much of the data is needed so the fleet is ready to go into the wild to outmatch human drivers in cities at scale?

Will providing in order of magnitude more real street driving clips of the edge-cases help companies reach full autonomy at scale across the world earlier?

Can the same recorded data be adapted for multiple autonomous systems and positions and characteristics of their sensors?

Now e. g. Tesla has an advantage of its 1000× bigger fleet and 200× more miles street-clips gathered (2 B miles) compared to Waymo and MobilEye claims to be collecting 6 millions km daily from their driving asistents in VWs, BMWs and other cars. And the gap grows.

There seems to be players with and without a direct potential to get abundant data. Like Tesla/MobilEye/Lyft level 5 vs. Waymo/Cruise/Aurora… Will the data be the final winning trick or does it have a lower importance in the whole mix of complexities of autonomous systems? Obviously those with the data are not willing to share.

Is the data what holds us back? What if everyone had enough data, especially about the rare accidents and the only limit would be an ability to use it? How would it accelerate autonomy?

What a car might encounter: a lady in a wheelchair chasing a duck with a broom in traffic (source: Google, gif: MailOnline)

How might we accelerate it?

What if we create consumer products that will collect real-world driving clips and behavior faster? There’s plenty of synergies in collecting data this way: from improving safety and security, evidence for accidents reporting, to monitoring pavement conditions, free parking spots and “real” “real-time” traffic info and more. (Note wejo’s traffic intelligence.)

There is already a first example: nauto collects real-world driving data by offering a driving assistant to improve safety of professional fleets on the road. They claim they collected 800 M miles with risk events identified. So far they partner only with car makers but not the autonomous cars companies. Is there too little “interesting moments” compared to billions of Tesla to make any difference?

Can we provide autonomous car makers an order of magnitude more real-life rare edge-cases data to train their systems cheaper? How could it accelerate maturing the autonomous systems?

Value presentation from nauto.com

Could “ASIAS for autonomous driving” that would allow autonomous vehicles creators to share data of rare edge-cases at high volumes help them the same way its aviation counterpart helps airlines to eliminate even subtle issues and remove operations blind-spots?

Are we there yet?

Although Tesla markets its autopilot to be “full self-driving capable” in the future, as Oliver Cameron pointed out in examples of driver interventions it needed from its driver, it’s clear how many remaining challenges they still have ahead to win the long-tail game. This seems more like some kind of BSD (Babysitted Self Driving) instead of the FSD (Full Self Driving).

Oliver Cameron appreciated Tesla's BSD, but pointed out remaining challenges.

Waymo announced no plans to mature out from their Phoenix sandbox into the wild yet.

While MobilEye announced the launch of their AV system in 2022 and demonstrated its capabilities in 40 mins uninterrupted driving video, it is unclear what level of autonomy under what constraints or limitations they can achieve at a wider scale in general conditions across cities.

I don’t dare to bet on who will win the race. Not even if lidar fail-safe or vision only approach will have an edge (depth perception just from cameras seems getting traction as light.co demonstrated, so these might even converge at the end of the day). But my personal guess is that it will take us at least 10 years for autonomous cars to reach major capital cities because it will take a huge amount of time and kilometers ridden to collect data to tame the long-tail of all the variations of the situations that can happen on the road to outperform humans reliably in general conditions without relying on the human interventions.

One diverted shortcut though

There are two areas though, where the autonomy will come sooner than the robo-taxis. Trucks in logistics and a last-mile delivery. Those will make services quicker and cheaper. But those won’t change cities as significantly as autonomous ride-hailing.

Only the ride-hailing will change the way we commute by helping us to get rid of all the hassle of car ownership, making it cheaper, more available to all while reclaiming car parking space cities back to people. And.. saving millions of lives.

Are we there yet?

What is your opinion? What do I miss? How might we accelerate the advent of autonomous vehicles? Share in comments.