Learning Robust Driving Policies
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems Tübingen

August 23, 2020

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupCollaborators

Eshed Ohn-Bar

Kashyap Chitta

Aditya Prakash

Aseem Behl

Andreas Geiger

2Imitation Learning

Motivation: Hard coding policies is difﬁcult ⇒ follow data-driven approach!
I Given: demonstrations or demonstrator
I Goal: train a policy to mimic decision3

Conditional Imitation Learning
Advantages:
I End-to-End Trainable
I Cheap Annotations
Limitations:
I Generalization
I High Sample Complexity
I Covariate Shift
I Interpretability

Codevilla, Santana, Lopez and Gaidon: Exploring the Limitations of Behavior Cloning for Autonomous Driving. ICCV, 2019.

4

How can we learn to drive under the vast diversityof all visual, planning and control scenarios?

Situational Driving

6

Inspiration: World Models

I Step 1: Learn generative model of game environments (VAE)
I Step 2: Learn dynamics model and control model in latent space (CMA-ES)
I Not sufﬁcient ⇒ we combine this idea with imitation learning
Ha and Schmidhuber: Recurrent World Models Facilitate Policy Evolution. NeurIPS, 2018.

7

Learning Situational Driving
Proposed
Situational
2. Context
EmbeddingDriving (LSD) Framework
EnvironmentLearningqÁz

1. Expert Policies
1

¼µ

2

¼µ

:::

3. Task-Driven
Refinement¼Kµ

Demonstrations

I Step 1: Learn a mixture of expert policies {αθk , πθk } via imitation (LSD)
I Step 2: Learn a general purpose context embedding qφ as a β-VAE
I Step 3: Perform task-driven policy reﬁnement by interacting with the simulationand maximizing a driving task reward (LSD+)
Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

8

Learning Situational Drivingqφ (I)πΘ (a|o, c) =αθk (o, c) πθk (a|o, c) + Ψ  v | {z } | {z }k=1cMixtureExpertWeights| {z }
ModelsK
X

Context
Embedding2πθk (a|o, c) = N a µkθ (o, c), diag(σ kθ (o, c))

Observations:Command:
Actions:o = [I, v] ∈ O
c ∈ C = {left, right, straight, follow}a ∈ A = [−1, 1]2

Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

9

Learning Situational Drivingqφ (I)πΘ (a|o, c) =αθk (o, c) πθk (a|o, c) + Ψ  v | {z } | {z }k=1cMixtureExpertWeights| {z }
ModelsK
X

Context
Embedding2πθk (a|o, c) = N a µkθ (o, c), diag(σ kθ (o, c))

Training:
I Step 1: Learn Mixture of Experts: LMoE = − log

hP

K
k k
k=1 αθ πθ

i+ LV + LR

I Step 2: Learn Context Embedding: LVAE = β KL (qφ (z|I) k p0 (z)) + kdφ (z) − Ik22
hPi
T
I Step 3: Task-driven optimization: JTASK (θ readout , Ψ) = EπΘrtt=0Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

9

Experiments

CARLA Benchmark

Training Town

Test Town

I Random start and end location, 4 known weathers, 2 unseen weathers
I Metric: Percentage of successfully completed episodes (success rate)
I Collision does not necessarily terminate episodeOhn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

11

NoCrash Benchmark

Empty

RegularDense

I Difﬁculty varies with number of dynamic agents in the scene
I Empty: 0 Agents

Regular: 65 Agents

Dense: 220 Agents

I All collisions terminate episodeOhn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

12

AnyWeather Benchmark

I Evaluation on 10 unseen weathers, quantiﬁes generalization performance
Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

13

Importance of Mixture Model
Training Data and Mixture Components
Evaluation Task
Straight (Static)
One Turn (Static)
Navigation (Static)
Navigation (Dynamic)

Navigation (Static, K=1)

Navigation (Dynamic, K=1)

Navigation (Dynamic, K=3)

99
98
96
40

64
74
78
78

100
100
98
92

Results of Mixture Model on CARLA Benchmark:I Static model solves static scenes well but cannot handle dynamic objects
I Dynamic model handles dynamic scenes better but degrades on static scenes
I Dynamic mixture model generalizes to all scenarios (without on-policy data)
Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

14

Importance of Mixture Model and Task-based Reﬁnement
Model

Success Rate (%)

Monolithic (K=1)

75

MoE Shared Backbone (K=3)
MoE Shared Backbone (K=5)
MoE Shared Backbone (K=8)

89
90
87

MoE Separate Backbone (K=3)
MoE Separate Backbone (K=5)
MoE Separate Backbone (K=8)

94
93
93

MoE Separate Backbone + Reﬁnement (K=3)

98

Results of Full Model on CARLA Benchmark:I Performance improves up to 3 or 5 mixture components
I Separate backbones increase diversity and generalization
I Tasked-based reﬁnement improves performance further
Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

15

Emergent Driving Modes

frequency-1

Expert 30

brake / throttle

1-1frequency

Expert 2
frequency

Expert 1

0

brake / throttle

1

-1

0

brake / throttle

1

Emergent Driving Modes:
I Acceleration distribution of three different experts during testing

Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

16

Results on CARLA Benchmark

Driving Task

CIRL

CILRS

CILRS (ours)

LSD (ours)

LSD+R (ours)

Straight
One Turn
Navigation
Navigation Dynamic

100
71
53
41

96
84
69
66

96
86
67
64

100
99
99
94

100
99
99
98

I Using reward-based optimization alone (CIRL) is not sufﬁcient
I LSD enables better driving behavior across all driving tasks
I Large improvements in the presence of dynamic objects

Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

17

Results on CARLA NoCrash Benchmark

Driving Task

CILRS

CILRS

LSD (ours)

LSD+R (ours)

Expert

Empty
RegularDense

66 ± 249 ± 523 ± 165 ± 246 ± 220 ± 193 ± 266 ± 227 ± 294 ± 168 ± 230 ± 496 ± 091 ± 141 ± 2I All methods perform worse due to challenges (density, collision terminations)
I Expert provided by CARLA often fails in dense environments (e.g., clogging)
I LSD enables better driving behavior across all driving tasks

Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

18

Results on AnyWeather Benchmark
Task

CILRS

LSD (ours)

LSD+R (ours)

Straight
One Turn
Navigation
Nav. Dynamic

83.2
78.4
76.4
75.6

85.2
80.4
78.8
77.2

85.6
81.6
79.6
78.4

I AnyWeather benchmark test generalization to challenging unseen weathers
I All methods can fail even on simple straight driving tasks
I Some challenging weathers lead to zero success rate for all methods
I More research is required to address these challenges
Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

19

Qualitative Results

Ohn-Bar, Prakash, Behl, Chitta and Geiger: Learning Situational Driving. CVPR, 2020.

20

How useful is data aggregation for self-driving?

Imitation Learning

Hard coding policies is often difﬁcult ⇒ Rather use a data-driven approach!
I Given: demonstrations or demonstrator
I Goal: train a policy to mimic decision22Formal Deﬁnition of Imitation Learning
General Imitation Learning:argmin Es∼P (s|πθ ) [L (π ∗ (s), πθ (s))]θ

I State distribution P (s|πθ ) depends on rollout
determined by current policy πθ
Behavior Cloning:argmin E(s∗ ,a∗ )∼P ∗ [L (a∗ , πθ (s∗ ))]
|
{z
}
θ
P
∗ ,π (s∗ )
= N
L
a
)
(
θ
i=1
i
i
I State distribution P ∗ provided by expert
I Reduces to supervised learning problem
23Challenges of Behavior Cloning

I Behavior cloning makes IID assumption
I Next state is sampled from states observed during expert demonstration
I Thus, next state is sampled independently from action predicted by current policy

I What if πθ makes a mistake?
I Enters new states that haven’t been observed before
I New states not sampled from same (expert) distribution anymore
I Cannot recover, can lead to catastrophic failure

24Experiment by Held and Hein

Held and Hein: Movement-produced stimulation in the development of visually guided behavior. Journal of Comparative and Physiological Psychology, 1963.

25

DAgger

Data Aggregation (DAgger):
I Iteratively build a set of inputs that the ﬁnal policy is likely to encounter based on
previous experience. Query expert for aggregate dataset.
I But can easily overﬁt to main mode of demonstrations
I High training variance (random initialization, order of data)
Ross, Gordon and Bagnell: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.

26

Distribution over Driving Actions
Imitation Learning
Brake

Steer Left

Go Straight

DAgger Iter 1
Steer Right

Brake

Steer Left

10000050000

0ClearNoon

Steer Right

WetNoon

HardRainNoon

10000050000

0ClearSunset

ClearNoon

WetNoonweather conditions

Steer Left

HardRainNoon

ClearSunset

weather conditions

DAgger Iter 2
Brake

DAgger Iter 3

Go StraightSteer Right

Brake

Steer Left

Go StraightSteer Right

150000# images in training data

150000# images in training data

Go Straight

150000# images in training data

# images in training data

15000010000050000

0ClearNoon

WetNoon

HardRainNoon

weather conditions

ClearSunset

10000050000

0ClearNoon

WetNoon

HardRainNoon

ClearSunset

weather conditions

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

27

DAgger with Critical States and Replay Buffer

Key Ideas:
1. Sample critical states from the collected on-policy data based on theutility they provide to the learned policy in terms of driving behavior
2. Incorporate a replay buffer which progressively focuses on the highuncertainty regions of the policy’s state distribution
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

28

DAgger with Critical States and Replay Buffer

Sampling Strategies:
I Task-based: Sample uniformly from “left”, “right”, “straight”
I Policy-based: Use test-time dropout to estimate epistemic uncertainty
I Expert-based: Highest loss or deviation in brake signal wrt. expert
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

28

Distribution over Driving Actions
Imitation Learning
Brake

Steer Left

Go Straight

Our Approach Iter 1
Steer Right

Brake

Steer Left

10000050000

0ClearNoon

Steer Right

WetNoon

HardRainNoon

10000050000

0ClearSunset

ClearNoon

WetNoonweather conditions

Steer Left

Go Straight

HardRainNoon

ClearSunset

weather conditions

Our Approach Iter 2
Brake

Our Approach Iter 3
Steer Right

Brake

Steer Left

Go StraightSteer Right

150000# images in training data

150000# images in training data

Go Straight

150000# images in training data

# images in training data

15000010000050000

0ClearNoon

WetNoon

HardRainNoon

weather conditions

ClearSunset

10000050000

0ClearNoon

WetNoon

HardRainNoon

ClearSunset

weather conditions

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

29

Experiments

Evaluation

Empty

RegularDense

I CARLA NoCrash benchmark
I Dense setting with 220 agents
I Comparison to various baselines with (+) and without data augmentation
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

31

Evaluation

Training Conditions
80

ExpertDA-RB+(E)DA-RB+
DA-RB

Success Rate

60

DART
SMILe+
SMILe

40

DAgger+DAggerCILRS+20

CILRS

Iter 0Iter 1

Iter 2Iter 3

I Data augmentation increases the performance of all methods
I DAgger overﬁts quickly (!), not better than data augmentation
I Our model consistently improves upon the baselines in all conditions
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

32

Evaluation

New Weather
80

ExpertDA-RB+(E)DA-RB+
DA-RB

Success Rate

60

DART
SMILe+
SMILe

40

DAgger+DAggerCILRS+20

CILRS

Iter 0Iter 1

Iter 2Iter 3

I Data augmentation increases the performance of all methods
I DAgger overﬁts quickly (!), not better than data augmentation
I Our model consistently improves upon the baselines in all conditions
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

32

Evaluation

New Town
50

ExpertDA-RB+(E)DA-RB+

40
Success Rate

DA-RB
DART
30

SMILe+
SMILe
DAgger+20

DAggerCILRS+10

CILRS

Iter 0Iter 1

Iter 2Iter 3

I Data augmentation increases the performance of all methods
I DAgger overﬁts quickly (!), not better than data augmentation
I Our model consistently improves upon the baselines in all conditions
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

32

Evaluation
Task

CILRS+DART

Training
New Weather
New Town
New Town & Weather

45±6
39±4
23±1
26±2

50±1
37±2
26±2
21±1

DA-RB+(Ours)
62±1
60±1
34±2
25±1

DA-RB+ (E)(Ours)
66±5
56±1
36±3
35±2

Expert
71±4
72±3
41±2
43±2

Mean and standard deviation over 3 evaluation runs.

I Ensemble (E) improves performance further (particularly in new environment)
I Expert provided by CARLA often fails in dense environments (e.g., clogging)

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

33

Infractions Analysis
CILRS+

DA-RB+

DA-RB+(E)Expert40% Episodes

30

20

10

0
Collision with
Pedestrians

Collision with
VehiclesCollision with
Static Objects

Timed Out
Scenarios

I Signﬁciant reduction in collisions with dynamic objects
I More time-outs due to less infractions (e.g., clogged scenes, red lights)
Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

34

Training Variance

Iter 0Iter 1
Iter 2Iter 3CILRS+

DAgger+

DA-RB+

14.6 ± 3.4-

14.6 ± 3.4
15.2 ± 5.1
13.2 ± 1.9
17.8 ± 3.6

14.6 ± 3.4
24.8 ± 1.9
25.4 ± 1.5
27.0 ± 0.9

Standard deviation wrt. 5 random training seeds (New Town & Weather)I Signiﬁcant reduction in variance compared to CILRS and DAgger
I Sampling the dataset based on critical states is crucial for variance reduction

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

35

Interpretability: GradCAM Attention MapsCILRS [Codevilla et al. 2019]

Our Approach

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

36

Interpretability: GradCAM Attention MapsCILRS [Codevilla et al. 2019]

Our Approach

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

37

Qualitative Results

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

38

SummarySummaryI A single imitation learner cannot capture the complexities of driving
I Mixture of experts can signiﬁcantly improve generalization
I Task-driven optimization is difﬁcult but important
I Data augmentation is important but can easily overﬁt in self-driving
I Sampling critical states in a replay buffer improves aggregation performance
I Training variance can be reduced using this strategy
I Generalization to all CARLA weathers remains unsolved
I Better experts and improvements in the CARLA simulation are needed(and on their way – current version is CARLA 0.9.9!)

Prakash, Behl, Ohn-bar, Chitta and Geiger: Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving. CVPR, 2020.

40

Thank you!
http://autonomousvision.github.io