Label Efﬁcient Visual Abstractions for Autonomous Driving
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems Tübingen

June 19, 2020

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupCollaborators

Aseem Behl

Aditya PrakashKashyap Chitta

Eshed Ohn-Bar

Andreas Geiger

2Approaches to Self-Driving
Steer

Sensory Input
Modular Pipeline
Low-level
Perception

SceneParsing

Path
Planning

VehicleControl

Gas+ Modular+ Interpretable- Expert decisions

Brake- Piece-wise training
Steer

Sensory Input
Imitation Learning / Reinforcement Learning
NeuralNetworkGas+ End-to-end+ Simple- Generalization- Interpretable- Data

Brake
3

Approaches to Self-Driving
Steer

Sensory Input
Direct Perception
NeuralNetwork

Intermediate
Representations

VehicleControlGas

Brake4

Approaches to Self-Driving
Steer

Sensory Input
Direct Perception
NeuralNetwork

Intermediate
Representations

VehicleControl

Gas

Brake

Which input modality?
I Images
I LidarI RadarI GPS4

Approaches to Self-Driving
Steer

Sensory Input
Direct Perception
NeuralNetwork

Intermediate
Representations

VehicleControlGas

Which input modality?

Brake

Which intermediate modality?

I Images

I Semantic segmentationI Lidar

I Bounding boxes

I RadarI Depth

I GPS

I Optical ﬂow
4Related Work

Conditional Imitation Learning
Conditional Imitation Learning:
I Direct mapping: Observations → actions
I Conditioned on command (“left”, “straight”, . . . )I Labeled training data obtained automatically
I Inertia problem ⇒ speed prediction
I Does not generalize well to new environments
I Large training variance(wrt. initialization, data sampling)

Codevilla, Santana, Lopez and Gaidon: Exploring the Limitations of Behavior Cloning for Autonomous Driving. ICCV, 2019.

6

Conditional Affordance Learning
Conditional Affordance Learning:
I Map: Observations → affordances → actions
I Affordances: angle wrt. road, distance to laneboundaries or other cars, etc.
I Decoupling of perception and action⇒Better generalization
I Rule-based controller
I Misspeciﬁcation of affordances

Sauer, Savinov and Geiger: Conditional Affordance Learning for Driving in Urban Environments. CoRL, 2018.

7

Does Computer Vision Matter for Action?

Does Computer Vision Matter for Action?
I Analyze various intermediate representations:segmentation, depth, normals, ﬂow, albedo
I Intermediate representations improve results
I Consistent gains across simulations / tasks
I Depth and semantic provide largest gains
I Better generalization performance

Zhou, Krähenbühl and Koltun: Does computer vision matter for action? Science Robotics, 2019.

8

Latent Space Distillation

I Minimizes distance between embedding of privileged teacher and student

Zhao, He, Liang, Huang, Van den Broeck and Soatto: LaTeS: Latent Space Distillation for Teacher-Student Driving Policy Learning. ARXIV, 2019.

9Related Findings
I Müller, Dosovitskiy, Ghanem and Koltun: Driving policy transfer via modularity andabstraction. In CoRL, 2018.
I Mousavian, Toshev, Fiser, Kosecka, Wahid and Davidson: Visual representations
for semantic target driven navigation. In ICRA, 2019.
I Sax, Emi, Zamir, Guibas, Savarese and Malik: Learning to navigate using mid-levelvisual priors. In CoRL, 2019.
I Wang, Devin, Cai, Yu and Darrell: Deep object centric policies for autonomous
driving. In ICRA, 2019.
But: So far no systematic study on label efﬁciency and representation granularity

10

Label-Efﬁcient Visual Abstractions

Visual Abstractions
What is a good visual abstraction?
I Invariant (hide irrelevant variations from policy)
I Universal (applicable to wide range of scenarios)

Pixel Space

Representation SpaceTrain

Test

I Data efﬁcient (in terms of memory/computation)
I Label efﬁcient (require little manual effort)

Figure Credit:
Alexander Sax

Semantic segmentation:
I Encodes task-relevant knowledge (e.g. road is drivable) and priors (e.g., grouping)
I Can be processed with standard 2D convolutional policy networks
Disadvantage:
I Labelling time: ∼90 min for 1 Cityscapes image12

Label Efﬁcient Visual Abstractions

Research Questions:
I What is the trade-off between annotation time and driving performance?
I Can selecting speciﬁc semantic classes ease policy learning?
I Are visual abstractions trained with few images competitive?
I Is ﬁne-grained annotation important?
I Are visual abstractions able to reduce training variance?

13

Label Efﬁcient Visual Abstractions

Model:I Visual abstraction network aψ : x 7→ s
I Control policy πθ : s, n, v 7→ c
I Composing both yields c = πθ (aφ (x))
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

14

Label Efﬁcient Visual Abstractions

Datasets:
sI ns images annotated with semantic labels S = {xi , si }ni=1c
I nc images annotated with expert driving controls C = {xi , ci }ni=1

I We assume ns  ncBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

14

Label Efﬁcient Visual Abstractions

Training:I Train visual abstraction network aφ (·) using semantic dataset ScI Apply this network to obtain control dataset Cφ = {aφ (xi ), ci }ni=1

I Train control policy πθ (·) using control dataset CφBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

14

Control Policy
Model:I CILRS [Codevilla et al., ICCV 2019]
Input:I Visual abstraction s
I Navigational command n
I Vehicle velocity v
Output:I Control ĉ and velocity v̂
Loss:
I L = ||c − ĉ||1 + λ ||v − v̂||1Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

15

Visual Abstractions

Privileged Segmentation (14 classes):
I Ground-truth semantic labels for 14 classes
I Upper bound for analysis
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

16

Visual Abstractions

Privileged Segmentation (6 classes):
I Ground-truth semantic labels for 2 stuff and 4 object classes
I Upper bound for analysis
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

16

Visual Abstractions

Inferred Segmentation (14 classes):
I Segmentation model trained on 14 classes
I ResNet and Feature Pyramid Network (FPN) with segmentation headBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

16

Visual Abstractions

Inferred Segmentation (6 classes):
I Segmentation model trained on 2 stuff and 4 object classes
I ResNet and Feature Pyramid Network (FPN) with segmentation headBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

16

Visual Abstractions

Hybrid Detection and Segmentation (6 classes):
I Segmentation model trained on 2 stuff classes: road, lane marking
I Object detection trained on 4 object classes: vehicle, pedestrian, trafﬁc light (r/g)
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

16

Experiments

Driving Task Evaluation

Training Town

Test Town

I CARLA 0.8.4 NoCrash benchmark
I Random start and end location
I Metric: Percentage of successfully completed episodes (success rate)
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

18

Trafﬁc Density

Empty

Dense

RegularI Difﬁculty varies with number of dynamic agents in the scene
I Empty: 0 Agents

Regular: 65 Agents

Dense: 220 Agents

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

19

Weathers

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

20

Identifying Most Relevant Classes (Privileged)
Empty

Regular

Dense

Overall

100%

100%

100%

100%

75%

75%

75%

75%

50%

50%

50%

50%25%

25%

25%

25%

0%

0%0%

TimeoutCollision

567

Number of Classes

14567

Number of Classes

14Success0%567

14

Number of Classes

567

14

Number of Classes

I 14 classes: road, lane marking, vehicle, pedestrian, green light, red light, sidewalk,building, fence, pole, vegetation, wall, trafﬁc sign, otherI 7 classes: road, lane marking, vehicle, pedestrian, green light, red light, sidewalk,building, fence, pole, vegetation, wall, trafﬁc sign, otherI 6 classes: road, lane marking, vehicle, pedestrian, green light , red light, sidewalk
I 5 classes: road, lane marking, vehicle, pedestrian, green light, red light
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

21

Identifying Most Relevant Classes (Privileged)

Empty

RegularDense

Overall

100%

100%100%100%

75%

75%

75%

75%

50%

50%

50%

50%

25%

25%

25%

25%

0%0%0%

Timeout
Collision

567

Number of Classes

14567

Number of Classes

14

Success0%
56714

5

Number of Classes

6714

Number of Classes

I Moving from 14 to 6 classes does not hurt driving performance (on contrary)
I Drastic performance drop when lane markings are removed

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

22

Identifying Most Relevant Classes (Privileged)

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

23

Identifying Most Relevant Classes (Inferred)
Regular837450862506Dense100100

14

Number of Classes

50Overall1007564767256250614

Number of Classes

Success Rate

75

Success Rate

Success Rate

10050250Standard1007530 2619 24614

Success Rate

Empty

Privileged755059675025610

Number of Classes

614

Number of Classes

I Small performance drop when using inferred segmentations
I 6-class representation consistently improves upon 14-class representation
I We use the 6-class representation for all following experiments

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

24

Number of Annotated Images
EmptyRegularDense

Overall

100%

100%

100%

100%75%75%75%

75%

50%

50%

50%

50%

25%25%

25%25%

0%

0%

0%

040

1600064

Number of Annotated Images

004000160064

Number of Annotated Images

040TimeoutCollision
Success0%001600

64

Number of Annotated Images

400001600

64

Number of Annotated Images

I No signiﬁcant differences between agents trained on 6400 or 1600 samples
I Slight drop when using 400 images
I Prior work exploiting semantics on CARLA uses millions of annotated images

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

25

Hybrid Representation
HybridStandard

100

Success Rate

75

8982
7067
5958

50
2525220pty

Em

larguReensDellera

Ov

I Performance of hybrid representation matches standard segmentation
I Annotation time (segmentation): ∼ 300 seconds per image and per class
I Annotation time (hybrid): ∼ 20 seconds per image and per class
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

26

Hybrid Representation

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

27

Variance Between Training Runs
Task

Seed 1

Seed 2

Seed 3

Empty
Regular
Dense
Overall

262401744262244230
425

Empty
Regular
Dense
Overall

76
64
2855

80
68
2256

82
72
1857

Seed 4
CILRS
48
32
4
28
Hybrid78723461

Seed 5

Mean ↑

Std ↓CV ↓

4640
1834

41.20
30.40
5.60
25.60

8.79
6.23
7.13
6.18

0.21
0.20
1.27
0.24

90
72
2261

81.20
69.60
24.80
58.00

5.40
3.57
6.26
2.82

0.06
0.05
0.25
0.04

I Our approach signiﬁcantly reduces standard deviation & coefﬁcient of variation
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

28

Comparison to State-of-the-Art
Task

CAL

CILRS

Empty
Regular
Dense

36±3
26±2
9±165±2
46±220±1

Empty
RegularDense

25±314±210±071±259±431±3

LaTeSStandard
Train Weather
92±191±2
74±2
77±1
29±3
27±7
Test Weather
83±195±1
68±7
75±6
29±2
29±5HybridExpert

87±182±141±196±0
91±041±2

79±171±132±596±0
92±045±2

I CAL: Conditional Affordance Learning [Sauer et al., CoRL 2018]
I CILRS: Conditional Imitation Learning [Codevilla et al., ICCV 2019]
I LaTeS: Latent Space Distillation [Zhao et al., Arxiv 2019]
Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

29

Qualitative Results

Behl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

30

SummarySummaryI Exploiting visual abstractions leads to more robust driving models
I Higher segmentation accuracy does not necessarily imply better driving
I Only few of the commonly used classes are relevant for driving task
I Lane marking class is critical for good performance
I Only few annotations are required (400 / 1600)
I Hybrid representations further reduce annotation costs at similar performance
I Box-based representations can improve performance on dynamic classes
I Training variance is high in behavior cloning and should always be reported
I Visual abstractions can signiﬁcantly lower training varianceBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

32

SummaryBehl, Chitta, Prakash, Ohn-Bar and Geiger: Label Efﬁcient Visual Abstractions for Autonomous Driving. ARXIV, 2020.

33

KITTI-360

KITTI-360

Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 2016.

35

KITTI-360
Sensors:
I Front-facing stereo camera
I 360◦ ﬁsheye cameras
I Velodyne HDL 64 laser scanner
I SICK pushbroom laser scanner
I IMU/GPS localization system
Features:
I Driving distance: 73.7 km

Frames: 4 × 83,000

I All frames accurately geolocalized (⇒ OpenStreetMap)
I Semantic label deﬁnition consistent with Cityscapes, 19 classes for evaluation
I Each instance assigned with a consistent instance ID across all frames
Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 2016.

36

3D Annotations

RGB

Bounding Box

Semantic

Instance

Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 2016.

37

2D Annotations

SemanticInstanceConﬁdence

Bounding Box

Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 2016.

38

Thank you!
http://autonomousvision.github.io