Driving with Attention
Andreas Geiger
Autonomous Vision Group
University of Tübingen and MPI for Intelligent Systems

Covered Papers

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Aditya Prakash, Kashyap Chitta and Andreas Geiger
CVPR 2021
NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Kashyap Chitta, Aditya Prakash and Andreas Geiger
ICCV 20212

Collaborators

Aditya PrakashKashyap Chitta

3TransFuser: Multi-Modal Fusion Transformer for
End-to-End Autonomous Driving

Motivation

Traﬃc

Traﬃc Lights

Ego-Vehicle
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

5Sensors
RGB Camera

LiDAR Point Cloud

Dense RGB input

3D information

Lacks reliable 3D information

Sparse input

Variation in weather

No trafﬁc light state

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

6

Research Questions

I How to integrate representations from multiple modalities?
I To what extent should the different modalities be processed independently?
I What kind of fusion mechanism to use for maximum performance?

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

7

Geometric Fusion
3DSpaceImageSpace
GeometricFeature
Projections

Z

X

Y

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

W

H

8

Geometric Fusion under-performs in complex urban scenarios

Collision
with vehicle

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

9

Geometric Fusion under-performs in complex urban scenarios

Collision
with cyclist

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

10

Geometric Fusion lacks global context

I For the yellow region, geometric fusion aggregates features from the blue region

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

11

Geometric Fusion lacks global context

I However, for safe navigation, it is essential to aggregate features from the redregion since it contains vehicles which are affected by the trafﬁc lightA. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

11

Key Idea

Use attention-based feature fusion
to capture the global context of thescene across modalities.

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

12

TransFuser Architecture
TransFuser
RGB
Image

64 x 64 x 64

32 x 32 x 128
Conv
+
Pool

Conv
+
Pool

Transformer

Conv
+
Pool

LiDAR
BEVTransformer

16 x 16 x 256

ResNet34
AvgPool+
FlattenTransformer

Conv
+
Pool

32 x 32 x 128

8 x 8 x 512
Conv
+
Pool

Conv+
Pool

Conv+Pool64 x 64 x 64

16 x 16 x 256

Transformer

Conv+
Pool

AvgPool+Flatten

8 x 8 x 512

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

ResNet18

13

Attention-based Feature Fusion

CH(2*H*W) x C

Self-AttentionW(2*H*W) x CTransformer

CHW

CCHxLHWW

I Consider feature maps as sets of tokens (cells of grid = tokens)
I Pass all tokens to self-attention module and reshape back into grid form
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

14Architecture
TransFuser
RGBImage

64 x 64 x 64

32 x 32 x 128
Conv
+
Pool

Conv
+
Pool

Transformer

16 x 16 x 256

Transformer

8 x 8 x 512
Conv
+
Pool

Conv
+
Pool

Transformer

ResNet34
AvgPool+
Flatten

MLP

Transformer

64
Conv
+
Pool

LiDAR
BEV

Conv
+Pool

64 x 64 x 64

Conv
+
Pool

32 x 32 x 128

16 x 16 x 256

Conv
+
Pool

AvgPool+
Flatten8 x 8 x 512

512!w1!w2!w3!w4GRU

GRU

GRU

GRU

w1w2

w3(0, 0)Goal Location

ResNet18

I GRU-based autoregressive waypoint prediction network (conditioned on goal)P
I Loss Function: L = 4t=1 ||wt − wtgt ||1 , waypoints are input to PID controller
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

15

Experiments
DatasetI 8 Towns and multiple weather conditions in CARLAI Routes of varying length in complex urban scenarios
I Expert policy based on A∗ planner and collision avoidance heuristics
Sensors
I RGB camera: 400×300 resolution, 100◦ FOV
I LiDAR: 80m range, 64 channels, 10 Hz rotation frequency

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

16

Results
Town05 Short
Town05 LongDS ↑RC ↑DS ↑RC ↑CILRS
7.47 ± 2.51 13.40 ± 1.09 3.68 ± 2.16 7.19 ± 2.95
Learning by Cheating 30.97 ± 4.17 55.01 ± 5.14 7.05 ± 2.13 32.09 ± 7.40
Waypoint Prediction 49.00 ± 6.83 81.07 ± 15.59 26.50 ± 4.82 60.66 ± 7.66
Late Fusion
51.56 ± 5.24 83.66 ± 11.04 31.30 ± 5.53 68.05 ± 5.39
Geometric Fusion 54.32 ± 4.85 86.91 ± 10.85 25.30 ± 4.08 69.17 ± 11.07
TransFuser
54.52 ± 4.29 78.41 ± 3.75 33.15 ± 4.04 56.36 ± 7.14
MethodMean and standard deviation over 9 runs (3 training seeds, 3 rollouts per seed)

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

17

Infraction Analysis
LBCAIM

Late Fusion

CollisionPedestrians

Collision
VehiclesGeometric FusionTransFuser

Collision
Layout

Red LightViolation

12840

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

18

CARLA Leaderboard

MethodDriving Score ↑Route Completion ↑CILRS
Learning by Cheating
Waypoint Prediction

5.37
8.94
12.88

14.40
17.54
41.52

Late Fusion
Geometric Fusion
TransFuser

13.27
14.47
16.93

42.10
40.99
51.82

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

19

Qualtitative Results

Generalization to New Town

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

20

Qualtitative Results

Generalization to New Weathers

A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

21

Attention Map Visualizations→Image: Trafﬁc Light

LiDAR: Vehicle

I Yellow: Query token in source modality. Green: Top-5 tokens in target modality.
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

22

Attention Map Visualizations→

LiDAR: Vehicles

Image: Trafﬁc Light and other Vehicles

I Yellow: Query token in source modality. Green: Top-5 tokens in target modality.
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

23

Attention Map Visualizations→

LiDAR: Vehicles

Image: Same Vehicles

I Yellow: Query token in source modality. Green: Top-5 tokens in target modality.
A. Prakash, K. Chitta and A. Geiger: Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. CVPR, 2021.

24

NEAT: Neural Attention Fields forEnd-to-End Autonomous Driving

Bird’s-Eye-View (BEV)
I Efﬁcient reasoning about
the semantic, spatial,
and temporal structureis crucial for self-driving
I Driving happens inphysical 3D space
I BEV ≈ 3D space
I How to obtain compact,
interpretable BEV repr.
from images as input?

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

26

BEV Semantics for Driving
Prior work:
I LBC [CoRL 2019]
I Roach [ICCV 2021]

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

27

BEV Semantics for Driving
Prior work:
I LBC [CoRL 2019]
I Roach [ICCV 2021]
NEAT:I Multi-task learning
I Single training stage
I Selective attention

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

27

Representation

I Implicit BEV semantics
and waypoints
I Arbitrary spatial andnoneroadobstacle
red light
green lightxyt

temporal resolution
I Small memory footprint(x’, y’)I Can make use of sparse
supervision signals(a) Scene BEV(x’, y’)(x’, y’)wT

wT+1

wT+2(b) t = T(c) t = T + 1(d) t = T + 2

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

28

Architecture: Encoder
EncodercResNet
ResNetTransformer

ResNetPositional Emb.

v

I A ResNet backbone yields 64 patches/tokens per image
I Patch features are combined with velocity and positional encoding
I A Transformer integrates contextual cues through self-attention

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

29

Architecture: NEAT and Decoder
Encoder
Neural Attention Field

Decodercic
ResNetResNetci-1
Transformer

ResNetPositional Emb.

v

xyt
x’y’

ai

xyt
x’y’

oi

six offset
y offset

L1
Lossnoneroadobstacle
red light
green lightCE
Loss
N iterations

I The encoder is not informed by the query (x, y, t) or target (x0 , y 0 ) location
I Instead, NEAT identiﬁes relevant patch features recurrently (dim(ci )  dim(c))
I The decoder predicts the semantic class and waypoint offset from cN
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

30

Architecture: Sampling and Control
Encoder
Neural Attention Field

cic
ResNetResNetci-1
Transformer

ResNetPositional Emb.

v

Sampling and Control
wT+1 ,wT+2… wT+Z

Decoderxytx’y’aixytx’y’oisix offsety offset
noneroadobstacle
red lightgreen light

Lateral PIDoNL1LossCE
Loss
N iterationsr

LongitudinalPIDsN

I At test time, we decode waypoints and a red light indicator from our model
I To extract both, we uniformly sample a sparse grid and query the decoder
I Grid pooling provides robustness at inference time; control via PID controllers

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

31

ExperimentsSensors
I 3 RGB cameras: (-60◦ , 0◦ and 60◦ )I No LiDAR
Evaluation
I 42 mixed-length routes (200-3000m) from 6 different towns
I 7 weather conditions (Clear, Cloudy, Wet, MidRain, WetCloudy, HardRain, SoftRain)I 6 daylight conditions (Night, Twilight, Dawn, Morning, Noon, Sunset)
I High density of dynamic agents and complex scenarios

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

32

Results
MethodAux. Sup.

DS ↑

CILRS
Learning by Cheating

Velocity
BEV Sem

22.97 ± 0.90
29.07 ± 0.67

Waypoint Prediction

None
2D SemBEV Sem

51.25 ± 0.17
57.95 ± 2.76
60.62 ± 2.33

NEATBEV Sem

65.10 ± 1.75

I CILRS / LBC use discrete / image-based target locations
I WP and NEAT use BEV-based target locations

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

33

CARLA Leaderboard
MethodDriving Score ↑WOR
MaRLnNEAT
Waypoint Prediction
TransFuser
Learning by Cheating
CILRS

31.37
24.98
21.83
19.38
16.93
8.94
5.37

I Sometimes too cautious and doesn’t complete route in time
I But safest driving and trained with 1-2 orders less data than WOR/MaRLn
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

34

Attention Visualizations

I White circle = query location (x, y)
I NEAT consistently attends to the region corresponding to the object of interest
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

35

Attention Visualizations

I White circle = query location (x, y)
I NEAT consistently attends to the region corresponding to the object of interest
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

35

Attention Visualizations

I White circle = query location (x, y)
I NEAT consistently attends to the region corresponding to the object of interest
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

35

Attention Visualizations

I White circle = query location (x, y)
I NEAT consistently attends to the region corresponding to the object of interest
K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

35

VideoK. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

36

Summary
Conclusions:
I Global contextual reasoning is crucial in complex urban scenarios
I Attention is effective in aggregating information from multiple modalities
I This allows for navigation of challenging intersection scenarios
I Joint BEV semantic prediction and trajectory planning leads to safer driving
I An implicit intermediate representation can efﬁciently solve this task
Code and Models:
I www.github.com/autonomousvision/transfuser
I www.github.com/autonomousvision/neat

K. Chitta, A. Prakash and A. Geiger: NEAT: Neural Attention Fields for End-to-End Autonomous Driving. ICCV, 2021.

37

Thank you!
http://autonomousvision.github.io