Supplementary Material for
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

1

Aditya Prakash∗1
Kashyap Chitta∗1,2
Andreas Geiger1,2
2
Max Planck Institute for Intelligent Systems, Tübingen
University of Tübingen{firstname.lastname}@tue.mpg.de

Abstract
In this supplementary document, we first provide additional information regarding our architecture (Sec. 1). We thendescribe our data generation pipeline and dataset statistics (Sec. 2). Further, we discuss details regarding the baselines(Sec. 3) and the hyper-parameter choices for all the models (Sec. 4). Finally, we present additional results and analysis
to supplement our findings in the main paper (Sec. 5). The supplementary video contains qualitative comparisons of ourTransFuser model against Geometric Fusion on the evaluation routes.

1. Architecture Details
In this section, we provide additional details regarding the multi-scale feature fusion component of the TransFuser and the
low-level PID controller that converts waypoints to steer, throttle, and brake of the vehicle.

1.1. Multi-Scale Feature Fusion
The TransFuser takes in an RGB image and LiDAR BEV representation as inputs and processes them with ResNet-34 [8]and ResNet-18 [8] modules respectively. It uses several transformers for the fusion of intermediate feature maps between
both modalities. For an input of resolution 256 × 256 pixels, the ResNet module produces intermediate feature maps ofdimensions 64 × 64 × 64, 32 × 32 × 128, 16 × 16 × 256 and 8 × 8 × 512 after max pool operations in different ResNet blocks.
Since processing feature maps at high spatial resolutions is computationally expensive, we downsample higher resolution
feature maps from the early encoder blocks using average pooling to a fixed resolution of 8 × 8 before passing them as inputs
to the fusion transformer. This results in feature maps of dimensions 8 × 8 × 64, 8 × 8 × 128, 8 × 8 × 256 and 8 × 8 × 512.
The feature extractor branches of both the modalities provide feature maps with the dimensions described earlier. Since weset Df = 512 for each transformer, we use 1 × 1 convolutions to match the embedding dimension of each feature map to512.
Next, we describe the fusion mechanism between the RGB and LiDAR BEV feature maps of dimension 8 × 8 × 512.
The RGB and LiDAR BEV feature maps of dimension 8 × 8 × 512 are stacked together to form a tensor of dimension(2 ∗ 8 ∗ 8) × 512, where 2 represents RGB and LiDAR BEV modalities. This tensor is passed as input to the transformer
which outputs a feature of dimension (2 ∗ 8 ∗ 8) × 512. It is then reshaped into 2 tensors of dimension 8 × 8 × 512 eachfor the 2 modalities. Each of these 2 tensors is then upsampled using bilinear interpolation to match the original feature
resolution and processed through 1 × 1 convolutions to match the embedding dimension of the original feature maps. It isthen combined with the existing feature maps via element-wise summation. We pass the fused features back into the feature
extraction branches since this provides the deeper ResNet layers access to the global context of the 3D scene encoded in theearlier layers. This results in a 512-dimensional feature vector output from both the image and LiDAR BEV stream, whichis combined via element-wise summation. This 512-dimensional feature vector constitutes a compact representation of theenvironment which incorporates global contextual reasoning.
∗ indicates equal contribution

1

Algorithm 1: Generating an action from waypoints.
input : v, {wt }Tt=1 ;output: steer ∈ (−1, 1), throttle ∈ (0, 1), brake ∈ (0, 1);
γ=0
for t ← 1 to T − 1 do
γ += λt ||wt+1 − wt ||;2)
ω = (w1 +w
2 

α = tan−1

ω[1]
ω[0]



// velocity and waypoints// vehicle control// desired velocity;// aim direction

steer = LatPID(α)
if γ < βmin or γ < vβratio thenthrottle = 0
brake = 1
else
throttle = LonPID(γ − v)brake = 0

1.2. PID Controller
To convert waypoints to vehicle controls, we use two PID controllers. This process is summarized in Algorithm 1. We
use the same configuration as in the author-provided codebase of [2]. We first compute the T − 1 vectors between waypointsof consecutive time-steps, with T = 4 in our configuration. The longitudinal controller (LonPID in Algorithm 1) takes ina weighted average of these vectors and tries to match the vehicle velocity v to the desired velocity γ as much as possible.
We find the weights λ = {1, 0, 0}, for the T − 1 vectors, prioritizing the closest pair of waypoints, give the best empiricalresults. The lateral PID controller (LatPID in Algorithm 1) orients the vehicle along the aim direction α which is computed
as the orientation of the midpoint of w1 and w2 . Each controller has 3 gain parameters, which are tuned on a subset of ourtraining routes. For the longitudinal controller, we set Kp = 5.0, Ki = 0.5, Kd = 1.0 and for the lateral controller, we set
Kp = 1.25, Ki = 0.75, Kd = 0.3. Both controllers use a buffer of size 40 to approximate the integral term as a runningaverage. The other parameters used are a brake threshold speed βmin and brake speed ratio βratio , which we tune jointlywith the gain parameters, and set to 0.4 and 1.1 respectively.

2. Datasets
In this section, we describe our data generation pipeline with details about the sensor configuration, expert, routes, and scenarios used in the CARLA simulator. We additionally provide statistics regarding the distribution of navigational commands,
weather conditions, and dynamic agents in our dataset.

2.1. Sensor Configuration
We use the front camera with a field of view (FOV) of 100◦ for the RGB image input. The camera is mounted at a heightof 2.3m from the ground level of the ego-vehicle and located 1.3m in front of the centroid of the ego-vehicle. We offset thecameras forward by 1.3m to avoid the hood of the ego-vehicle from occluding a portion of the rendered images. We extract
images from this camera at a resolution of 400 × 300 pixels. For the LiDAR point cloud, we use a ray-cast-based Velodyne
64 LiDAR with an 85m range and rotation frequency of 10 FPS. The upper FOV of the LiDAR is set at 10◦ and the lower
FOV is set at -30◦ . The LiDAR sensor is mounted at a height of 2.5m from the ground level of the ego-vehicle and located1.3m in front of the ego-vehicle. We further use additional sensors such as IMU to get the orientation, GPS for localization,and speedometer to get the current speed of the ego-vehicle.

2.2. Expert
For generating training data, we roll out an expert policy designed to drive using privileged information from the simulator.
The expert policy consists of an A* planner followed by 2 PID controllers (for lateral and longitudinal control). Several
heuristics based on the global position of the dynamic agents and traffic lights are used by the expert to avoid collisions and2traffic violations. We build upon the code1 provided by the authors of [2]. We make two modifications to their code, (1)We change the distance and speed heuristics used for avoiding collisions and following traffic lights to make the expert morecautious, (2) We add stop sign avoidance functionality since it is penalized in the CARLA Leaderboard.
Given the route to be followed as a sequence of sparse waypoints, the expert policy uses the A* planner to interpolatethese sparse waypoints to dense waypoints with a sampling resolution of 1m. From these dense waypoints, it selects two
waypoints - the nearest waypoint after 4m from its current position as the near node and the nearest waypoint after 7m fromits current position as the far node. It uses the orientation of the near node as the aim direction, which is fed to the lateral
PID controller, with the steering value as its output. It uses the orientation of the far node as an indication of an approachingturn. The expert uses a discrete set of speed values, {0, 4, 7}m/s as the target speed for driving, depending on its current
circumstances. If the expert is in close proximity to pedestrians, vehicles, red light, or a stop sign, then the target speed is
set to 0 m/s. If the expert has to steer by more than 5◦ or is approaching a turn, as indicated by the orientation of the far
node, then the target speed is set to 4 m/s. Under every other circumstance, the target speed is set to 7 m/s. The longitudinal
controller then tries to match the speed of the expert policy to this target speed as much as possible. For the longitudinal
controller, we set Kp = 5.0, Ki = 0.5, Kd = 1.0 and for the lateral controller, we set Kp = 1.25, Ki = 0.75, Kd = 0.3.
Both controllers use a buffer of size 40 to approximate the integral term as a running average.

2.3. Routes
The CARLA Leaderboard repository provides a set of 76 routes as a starting point for training (50 routes) and evaluating(26 routes) the agent2 . These routes were originally released along with the 2019 CARLA challenge. These routes span
6 towns and each of them is defined by a sequence of waypoints. In addition to these, we also generate extra routes toincorporate more turnings and intersections in the training data using the procedure described below.
We first sample the intersections present in the town map. For this, we use the positions of the traffic lights since mostintersections contain traffic lights. For each such sampled intersection, we generate multiple pairs of start and end waypoints,
by sampling points along the canonical axes in a 100m × 100m grid centered at the intersection since road segments atintersections in CARLA are aligned with the canonical axes in world coordinates. From this set of sampled points, we selectpairs of points that lie on different road segments of the intersections. In this manner, we sample 8 pairs of turn routes at
each intersection. However, if any of these sampled points do not lie on a road segment, e.g. in 3-way intersections, thesimulator automatically maps the point lying off the road segment to the nearest point on a road segment. This results inseveral long routes encompassing multiple intersections. Next, we explicitly sample individual turns along these routes. For
this, we interpolate each of the routes, obtaining a dense set of waypoints along each. Then, we sample points along theseroutes where the navigational command changes from ’Follow Lane’ to ’Turn Left’, ’Turn Right’ or ’Go Straight’. These
points indicate the starting position of an intersection. When the navigation command changes back to ’Follow Lane’ along
the interpolated route, that indicates the end of an intersection. In this manner, we sample several pairs of start and end points
along each long route with each pair representing a single intersection of length 25-50m.
Adding these routes significantly improves the distribution of navigational commands in the training dataset (Fig. 1a) and
explicitly incorporates more turns and intersections. The total data collected using the original 76 routes contains around 70k
frames whereas the data collected using our routes contains around 150k frames.

2.4. Scenarios
The CARLA Leaderboard repository also provides a set of scenarios3 for each town. Each scenario is defined by (1) atrigger ’transform’ which indicates the spawn location and the orientation of that scenario in a particular town and (2) ’otheractors’ which contains information about additional agents present in that scenario. However, this provided set contains only
3 scenarios - control loss without previous action, obstacle avoidance without prior action, and obstacle avoidance with prioraction, whereas the Leaderboard evaluation consists of 10 scenarios, including situations such as other vehicles running red
lights and unprotected turns. Therefore, we also include additional scenarios while generating training data, so that our agent
is robust to these adversarial situations.
Specifically, we include Scenarios4 1, 3, 4, 7, 8, 9, 10 in our training dataset. Since we focus on safety-critical scenarios,
e.g. pedestrians emerging from occluded regions to cross the road at random locations, vehicles running red lights, andunprotected turnings, we do not include Scenarios 5 and 6, which involve lane changing. We also do not include Scenario 2
1 https://github.com/bradyz/2020 CARLA challenge
2 https://github.com/carla-simulator/leaderboard/tree/stable/data
3 https://github.com/carla-simulator/leaderboard/blob/stable/data/all towns traffic scenarios public.json
4 https://leaderboard.carla.org/scenarios/

3Original Routes

Our Routes

Experiments Data

100

40

% of data75% of data

Leaderboard Data

50

50

25

30

20

10
0

Turn
Left

Turn
Right

Go
Straight

Follow
Lane

ChangeChangeLane Left Lane Right0

1

Navigational Command

2

3

4

5

6

7

8

9

10

11

12

13

14

Weather ID(a) Distribution of Navigational Commands. We report the distribution of navigational commands for the original routes provided in the Leaderboard repository and our routes used for generating data. Our routes incorporate more turns and intersections in
the training data.

(b) Distribution of Weathers. 1: ClearNoon, 2: ClearSunset, 3:
CloudyNoon, 4: CloudySunset, 5: WetNoon, 6: WetSunset, 7:
MidRainyNoon, 8: MidRainSunset, 9: WetCloudyNoon, 10: WetCloudySunset, 11: HardRainNoon, 12: HardRainSunset, 13: SoftRainNoon, 14: SoftRainSunset

since it occurs naturally during the driving in the presence of other dynamic agents. To generate trigger transforms for these
additional scenarios, we consider the trigger transforms provided in the Leaderboard set of scenarios and sample additional
spawn locations at a 5m distance along the canonical axes in a 10m × 10m grid centered at each available trigger position. For
the spawn orientation, we select yaw ∈ {0◦ , 90◦ , 180◦ , 270◦ }. Therefore, for each provided trigger transform in a particular
town, we generate 36 trigger transforms for each scenario. While the CARLA leaderboard framework5 considers prioritizedsampling of scenarios (scenarios with higher ID are given preference, e.g. scenario 10 is prioritized over scenario 9 and so on)
in case multiple scenarios can be spawned at a particular trigger location, we consider uniform sampling (randomly selecting
one scenario per trigger location) in our implementation.

2.5. Data Statistics
We use 7 towns (Fig. 2) and 14 weather conditions (Fig. 1b) for generating training data. We use the set of routes andscenarios generated by the process described in Sec. 2.3 and Sec. 2.4 and store data at 2 FPS. We sequentially vary the weather
condition after every 30 seconds in each route. However, since a significant proportion of the routes used for generating data
are short, the weather distribution is skewed towards the initial weathers (Experiments data in Fig. 1b) since all the 14 kinds ofweather are not iterated through. This is not a problem for our internal evaluation since we fix the environmental conditions to’ClearNoon’ during evaluation, to decouple the main new challenge (scenarios) from factors related to visual generalization.
However, for submission to the CARLA Leaderboard (Sec. 5), we regenerate our data while varying the weather condition
after every 0.5 seconds of driving so that the distribution is uniform across all weathers (Leaderboard Data in Fig. 1b). The
number of dynamic agents in each town is set as - Town01: 120, Town02: 100, Town03: 120, Town04: 200, Town05: 120,
Town06: 150, Town07: 110, Town10: 120.

3. Baselines
In this section, we describe in detail the baselines (Tab. 1) used in our experiments - CILRS [5], LBC [2], Auto-regressive
IMage based waypoint prediction (AIM), Late Fusion and Geometric Fusion.

3.1. CILRSCILRS [5] is a conditional imitation learning method in which the agent learns to predict vehicle controls from a single
front camera image while being conditioned on the navigational command. The front camera image is processed by a
ResNet-34 encoder which outputs a 512-dimensional feature vector. In addition, CILRS also takes in the current speed v̂ as
input which is processed by a 1-layer MLP consisting of 256 units to produce a 512-dimensional feature vector, which is
combined with the feature vector of the image branch via element-wise summation. While the original architecture combined
the velocity vector and the image feature vector via concatenation followed by an MLP, we found element-wise summation
5 https://github.com/carla-simulator/leaderboard

4

(a) Town01(b) Town02(c) Town03(d) Town04(e) Town05(f) Town06(g) Town07(h) Town10

Figure 2: CARLA Towns. Town01, Town02, Town07 are EU style towns whereas the others are US style.

to work better empirically. This 512-dimensional vector is then fed to the controller module which is conditioned on discrete
navigational commands. While the original architecture used 4 navigational commands (follow lane, turn left / turn right / gostraight at intersection), we adapted it to include 2 additional commands for lane changes (left and right). Since the space ofnavigational commands is discrete, CILRS uses a conditional module to select one of several command branches based on
the input command. The command branch directly outputs control values (steer, throttle, and brake).
Additionally, the output of the ResNet-34 image encoder in CILRS is used for predicting the current vehicle speed v,
which is compared to the actual vehicle speed v̂ with the velocity loss Lvelocity (Eq. 1). The final loss function for CILRS isa weighted sum of an L1 imitation loss (Eq. 1), between the predicted control a and the ground-truth expert control a∗ , andthe velocity loss, with a scalar weight ζ. We set ζ = 0.05 since it gives the best empirical performance.
Limitiation = ||a − a∗ ||1

Lvelocity = ||v − v̂||1

L = Limitiation + ζLvelocity

(1)
(2)

3.2. LBC
LBC is a knowledge distillation approach where a teacher model with access to ground truth BEV semantic maps isfirst trained using expert supervision to predict waypoints, followed by an image-based student model which is trainedusing supervision from the teacher. While the original LBC architecture uses only the front camera image as input and isconditioned on discrete navigational commands (similar to CILRS), the authors recently released an updated version of theirarchitecture6 with 2 major modifications, (1) multi-view camera inputs to a ResNet-50 encoder (front, 45◦ left, and 45◦ rightimages are stacked as channels), and (2) target heatmap as input (instead of the navigational command) which is formedby projecting the target point into front camera image coordinates. We directly use this codebase in our experiments, with
one minor adaptation: the official implementation did not use the stop sign semantic class for the teacher model, which we
include since this infraction is penalized during evaluation.

3.3. AIM
We implement our auto-regressive waypoint prediction network with an image-based ResNet-34 encoder, initialized from
ImageNet [6] pre-trained weights, which takes just the front camera image as input. This baseline is equivalent to adapting
the CILRS model to predict waypoints conditioned on sparse goal locations rather than vehicle controls conditioned on
navigational commands. The image encoder used for this is the same as CILRS and our TransFuser model.
6 https://github.com/bradyz/2020 CARLA challenge

5

ResNetMLP
Predicted Speed

RGB Image
Conditional
Module
MLP

512

VehicleControls

Measured Speed
Navigational
Commands(a) CILRS(b) LBCRGB Image

ResNet34

MLP

ResNet64

RGB Image

𝛿w1

𝛿w2

𝛿w3

𝛿w4

GRU

GRU

GRU

GRU

w1

w2

w3

(0, 0)

MLP

𝛿w2

𝛿w3

𝛿w4

GRU

GRU

GRU

w1

w2

w3

(0, 0)

64

Goal Location

512

Goal Location

512

𝛿w1
GRU

ResNet18

LiDAR BEV(c) AIM
RGB
Image

64 x 64 x 64(d) Late Fusion
32 x 32 x 128

Conv
+
Pool

Conv
+
Pool

16 x 16 x 256

8 x 8 x 512
Conv
+
Pool

Conv
+
Pool

ResNet34
AvgPool+
Flatten𝛿w1

Geometric
FeatureProjections

Geometric
FeatureProjections

Geometric
FeatureProjections

Geometric
FeatureProjections

MLP
64
512

Conv
+
Pool

LiDAR
BEV

Conv
+
Pool

64 x 64 x 64

Conv
+
Pool

32 x 32 x 128

Conv
+
Pool

16 x 16 x 256

𝛿w2

𝛿w3

𝛿w4

GRU

GRU

GRU

GRU

(0, 0)

w1

w2

w3

Goal Location

AvgPool+
Flatten8 x 8 x 512

ResNet18(e) Geometric Fusion
Table 1: Baselines. Illustration of baseline methods used in our experiments.

3.4. Late Fusion
We implement a version of our architecture where the RGB image and the LiDAR BEV streams are processed independently of each other. The image features are extracted using ResNet-34, which is initialized from ImageNet pre-trained
weights, whereas the LiDAR BEV features are extracted using ResNet-18, which is trained from scratch. These features arethen combined via element-wise summation and passed to the waypoint prediction network. This architecture is equivalent
to removing the transformer modules from the TransFuser. This is similar to the encoder of [17] which also processed theimage feature and LiDAR features separately. While Sobh et al. [17] used a learned controller module conditioned on discrete
navigational commands (similar to CILRS), we use our auto-regressive waypoint prediction network since it gives a better
empirical performance.

3.5. Geometric Fusion
We implement a multi-scale geometry-based fusion method, inspired by [11, 12], involving both image-to-LiDAR and
LiDAR-to-image feature fusion. We unproject each 0.125m × 0.125m block in our LiDAR BEV representation into 3D
space resulting in a 3D volume. We randomly select 5 points from the LiDAR point cloud lying in this 3D volume and6project them into the image space. We aggregate the image features of these points via element-wise summation before
passing to a 3-layer MLP, consisting of 512 units each. The output of the MLP is then combined with the LiDAR BEV
feature of the corresponding 0.125m × 0.125m block at multiple resolutions throughout the feature extractor.
Similarly, for each image pixel, we randomly select 5 points in the LiDAR point cloud which project to that pixel in image
space, then project these points into the BEV space, aggregate their features via element-wise summation before passing to a
3-layer MLP, consisting of 512 units each. The output of the MLP is then combined with the features of the corresponding
image pixels at multiple resolutions throughout the feature extractor.
Similar to the TransFuser, we downsample the higher spatial resolution feature maps to a fixed resolution of 8 × 8 anduse 1 × 1 convolutions to match the embedding dimension of the feature maps to 512. After the features are combined, they
are upsampled to the original resolution using bilinear interpolation and processed through 1 × 1 convolutions to match theembedding dimension of the original feature map, before being fed into the individual feature extractors of the two modalities.
This baseline is equivalent to replacing the transformers in our architecture with geometry-based feature projections.

4. Implementation Details
In this section, we discuss the training hyper-parameter choices and schedules used for all the baselines and our TransFuser.
We perform single-GPU training for all models on GTX 1080Ti GPU. We train the LBC model using the codebase provided
by the authors, which uses the Adam [9] optimizer with a learning rate of 10−4 and weight decay of 0.0. The model isvalidated after every epoch, using a separate partition of our dataset, and the learning rate is reduced by a factor of 2 if thevalidation loss does not improve for 5 epochs. We train the model for a maximum of 100 epochs and terminate the training if
the learning rate goes below 10−6 or if the validation loss has not improved for 15 epochs. For all the other models, we use
the AdamW optimizer [13], which is a variant of Adam. The learning rate is set to 10−4 , weight decay to 0.01, and Adambeta values to the PyTorch defaults of 0.9 and 0.999. All the models are trained for a maximum of 100 epochs. The model isvalidated after every 5 epochs and we use early stopping to terminate the training if the validation loss has not improved for
25 epochs. We use the checkpoint with the best validation loss for the online evaluation of all the models.

5. Experiments
In this section, we describe the official CARLA Leaderboard metrics7 in detail and present the results of our submission
to the CARLA Leaderboard. We then provide additional results and analysis to supplement our findings in the main paper.

5.1. Metrics(1) Route Completion (RC): percentage of route distance completed, Ri by the agent in route i, averaged across N routes.
N

RC =

1 X
Ri .
N i

(3)

(2) Infraction Multiplier (IM): geometric series of infraction penalty coefficients, pj for every instance of infraction j
incurred by the agent during the route. Agents start with an ideal 1.0 base score, which is reduced by a penalty coefficient forevery infraction.
ped,....,stopYIM =(pj )# infractionsj .
(4)jThe penalty coefficient for each infraction is pre-defined and set to 0.50 for collision with a pedestrian, 0.60 for collision witha vehicle, 0.65 for collision with static layout, 0.7 for red light violation, and 0.8 for stop sign violation. Besides these, if anagent drives off-road then the route completion is reduced by a multiplier (1- % off route).
(3) Driving Score (DS): weighted average of the route completion with infraction multiplier Pi
N

DS =

1 X
R i Pi .
N i(5)(4) Infractions per km: the infractions considered are collisions with pedestrians, vehicles, and static elements, running a
red light, running a stop sign, off-road infractions, route deviations, timeouts, and vehicle blocked.
7 https://leaderboard.carla.org

7

Method
CILRS [5]
LBC [2]
AIM
Late Fusion
Geometric Fusion
TransFuser (Ours)
CIL-WPNEAT
MaRLn [18]Auxiliary Supervision
Velocity
BEV Semantics
None
None
None
None
2D Semantics + Depth
BEV Semantics
2D Semantics + Affordances

Sensors
1 camera
3 cameras
1 camera
1 camera + LiDAR
1 camera + LiDAR
1 camera + LiDAR
1 camera
3 cameras
1 camera

DS
5.37
8.94
12.88
13.27
14.47
16.93
19.38
21.83
24.98

RC
14.40
17.54
41.52
42.10
40.99
51.82
67.02
41.71
46.97

IM
0.55
0.73
0.59
0.54
0.48
0.42
0.39
0.65
0.52

Table 2: CARLA Leaderboard Evaluation. We report the mean route completion and driving score over 100 secret evaluation routes on the official evaluation server, comparing to the publicly visible submissions.

5.2. CARLA Leaderboard Evaluation
We submit all the models from our study to the CARLA Autonomous Driving Leaderboard which contains a secret set of
100 evaluation routes and report the results in Tab. 2. Among the models that do not use auxiliary supervision, TransFuser
achieves the highest DS, outperforming Geometric Fusion by 17%. CIL-WP has a similar architecture to AIM and uses 2D
semantics and depth as auxiliary supervision. NEAT employs an implicit function to map locations in BEV space to input
image features using attention along with BEV semantic prediction as an auxiliary task. Incorporating auxiliary supervision
as in these approaches has shown success in recent work on Imitation Learning, and is likely to also improve the performanceof TransFuser [1, 10].
The top Leaderboard entry, MaRLn, builds on top of the Reinforcement Learning (RL) method presented in [18]. In thisapproach, an encoder is first trained to predict both the 2D semantics and specific affordances such as the scene traffic lightstate, and the relative position and orientation between the vehicle and lane. The encoder is then frozen and used to train
a value function-based RL method. The authors of MaRLn indicated that training a model on a single town required 20days of simulation time [18]. In comparison, our training dataset for all CARLA towns only requires 21 hours of simulation
time and has the potential for further improvements through orthogonal techniques such as Active Learning [3, 4, 7] andDAgger [15, 16]. Further, it is possible to perform RL-based fine-tuning of imitation learning policies like ours, which iscomputationally expensive but known to be beneficial for performance [14].

5.3. Additional Results
In Table 1(a) of the main paper, we report the mean and standard deviation over 9 evaluation runs of each method in
Town05 Short and Town05 Long evaluation settings consisting of scenarios. In Tab. 3, we also provide results on Town05
Short and Town05 Long evaluation settings without any scenarios. We observe that in the absence of scenarios, Late Fusion
performs slightly better compared to TransFuser but in the presence of scenarios, TransFuser achieves a higher driving score.
We also provide detailed results (Tab. 4) for 1 run of each model on all the metrics described in Sec. 5.1 on Town05 Longevaluation setting. We observe that our TransFuser model consistently performs well among all the models and across all themetrics. Compared to the Geometric Fusion approach, TransFuser achieves a better DS while sustaining fewer infractions,as evident by significantly higher IM. Specifically, it outperforms Geometric Fusion by 37% on DS and 24% on IM in the No
Scenarios setting and by 22% on DS and 78% on IM in the Scenarios setting.

5.4. Attention Map Visualizations
The transformer takes in 64 image feature tokens and 64 LiDAR feature tokens as input where each token corresponds toa 32 × 32 patch in the input modality. We consider 1000 frames from Town05 intersections and examine the top-5 attention
weights for source tokens in image and LiDAR input. We provide visualization of 16 such frames in Fig. 3. We observe that
TransFuser is able to map (1) traffic lights in the image input to vehicles in LiDAR BEV, (2) vehicles in LiDAR BEV to samevehicles in image input, (3) vehicles in LiDAR BEV to different vehicles and traffic lights in image input. While TransFuser
attends to the objects of interest at intersections, we also observe cases where the dependency between the source entity andthe attended entity is inaccurate, e.g. traffic light attends to vehicles in the wrong direction.
8

Method

Town05 Short
Town05 LongNo Scenarios
Scenarios
No Scenarios
Scenarios
DS ↑
RC ↑
DS ↑
RC ↑
DS ↑
RC ↑
DS ↑
RC ↑
CILRS [5] 9.09 ± 2.07 14.09 ± 0.10 7.47 ± 2.51 13.40 ± 1.09 4.75 ± 3.68 8.34 ± 3.55 3.68 ± 2.16 7.19 ± 2.95
LBC [2] 47.95 ± 6.54 72.59 ± 5.00 30.97 ± 4.17 55.01 ± 5.14 11.98 ± 2.70 58.25 ± 9.00 7.05 ± 2.13 32.09 ± 7.40
AIM
64.46 ± 11.34 82.31 ± 13.73 49.00 ± 6.83 81.07 ± 15.59 33.20 ± 6.54 73.77 ± 12.20 26.50 ± 4.82 60.66 ± 7.66
76.27 ± 9.00 87.12 ± 7.78 51.56 ± 5.24 83.66 ± 11.04 44.87 ± 5.39 89.52 ± 9.55 31.30 ± 5.53 68.05 ± 5.39
LF
GF
67.36 ± 7.38 91.80 ± 6.72 54.32 ± 4.85 86.91 ± 10.85 30.92 ± 4.02 78.06 ± 10.10 25.30 ± 4.08 69.17 ± 11.07
TF (Ours) 74.64 ± 3.69 87.63 ± 4.16 54.52 ± 4.29 78.41 ± 3.75 43.68 ± 2.40 79.64 ± 8.51 33.15 ± 4.04 56.36 ± 7.14
Expert99.86 ± 0.02 99.86 ± 0.02 84.67 ± 6.21 98.59 ± 2.17 72.45 ± 8.48 100.00 ± 0.00 38.60 ± 4.00 77.47 ± 1.86

Table 3: Driving Performance. We report the mean and standard deviation over 9 runs of each method (3 training seeds,
each trained model evaluated 3 times) on Route Completion (RC) and Driving Score (DS) in Town05 Short and Town05
Long settings with scenario and without scenarios. LF: Late Fusion, GF: Geometric Fusion, TF: TransFuser
Method

DS ↑

RC ↑

IM ↑

Ped ↓

Veh ↓

CILRS [5]
LBC [2]
AIM
Late Fusion
Geometric Fusion
TransFuser (Ours)

6.96
9.16
30.24
39.49
33.32
41.69

10.96
56.52
65.19
85.75
86.38
82.75

0.64
0.21
0.49
0.46
0.45
0.59

0
0
0
0
0
0

2.26
0.17
0.10
0.04
0.03
0.03

CILRS [5]
LBC [2]
AIM
Late Fusion
Geometric Fusion
TransFuser (Ours)

4.84
6.17
30.83
34.96
29.17
39.66

9.66
24.59
52.96
60.33
81.09
66.19

0.65
0.48
0.71
0.70
0.37
0.63

0
0
0
0.02
0.02
0.01

0.33
0.23
0.08
0.06
0.18
0.06

Static ↓ Red ↓ Stop ↓ Off-road ↓Town05 Long No Scenarios
1.97
0.06
0
1.39
0.42
0.52
0.03
0.10
0.66
0.06
0.03
0.40
0
0.10
0.15
0.02
0
0.17
0.12
0.07
0
0.09
0.03
0.02
Town05 Long Scenarios
2.07
0.15
0
2.11
0.32
0.37
0
0.15
0.02
0.01
0.02
0.02
0.02
0.03
0.01
0.12
0.17
0.07
0.03
0.06
0
0.14
0.02
0.11

Dev ↓

TO ↓

Blocked ↓

0.67
0.02
0
0
0
0

0
0.02
0.01
0.01
0
0.01

2.09
0.20
0.45
0.05
0.05
0.25

0.65
0
0
0
0
0

0
0
0
0.01
0.03
0

2.6
1.35
0.52
0.46
0.19
0.17

Table 4: Detailed Results. We report detailed results on all the metrics for 1 run of each model on the Town05 Longevaluation setting. DS: Driving Score, RC: Route Completion, IM: Infraction Multiplier, Ped: Collision with pedestrian,
Veh: Collision with vehicle, Static: Collision with static layout, Red: Red light violation, Stop: Stop sign infractions, Offroad: Off-road driving, Dev: Route deviation, TO: Timeout, Blocked: Vehicle Blocked

5.5. Ablation on Model Capacity
All the fusion methods in the main paper use ResNet-34 as the image encoder and ResNet-18 as the LiDAR BEV encoder.
In our preliminary experiments, we experimented with using different perception backbone for the Late Fusion model toinvestigate if increasing the model capacity or changing the architecture of the perception backbone affects the performanceof the model. We observed a relative decrease of (1) 20.14% when using ResNet34 for LiDAR (2) 30.77% when using
Inception v3 for image (3) 52.83% when using Inception v3 for both image and LiDAR. This indicates that simply adding
more layers and increasing the depth of the perception backbone hampers performance, potentially due to overfitting.

References[1] Aseem Behl, Kashyap Chitta, Aditya Prakash, Eshed Ohn-Bar, and Andreas Geiger. Label efficient visual abstractions for autonomous
driving. In Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS), 2020. 8[2] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Proc. Conf. on Robot Learning (CoRL),
2019. 2, 3, 4, 8, 9
[3] Kashyap Chitta, Jose M. Alvarez, Elmar Haussmann, and Clement Farabet. Training data subset search with ensemble active learning.
arXiv.org, 1905.12737, 2019. 8[4] Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski. Large-scale visual active learning with deep probabilistic ensembles.
arXiv.org, 1811.03575, 2018. 8

9Figure 3: Attention Maps. For the yellow query token, we show the top-5 attended tokens in green and highlight the presenceof vehicles in the LiDAR in red. TransFuser attends to the vehicles and traffic lights at intersections in the input modalities,
albeit at a slightly different location.

[5] Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous
driving. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 4, 8, 9
[6] Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-fei. Imagenet: A large-scale hierarchical image database. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2009. 5[7] Elmar Haussmann, Michele Fenzi, Kashyap Chitta, Jan Ivanecky, Hanson Xu, Donna Roy, Akshita Mittel, Nicolas Koumchatzky,
Clement Farabet, and Jose M. Alvarez. Scalable Active Learning for Object Detection. In Proc. IEEE Intelligent Vehicles Symposium(IV), 2020. 8[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2016. 1[9] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Learning
Representations (ICLR), 2015. 7[10] Zhihao Li, Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, and Shigeki Sugano. Rethinking self-driving: Multi-task knowledge
for better generalization and accident explanation ability. arXiv.org, abs/1809.11100, 2018. 8[11] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 6[12] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proc.
of the European Conf. on Computer Vision (ECCV), 2018. 6[13] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. of the International Conf. on Learning Representations (ICLR), 2019. 7[14] Eshed Ohn-Bar, Aditya Prakash, Aseem Behl, Kashyap Chitta, and Andreas Geiger. Learning situational driving. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 8[15] Aditya Prakash, Aseem Behl, Eshed Ohn-Bar, Kashyap Chitta, and Andreas Geiger. Exploring data aggregation in policy learningfor vision-based urban autonomous driving. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 8[16] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online
learning. In Conference on Artificial Intelligence and Statistics (AISTATS), 2011. 8[17] Ibrahim Sobh, Loay Amin, Sherif Abdelkarim, Khaled Elmadawy, M. Saeed, Omar Abdeltawab, M. Gamal, and Ahmad El Sallab.
End-to-end multi-modal sensors fusion system for urban automated driving. In Advances in Neural Information Processing Systems(NeurIPS) Workshops, 2018. 6

10

[18] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using
implicit affordances. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 8

11