Supplementary Material for
KING: Generating Safety-Critical Driving Scenarios
for Robust Imitation via Kinematics Gradients
Niklas Hanselmann1,2,3 , Katrin Renz2,3 , Kashyap Chitta2,3 , Apratim
Bhattacharyya2,3 , and Andreas Geiger2,31

Mercedes-Benz AG R&D, Stuttgart
2
University of Tübingen
3
Max Planck Institute for Intelligent Systems, Tübingen

Abstract. In this supplementary document, we first provide additional information regarding AIM-BEV (Sec. 1.1) and the rule-based expert (Sec. 1.2). Next,
we provide additional information on the kinematics model (Sec. 2) and a detailed
description of our cost functions (Sec. 3). Furthermore, we discuss details regarding the maps, routes and traffic initialization of the scenarios in Sec. 4. Finally,
we present details on the hyperparameter choices for all optimization methods,
further results, as well as additional qualitative examples in Sec. 5.

1

Driving AgentsIn this work, we consider three driving agents: (1) Our AIM-VA [3] variant termed
AIM-BEV, (2) TransFuser [10] and (3) a privileged rule-based expert, which provides
supervision for AIM-BEV and TransFuser and is additionally used to determine solvability for the scenarios generated by KING. In the following we provide additional
details on AIM-BEV and the rule-based expert. For TransFuser, we refer readers to theoriginal paper. In addition, we provide further information on the driving metrics used
to benchmark their performance in the CARLA simulator.
1.1

AIM-BEV

Model: Our architecture is similar to the AIM models introduced in previous work
[3, 10]. For efficiency during simulation, we use a lightweight MobileNetV3-Small [8]instead of the ResNet34 used in AIM as a convolutional backbone. As described in themain paper, it takes a bird’s-eye view (BEV) semantic occupancy grid observation ot
as input. From this, a 512-dimensional feature vector is extracted (same as in AIM).
This is then further processed by three fully-connected layers (256, 128 and 64 units
with ReLU activations) to yield a 64-dimensional representation of ot . To be able totrain and evaluate the model in the CARLA simulator [5], we concatenate a binary
variable indicating if the vehicle should stop for a red light to this representation. As
we do not model traffic lights in KING scenarios, this variable is always set to zero2

N. Hanselmann et al.

Fig. 1: Architecture Overview. Illustration of the AIM-BEV architecture. First, a lowdimensional embedding from the input BEV semantic grid map observation ot is extracted. This is then used as the basis for autoregressive waypoint prediction, fromwhich the final actions are obtained via lateral and longitudinal PID controllers.

in our proxy simulation.4 Finally, the resulting feature is input to a single layer gatedrecurrent unit (GRU) with a linear decoder, which autoregressively estimates a set of
4four future waypoints {wi }i=1 representing the desired trajectory. In each timestep, the
GRU is additionally conditioned on a sparse goal location xgoal indicating the intended
high-level route. To convert the waypoints into low-level actions we use a set of two
PID controllers, which we describe next.
Controllers: We obtain actions for steering and acceleration from separate lateral andlongitudinal controllers (LatPID and LonPID in Algorithm 1). These are similar to thoseused in [2, 3, 9], with slight modifications. In particular, we replace hard clipping andthresholding operations with soft versions to maintain differentiability. For these, wetune the shape parameters βeb , βdecel and clipping bounds cδ , cthrottle to approximate
the original non-differentiable implementation. For steering, we obtain a control target
by computing the relative orientation Φd of the halfway point w∗ between the first andsecond waypoint. This is then input to the lateral PID controller (with gains Kp = 1.25,
Ki = 0.75, Kd = 0.3), which returns the corresponding steering action. Because thisis unbounded, we constrain it to the [−1, 1]-interval by applying the tanh function.
We have found a buffer of length 4 to be sufficient for computing the integral andderivative terms in both PID controllers. For throttle, we obtain the desired speed vd bycomputing the norm of the vector connecting the first and second waypoint. From this
we then compute a throttle value for the desired acceleration using the longitudinal PID
controller (Kp = 5, Ki = 0.5, Kd = 1), as well as a weighting factor α indicating if theagent should decelerate (throttle = -1). The latter is high either when the current desired
speed vd is lower than an emergency brake threshold τeb or when the current speed ishigher than vd . In our kinematics model, which we detail in Section 2, the state updateis then computed using the α-weighted sum of both the acceleration and deceleration
throttle values. As this effectively allows to express the entire [−1, 1] range of throttle
2
values, we denote the action space of the driving policy as a0t ∈ [−1, 1] in the main
paper for notational convenience.
4

Similarly, we use an additional channel in the BEV representation to encode pedestrians for
deployment in CARLA, which is inactive when optimizing for and evaluating on KING scenarios.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

3

Algorithm 1: Computing low-level actions from waypoints.
parameters: τeb ← 0.6, βeb ← 10, βdecel ← 1, cδ ← 0.25, cthrottle ← 0.75
input
: v, {wi }4i=1
output: steer ∈ [−1, 1], throttle ∈ (0, 1), α ∈ (0, 1)
w1 +w2
∗
1 w ←
;
2 

∗
−1 w [1]2 Φd ← tan
;
∗
w [0]
steer ← tanh (LatPID (Φd ));vd ← ∥w2 − w1 ∥ × 2;
5 αeb ← sigmoid (−βeb × (vd − τeb ));6 αdecel ← tanh (βdecel × ReLU (v − vd ));7 if vd < τeb then8α ← αeb
9 else10α ← αdecel
11 end12 δ ← cδ × tanh (βδ × ReLU (vd − v));
13 throttle ← cthrottle × tanh (ReLU (LonPID (δ)));3

4

Training on regular data: For initial training of AIM-BEV, we collect a dataset Dregof demonstrations by the rule-based expert described in Section 1.2 in regular traffic.
We sample a dense set of routes for the expert to complete, covering all possible junctions in Town01-Town06 available in the CARLA simulator as well as routes outsideof the junctions (e.g., highways). We include the hand-crafted CARLA scenarios during data collection to cover basic critical situations. In total, the dataset contains 91000
demonstrations, the equivalent of 12.6 h of driving. We obtain ground truth supervisionon the waypoints predicted by the driving policy from the future trajectory driven by theexpert during data collection. The goal location xgoal is obtained by the route plannerof [2], which is based on an A* planner. Using this, the sparse waypoint at least 7.5 m
ahead of the current position provided by CARLA is queried to obtain xgoal . To train
the driving policy using this dataset, we further employ data augmentation. For this,we rotate the BEV, waypoints and goal location around the vehicle center to simulatesituations where the policy has to recover to stable lane keeping. We augment 50% of
the samples with rotation angles sampled uniformly between ±20 degrees. The policy
is then trained using an L1-Loss on the waypoints for 350 epochs. We use the AdamW
optimizer with a learning rate of 0.001 and a batch size of 128.
Fine-tuning with safety-critical scenarios: Similar to Dreg , we build Dcrit by collecting demonstrations from the rule-based expert. The overall approach remains similar.
However now, instead of driving in CARLA under regular traffic, the expert solves scenarios generated by KING in our closed-loop proxy simulation. As described in themain paper, we first collect a larger set of KING scenarios that are solvable by theexpert, containing around 8000 training samples. When training on a union of bothdatasets Dcrit ∪ Dreg , we ensure a certain proportion of critical to regular data ineach minibatch. We find a mix of 60% regular and 40% critical data to work well in

4

N. Hanselmann et al.

practice. Aside from an increased batch size of 512 the training hyperparameters andaugmentations remain the same.

1.2

Rule-based Expert

We use a rule-based expert policy based on privileged information to generate training
data and determine solvability for the scenarios generated by KING. For the initialization of non-critical scenarios (see Section 4) we further adapt this expert to also workin our proxy simulation. We build on the code provided by [2, 3], which is based on the
structure of the CARLA traffic manager5 . It consists of three main components, (1) acollision stage, (2) a traffic light stage and (3) a motion planner stage, which we detail
next.
Collision stage: In order to predict possible future collisions we use a kinematic bicycle
model to extrapolate the position and orientation of all other agents within a radius of
30 m. For the extrapolation we use an action repeat assumption (i.e., the exact same
throttle and steer value is repeated for all extrapolation timesteps) since we do not have
any information about the future ground truth actions. For the ego agent, we extrapolate
assuming actions that maintain its current target speed. This allows reasoning about
the possibility of a collision and necessary braking maneuvers. Outside of junctions,
we extrapolate for a horizon of 1 s. In junctions we extrapolate for a longer horizon of
4 s due to a higher probability of collision. We express each future timestep through abounding box with the size of the respective vehicle and the extrapolated future positionand orientation. With these bounding boxes we then check for overlaps between the ego
and the other vehicles in each timestep to detect possible future collisions. In case of abounding box intersection we set a brake hazard flag. As an additional security measure
we extend the ego bounding box of the current timesteps to the front and check forintersections in this timestep. The extent is computed depending on the braking distancefor the current ego speed.
Traffic light stage: In the traffic light stage we detect traffic lights and stop signs. For
this we use privileged information provided by CARLA through trigger boxes. If there
is an intersection of the boxes and the internal CARLA state of the traffic light is red or
yellow we set the brake hazard flag.
Motion planner stage: For the main logic we refer to Section 2.2 of the supplementarymaterial of [3]. However, we observed a higher driving score when modifying the setof target speeds. We use a default target speed of 4 m/s. When entering an intersection
this is increased to 5 m/s. We observed that moving swiftly in intersections lowers theprobability of collision. When the brake hazard flag is set, the target speed is always
0 m/s. To adapt to the lower frame rate in our simulator (4 fps vs 20 fps), we use adjusted gains (Kp = 1.75, Ki = 0.75, Kd = 3.5) for the lateral controller (only in our
simulator). We leave the longitudinal controller unchanged.
5

https://carla.readthedocs.io/en/latest/adv traffic manager/

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

1.3

5

Driving Metrics for CARLA Benchmarking

In the main paper we benchmark the driving agents wrt. to their driving performance in
CARLA. This is measured by the Driving Score (DS), which is composed of two terms,
the Route Completion and the Infraction Score, which we detail in the following. The
Route Completion (RC) is given by
N

RC =

R
1 X
Ri
NR i

(1)

where Ri is the percentage of the total distance to be driven on route i that theagent has completed, and Nr is the total number of routes. This is complemented by
the Infraction Score (IS), which measures the agent’s compliance with traffic rules. It
is computed as a product of infraction penalties pji for each instance i of an infractionof type j. An overview of the type-specific penalties is shown in Table 1. In addition
to these, there is one further penalty for deviations from the drivable area. This is computed as (1−Rof f road ), where Rof f road denotes the percentage of the route completed
outside of the drivable area. Overall, the agent starts with a base IS of one, which is reduced for each infraction. The DS is then computed as the product of the RC and IS,
averaged over all routes.

Infraction Type

Penalty

Collisions with pedestrians
Collisions with other vehicles
Collisions with static elements
Running a red light
Running a stop sign

0.50
0.60
0.65
0.70
0.80

Table 1: Infraction Score (IS) penalties for different infraction types.

2

Kinematics Model

For the kinematic bicycle model we follow previous work [1] and use the author provided code and parameters, which were fitted to the CARLA dynamics and default
vehicle model. Their implementation constrains the speed to be positive via a hardthreshold to prevent backwards motion. This zeroes out gradients through the indirect
path in timesteps where the current throttle action would result in negative speeds. We
hence replace the hard threshold with a softplus activation. We set its shape parameterβ to 7, which we found to result in sufficiently low error.

3

Cost Functions

Here, we provide more details of the costs from Section 3 Eq. 2 of the main paper.
We begin with the ego-collision cost φegocol (S) followed by the two regularization terms
adv
φadv
(S)
and
φ
(S).
col
dev

6

N. Hanselmann et al.

Ego Collision Cost: The ego collision cost φegocol (S) is an attractive potential encouraging close encounters between the driving policy and the adversarial agents. This attractive potential is proportional to the distance between the ego agent and the adversarial
agents, for which the geometry of the ego agent and the adversarial agents is taken into
account. Let dbb (sit , sjt ) denote the Euclidean distance in R2 between closest points on
the bounding polygons of the ith and j th agents at time t. We minimize the Euclidean
distance between the ego agent and closest adversarial agent. We choose only the closest
adversarial agent in order to discourage situations where multiple adversaries deviate
from their trajectory to collide with the ego agent:Tφego
col (S) =1X
dbb (s0t , sit ).
i∈{1,...,N } T
t=0
min

(2)

The distance dbb (sit , sjt ) can be efficiently computed in parallel as the minimum of the
distances between every vertex-edge pair of the two agents’ bounding boxes.
Adversarial Collision Regularization: To improve the physical plausibility of thegenerated scenarios, we discourage collisions between adversarial agents through a repulsive potential φadv
col (S). This potential is inversely proportional to the distance between the closest pair of adversarial agents:


(3)
φadv
min
dbb (sit , sjt ), τ .
col (S) = − mini,j∈{1,...,N }, t∈{0,...,T }We find it sufficient to apply the repulsive potential φadv
col to the closest pair of adversarial agents. Finally, we apply a threshold τ to the distance, acting as a safety margin.
This ensures that the repulsive potential is active only in a local neighbourhood aroundeach adversarial agent j, preventing penalization of agents which are further apart than
this threshold. We find τ = 1.25 m to work well in practice.
Out-of-bounds Regularization: Finally, we regularize adversarial agents to stay within
drivable parts of the map. For this, we use a repulsive potential φadvdev (S) between theadversarial agents and off-road areas. To ensure that this is compliant with the geometry of the vehicle, the overall cost is composed of multiple terms defined as Gaussian
potentials g(Bit (ck )) at each corner ck of the bounding rectangle Bit corresponding to
the adversarial agent i. The potential at each corner is obtained through a convolution operation between the Gaussian potential g(Bit (ck )) and the map Moob , which isg(Bit (ck )) ⊛ Moob . We consider a binary map Moob which marks the non-drivable areas. The potential g(Bit (ck )) ⊛ Moob is thus large when the corner ck is near (or on) a
non-drivable area. The full “out-of-bounds” repulsive potential is defined as sum overall corners of the polygon Bit across agents and over all timesteps:N

φadv
rddev =

T

C

1 XXX
g(Bit (ck )) ⊛ M
T i=0 t=0

(4)

k=0

In practice, a variance of 1 m works well for the Gaussians. This ensures that the potential is large when any corner is ∼0.5 m from any non-drivable area.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

7

Cost Weighting: As the number of actors in the scene is always known ahead of optimization, we use separate cost weightings for each traffic density. This accommodates
for the fact that for different densities, a slightly different emphasis of the regularization
terms can be optimal. The used weightings can be found in Tab. 2.

# agents

γ (for φadv
dev )λ (for φadv
col )

1
2
4

20
23
20

0
5
3

Table 2: Cost weightings. Weighting of the costs for different traffic densities.

4

Scenarios

In this section we provide additional information on the maps and routes as well as the
traffic initialization scheme, which provide the basis for the dataset of initial non-criticalscenarios used throughout the main paper.

(a) Town03(b) Town04(c) Town05(d) Town06

Fig. 2: Town layouts. Each town has unique features, such as roundabouts (a) or highway junctions (b, d).

Maps and Routes: We use routes from towns 3-6 provided by the CARLA simulator
for optimization. The layouts of these towns are illustrated in Fig. 2. Each town has
unique features in terms of road topology, such as multi-lane highways, highway exits orroundabouts. From each of these towns we densely sample routes through junctions andintersections to obtain a set of 714 routes in total, each of which is a potential mission
for the ego agent. This forms a diverse basis for possible scenarios. For evaluationon the Town10 intersections, we sample additional routes using the same samplingmechanism from CARLA’s Town10, which we found to be the most challenging forthe driving agents. This set consists of 82 different routes. We use the maximum traffic
density for Town10 (156 background vehicles) and spawn one hand-crafted CARLA8

N. Hanselmann et al.

scenario6 along each route. Specifically, we consider Scenarios 7, 8, 9, and 10, which
involve multiple vehicles simultaneously entering the intersection.
Initializating Traffic: As a starting point for safety-critical scenario generation, we
require initial non-critical scenarios. Furthermore, to be able to study the effects ofdifferent traffic densities, the number of adversarial agents present in these initializations should be controllable. To this end, we mimic CARLA traffic by controlling theadversarial agents with the rule-based expert described in Section 1.2 to obtain an initialization. Specifically, we first sample a route that passes by the ego agent’s routefrom the dense set of 714 routes for every adversarial agent. For this we consider routes
which start and end at least 4 m and at most 100 m away from the ego agent’s route.
If possible, each agent is spawned on an individual route at the corresponding start location. If the set of candidate routes is smaller than the number of adversarial agents,
additional spawn locations further along already occupied routes are used for the remaining agents while making sure a distance of 6 m between spawn locations is kept.
Finally, using the expert, we roll out initial trajectories for the adversarial agents that
follow these routes while avoiding collisions with each other and the ego agent. While
this strategy lends itself well to study the effects of different traffic densities, KING isagnostic to the source of initial scenarios and other choices, for example a dataset ofrecorded traffic logs, may be more suitable in practice.
Pre-filtering: For our experiments we sample a subset of the available routes. Insteadof directly sampling from the total set of 714 routes, we first apply a pre-filtering step.
For this, we initialize a scenario on every route as described above. Then, we remove allroutes for which the scenario terminates due to an adversarial agent collision within thefirst 10 timesteps. This is done to handle edge cases not covered by the spawning logic.
After pre-filtering, we sample 20 routes and corresponding initial scenarios per town
from the remaining set to obtain the benchmark routes used to compare KING against
black-box optimization (BBO) baselines in Tab. 2 of the main paper. To build the larger
set used in the fine-tuning experiments for improving robustness, we use the entire setof routes remaining after pre-filtering.

5

Additional Results and Details

In this section we provide additional results supplementing the findings in the main paper. First, we detail the optimization hyperparameter choices for all methods used togenerate safety-critical scenarios, investigate the efficacy of using second-order optimizers for gradient-based generation and perform an ablation study on the regularization terms in the generation objective. Next, we assess the impact of their higher computational burden for KING with both the direct and indirect path and for Bandit-TD.
Then, we provide experiments on an additional simple baseline. Furthermore, we evaluate AIM-BEV on a benchmark of longer routes involving rural and highway driving.
Finally, we provide additional qualitative examples of safety-critical scenarios generated by KING.
6

https://leaderboard.carla.org/scenarios/

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

9

MethodHyperparameter

Sweep Range

N =1

N =2

Random Search

Perturbation Bounds[0.01, 0.3]

0.2

0.2

N =4
0.2

Bayesian Optimization

Exploration Kappa κ
Initial Random Points[0.1, 10]
[1, 50]

5
20

10
20

0.1
20

SimBA [6]Perturbation Bounds[0.001, 0.2]

0.05

0.05

0.05

CMA-ES [7]Initial Standard Deviation σ[0.1, 0.4]

0.2

0.1

0.4

Table 3: Hyperparameter choices for BBO baselines.

5.1

Optimization Hyperparameters

For gradient-based generation via KING, we use the Adam optimizer with a learning
rate of 5e − 3. We use a decay rate of β1 = 0.8 for all traffic densities, and β2 = 0.999
for a traffic density of N = 1 and β2 = 0.99 for N ∈ {2, 4}. We also adopt thisto optimize the scenario parameters via the numerical gradients obtained from BanditTD. For the experiment involving both the direct and the indirect path (cf. Tab 2 ofthe main paper), we observed exploding gradients. To combat this, we clip the normof the gradients through the rasterized BEV, where we found a maximum norm of
5e − 2 to work well. An overview of the remaining hyperparameters can be found in
Tab. 3. For Random Search and SimBA [6], we tune the bounds of the interval fromwhich perturbations can be sampled. For Bayesian Optimization, we search for boththe optimal exploration value κ and the number of initial random points. Finally, for
CMA-ES [7], we select an optimal initial standard deviation σ.
5.2

Second-Order Optimizers

Relative to the standards of modern deep learning, the parameter space is low-dimensional
and the computational graph is simple for KING. This suggests that utilizing secondorder gradients might be feasible for this problem, which we investigate in this experiment. On the example of Newton’s method, we find that computing the exact Hessianwith PyTorch’s autograd engine, even for our low-dimensional search space, leads to 12 orders of magnitude slower runtime compared to Adam (see Table 4). This is a resultof the additional backward passes needed to compute second-order derivatives. Thisoverhead does not permit a single optimization step within our computational budget
for the 4 agents setting. We also investigated quasi-Newton methods (L-BFGS), but did
not observe any benefits over Adam (Table 5). This is due to two factors: the runtime
penalty of requiring multiple function evaluations, each of which is a complete unrollof the simulation (we found a maximum of five to work best), and a susceptibility tolocal minima.
5.3

Cost Ablation Study

We study the influence of the regularization terms in our objective in Tab 6. While
the out-of-bounds term φadvdev (S) leads to a drop in performance when considering theregularization terms in isolation due to its computational expense, it is evident that

10

N. Hanselmann et al.
Method1 Agent

2 Agents

4 Agents

KING, 1st-order (cf. Tab. 2 of main paper)
KING, 2nd-order

1.78
81.31

1.88
156.712.03
316.89

Table 4: Timings for second-order gradients in s/it.

Optimizer

CR

t50%

s/it

Adam
L-BFGS

78.75
65.00

6.40
16.79

2.03
9.31

Table 5: Results on KING using L-BGFS in the 4 agent setting.

φego
col

φadv
dev

φadv
col

CR ↑

t50% ↓

s/it ↓

✓
✓
✓

✓
-

✓
-

66.25
57.50
60.00

6.94
10.21
7.08

1.59
2.03
1.59

✓

✓

✓

78.75

6.40

2.03

Table 6: Cost ablation. We show the performance for different regularization combinations for the 4 agent setting. Both regularization terms in combination with our main
objective results in the best performance.
combining both terms is crucial to achieve good results. Note that the losses for egoadversarial and adversarial-adversarial collisions are computed in parallel as a batched
operation, resulting in the same runtime when deactivating one of them.
5.4

Convergence

Bandit-TD and KING with both the direct and indirect gradient path are disadvantagedby their higher computational expense when imposing a fixed wall clock budget. We
are interested in assessing their performance in the absence of this. For this, we show
both the overall CR over GPU seconds for a less strict budget and the overall CR overoptimization iterations (Fig. 3). As can be seen, given a high enough budget, BanditTD can eventually partially close the gap to other BBO methods. For KING with both
paths, we observe that given the same number of optimization iterations, it achieves
similar results compared to the indirect path only. Note that the results for KING withboth paths stem from a preliminary experiment and thus perform slightly worse than
the final version reported in Tab. 2 of the main paper.
5.5

Additional Baseline: Pose Jittering

As both AIM-BEV and TransFuser do not take temporal inputs, simply jittering agent
poses for the single input frame is a reasonable simple baseline. However, even though
they do not require temporally consistent inputs, temporally consistent rollouts from thejittered frame need to be possible in order to obtain ground-truth future waypoints from

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

0.8

0.8

0.6

CR

0.6

CR

11

0.4

0.2

0.4

0.2

direct + indirect

direct + indirect

KING (ours)

KING (ours)

Bandit_TD

0.0
0

200

400

600

800

Bandit_TD

0.0
1000

0

50

GPUsec

100

150

200

250

Iter

Fig. 3: Results at higher compute budget. We compare KING to KING with both thedirect and indirect path, and to Bandit-TD for a larger compute budget (left) and interms of optimization iterations (right).
the expert. To gauge the effectiveness of jittering the pose to find safety-critical perturbations, while maintaining temporally consistent rollouts, we construct the following
experiment: We first find the time instant and adversarial agent closest to causing acollision in the initial scenario. We then jitter the pose of that agent in a ∆t seconds
earlier timestep by sampling perturbations uniformly from a pre-defined interval, while
making sure the perturbations do not trigger our termination conditions. We then continue rolling out the scenario from the jittered state for 2 seconds to be able to collectground-truth waypoints. The results for different ∆t and perturbation magnitudes areshown in Table 7. In addition to being applicable to only single frame agents, we observe this approach to be much less effective than KING at generating safety-critical
data. It struggles to find plausible perturbations, resulting in a significantly reduced CR.
Furthermore, we observe that among the obtained perturbations a much smaller percentage are solvable by the rule-based expert. In conjunction, this results in an overallsignificantly smaller volume of usable safety-critical data.

MethodInterval

∆t

CR

t50%

s/it

Solvable

Jitter

±1m / ±15°±1m / ±15°±2m / ±15°±2m / ±15°

-1s
-2s
-1s
-2s

42.50
43.75
61.25
55.00

9.04
7.84

1.10
1.10
1.10
1.10

64.71%
65.72%
53.06%
47.73%

–

–

78.75

6.40

2.03

76.25%

KING (Ours)

Table 7: Pose jittering in the 4 agent setting for AIM-BEV.

5.6

Evaluation on the Longest6 Benchmark

We also evaluate AIM-BEV before and after fine-tuning on Dcrit ∪ Dreg on theLongest6 benchmark [4]. It contains long routes, which in addition to urban settings,

12

N. Hanselmann et al.

also involve rural and highway driving similar to the NEAT validation benchmark [3],but at the maximum traffic density allowed by the CARLA simulator. Here, both theregular and fine-tuned version of AIM-BEV achieve driving scores of 35.38±1.55 and38.84±2.96 respectively. Because this benchmark places additional emphasis on otheraspects of the driving task not specifically targeted by KING, such as safe lane-changing
and -keeping on multilane roads over extended distances, we instead choose the dense
urban intersection setting in Town10 for evaluation in the main paper.
5.7

Additional Qualitative Examples and Failure Cases

Here, we show additional qualitative examples of safety-critical scenarios for both
AIM-BEV (Fig. 4-9) and TransFuser (Fig. 10). As can be seen, they cover diverse situations and highlight the brittleness of current imitation learning (IL)-based driving
agents.
Finally, we discuss failure cases of KING and possible strategies for mitigation. As
we aim to cover the long tail of the distribution of traffic scenarios, we impose few constraints on the trajectories of other traffic participants. The formulation of the objective
C(S) in Eq. (2) of the main paper only penalizes deviations from drivable areas, butdoes not enforce any further restrictions within those areas. While this allows flexibility in the discovery of diverse safety-critical scenarios, in some cases, the trajectories
of background agents that do not interact with the ego agent may be perturbed, causing
multiple vehicles to leave the trajectory they followed at initialization. This is illustrated
in Fig. 11. Nonetheless, as can be seen from the experiments in Sec. 4.4 of the main paper, these scenarios provide useful training data that lead to improved robustness.
These non-interactive perturbations could be mitigated by adding additional regularizing terms to the overall objective, e.g. a penalty for deviation from an assigned lane
center. However, using a more restrictive objective may lead to less safety-critical data
overall and reduced coverage of out-of-distribution situations.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

13

Fig. 4: AIM-BEV - Cluster (a). The driving agent collides with a slow moving vehicle while taking a turn (left). Additionally, it often struggles to handle lane-change
scenarios safely (right). This is also evident in cluster (c).

14

N. Hanselmann et al.

Fig. 5: AIM-BEV - Cluster (b). The driving agent understeers when taking a turn,colliding with another vehicle that has been slightly displaced from its lane center by thesafety-critical perturbation (left). Background traffic in CARLA usually closely adheresto the lane centers, making penalization of bad lane following less likely. On the righthand side, overconfident behavior in an unprotected intersection leads to collision.

Fig. 6: AIM-BEV - Cluster (c). The driving agent fails to handle merges and lanechanges in a roundabout safely. These maneuvers are especially problematic due to
AIM-BEV’s forward facing field of view. This shortcoming is not as apparent in regular
CARLA traffic, highlighting KING’s ability to draw attention to possible failure modes.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

15

Fig. 7: AIM-BEV - Cluster (d). This cluster primarily features collisions at the side ofthe driving agent. As depicted, this type of collision can arise from insufficient safety
distance while turning (left) or failure to yield for another vehicle taking an aggressive
turn (right).

16

N. Hanselmann et al.

Fig. 8: AIM-BEV - Cluster (e). The driving agent fails to brake when traffic in theintersection does not yield. As a result the agent collides into the side of another vehicle.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

17

Fig. 9: AIM-BEV - Cluster (f). The examples shown here depict similar situations as
in cluster (e). The ego agent fails to correctly assess if it is safe to continue on, resulting
in other traffic colliding into its side.

18

N. Hanselmann et al.

Fig. 10: Additional Qualitative Examples for TransFuser. We show additional examples of safety-critical perturbations for TransFuser. It shares many of the failure modesof AIM-BEV, such as difficulty handling merges (top), unsafe behaviour when moving
through intersections (middle) or the reliance on other vehicles strictly adhering to theirlane centers and stopping points at intersections (bottom). The common failure modesof AIM-BEV and TransFuser suggest that these problems indeed stem from insufficient
representation in the regular traffic data obtained from CARLA.

Generating Safety-Critical Driving Scenarios via Kinematics Gradients

19

Fig. 11: KING failure cases. As we aim to cover the long tail of the distribution of
traffic scenarios, we impose few constraints on the trajectories of other traffic participants. This can lead to unlikely behaviour in the background actors by definition, suchas moving into opposing lanes (left) or performing seemingly unnecessary lane changes(right).

20

N. Hanselmann et al.

References
1. Chen, D., Koltun, V., Krähenbühl, P.: Learning to drive from a world on rails. In: Proc. of theIEEE International Conf. on Computer Vision (ICCV) (2021) 5
2. Chen, D., Zhou, B., Koltun, V., Krähenbühl, P.: Learning by cheating. In: Proc. Conf. on
Robot Learning (CoRL) (2019) 2, 3, 4
3. Chitta, K., Prakash, A., Geiger, A.: Neat: Neural attention fields for end-to-end autonomous
driving. In: Proc. of the IEEE International Conf. on Computer Vision (ICCV) (2021) 1, 2,4, 12
4. Chitta, K., Prakash, A., Jaeger, B., Yu, Z., Renz, K., Geiger, A.: Transfuser: Imitation withtransformer-based sensor fusion for autonomous driving. In: arXiv.org (2022) 11
5. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proc. Conf. on Robot Learning (CoRL) (2017) 1
6. Guo, C., Gardner, J.R., You, Y., Wilson, A.G., Weinberger, K.Q.: Simple black-box adversarial attacks. In: Proc. of the International Conf. on Machine learning (ICML) (2019) 9
7. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 9(2), 159–195 (2001) 9
8. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang,
R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetv3. In: Proc. of the IEEE
International Conf. on Computer Vision (ICCV) (2019) 1
9. Prakash, A., Behl, A., Ohn-Bar, E., Chitta, K., Geiger, A.: Exploring data aggregation in policy learning for vision-based urban autonomous driving. In: Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR) (2020) 2
10. Prakash, A., Chitta, K., Geiger, A.: Multi-modal fusion transformer for end-to-end autonomous driving. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR) (2021) 1