CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning
Authors: Luke Rowe
Roger Girgis
Anthony Gosselin
Bruno Carrez
Florian Golemo
Felix Heide
Liam Paull
Christopher Pal
Venue: CoRL 2024 (2024)
Keywords: offline reinforcement learning
autonomous driving
simulation
TL;DR: We propose reactive and controllable agents in a driving simulator using offline reinforcement learning.

Abstract: Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through a physics-enhanced Nocturne simulator to generate a diverse offline reinforcement learning dataset, annotated with various reward terms. We then train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including adversarial behaviours. We demonstrate that CtRL-Sim can generate diverse and realistic safety-critical scenarios while providing fine-grained control over agent behaviours.

CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning
Luke R.we 21.2, Roger Gi,, Anthony Gosselin Brumo Carrez', Florian Golemo', Felix Heide4.6, Liam P.,5 Christopher Pal P,35. Mila, Université de Montréal, PPolytechiqqee Montréal, "Princeton University, SCIAAR AI Chair, Tore Robotics
http:://nontrearrroccccccss httprrooooiiccccccccreeeeeeeeeee
Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism necessitate costly iterative sampling procedures to control the generated be- haviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning (RI.) to efficiently generate reactive and controllable traffic agents. Specifically, we pro- cess real-world driving data through a enhanced Nocturne simulator to generate diverse offline RI. dataset, annotated with various rewards. With this dataset, train return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of wide range of driving behaviours beyond the scope of the initial dataset, including adversarial behaviours. We show that CtRL-Sim can generate realistic safety- critical scenarios while providing fine-grained grained control over agent behaviours.
Abstract:
Keywords: Autonomous Driving, Simulation, Offline Reinforcement Learning
1
Introduction
Recent advances in autonomous driving has enhanced their ability to safely navigate the complex- ities of urban driving [1]. Despite this progress, ensuring operational safety in long-tail scenarios, such as unexpected pedestrian behaviours and distracted driving. remains a significant barrier to widespread adoption. Simulation has emerged as a promising tool for efficiently validating the safety autonomous vehicles (AVs) in these long-tail scenarios. However, a core challenge in de- veloping simulator for AVs the need for other agents within the simulation to exhibit realistic and diverse behaviours that are reactive to the AV, while being easily controllable. The traditional approach for evaluating AVs in simulation involves fixing the behaviour of agents the behaviours exhibited in pre-recorded driving data. However, this testing approach does not allow the other agents to react to the AV, which yields unrealistic interactions between the AV and the other agents.
To address the issues inherent non-reactive log-replay testing, prior work has proposed rule-based methods [2. 3] to enable reactive agents. However, the behaviour of these rule-based agents often lacks diversity and is unrealistic. More recently, generative models learned from real-world data have been proposed to enhance the realism of simulated agent behaviours [4, 5, 6. 7, 8, 9]. While these methods produce more realistic behaviours, they are either not easily controllable [4, 5, 9] or require costly sampling procedures to control the agent behaviours [10, 8, 7, 11, 12].
Denotes equal contribution. Corresponding email: luke.rowe Iukerowe@@mil..quhees
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

In this paper, we propose CtRL-Sim to address these lim- itations of prior work. The CtRL-Sim framework uti- lizes return-conditioned offline reinforcement learning (RL) to enable reactive, closed-loop, controllable, and prob- abilistic behaviour simulation within physics-enhaneed Nocturne [13] environment. We process scenes from the Waymo Open Motion Dataset [14] through Nocturne to curate an offline RI. dataset for training that is annotated with reward terms such as "vehicle-veiicle collision" and "goal achieved".. We propose a return-conditioned multi- agent autoregressive Transformer architecture [15] within the CtRL-Sim framework imitate the driving behaviours in the curated dataset. We then leverage exponential tilt- ing of the predicted return distribution [16] simple yet effective mechanism to control the simulated agent be- haviours. While [16] exponentially tilts towards more op- timal outcomes for the task of reward-maximizing control, we instead propose to tilt either direction to provide con- trol over both good and bad simulated driving behaviours.
Goll
Figure/ CtRL-Sim allows for con- trollable agent behaviour from ex- isting datasets. This allows users to create interesting edge cases for test- ing and evaluating AV planners.
We show examples of how CtRL-Sim can be used to generate counterfactual scenes when expo- nentially tilting the different reward axes in Figure 1. For controllable generation, CtRL-Sim simply requires specifying a tilting coefficient along each reward axis, which circumvents the costly iterative sampling required by prior methods. CtRL-Sim scenarios are simulated within our physics- Nocturne environment. We summarize our main contributions: We propose CtRL.-Sim, which is, the best of our knowledge, the first framework applying return-ondditonedd offline RL for con- trollable and reactive behaviour simulation. Specifically, CtRL-Sim employs exponential tilting of factorized reward-to-ge to control different of agent behaviours. 2. We propose autoregres- sive multi-agent encoder-decoder Transformer architecture within the CtRL-Sim framework that is tailored for controllable behaviour simulation. 3. We extend the Nocturne simulator [13] with a Box2D physics engine, which facilitates realistic vehicle dynamics and collision interactions.
We demonstrate the effectiveness of CtRL-Sim producing controllable and realistic agent be- haviours compared to prior methods. We also show that finetuning model in Nocturne with simulated adversarial scenarios enhances control over adversarial behaviours. CtRL-Sim has the potential to serve as a useful framework for enhancing the safety and robustness of AV planner policies through simulation-based training and evaluation.
2 CtRL-Sim
In this section, we present the proposed CtRL-Sim framework for behaviour simulation. We first introduce CtRL-Sim in the single-agent setting, and subsequently show how it extends to the multi- agent setting, Given the state of an agent at timestep / and additional context (e.g., the road struc- ture. the agent's 's goal), the behaviour simulation model employs a driving policy π (ar/se. m, sG) and a forward transition model (s 18,,, a:) to control the agent in the scene. Note that a, the ac- tion, m the map context, and 8G is the prescribed goal state. Using the physics-extended Nocturne simulator, we have access to physically-rees interested in modelling the policy л (ag St, m, sG) such that we can both imitate the real distribution of driving behaviour and control the agent's behavior to generate long-tai tail counterfactual scenes.
forward transition model P. In this work, we are
2.1 Our Approach to Controllable Simulation via Offline RL
We consider the common offline RL setup where
given dataset D of trajectories 7; {..., St, Ge, ... with states 8, S, actions a, E A and rewards Tt- These trajectories generated using (suboptimal) behaviour policy TB(a,)s) executed in finite-horizon Markov

---

agent n
(G, a, S
Full Altention Encoder
Temporaliy- Causal Decoder
Tilting
Encoder
Encoder
Embed
Decoder
state
action
states
agent info at
map dojects
Simecee O
Imcstop
(a) Model Overview
(b) Inference Time Forward Pass
Figure 2: (left) The agent and map data at = 0 are encoded and fed through Transformer encoder as context for the decoder, similar to [9]. Trajectories are arranged first by agents, then by timesteps, embedded, and fed through the decoder. For each agent, we encode (st. Gt, az) (i.e. state, return-togoo, action) and predict from these (G,, dg St+1s...: ST). 2b (right) At inference time, the state predicts the return-to-go The return-to-go is tilted (i.c., reweighed to encourage specific behaviors) and is used to predict the action, which in turn is used to predict the next states.
decision process. The return-to-pe at timestep t is defined as the cumulative sum of scalar rewards obtained in the trajectory from timestep I, G, = Σi= Te. The objective of offline RL to learn policies that perform as well as or better than the best agent behaviours observed in D.
The primary insight of this work is the observation that offline can be an effective way to perform controllable simulation. That is, the policy distribution over actions be tilted at inference time towards desirable or undesirable behaviors by specifying different values of returnto-go Gs. This re- quires different formulation of the policy such that it is conditioned on the return π <a St, Gt, sa)?. In Table 3, we outline how different approaches in offline RL have learned return-conditinn conuun-condd poli- cies. In this work, we adopt an approach that learns the joint distribution of returns and actions of an agent
given dataset. Specifically, (a Grlst, SG) = no (aelse, SG, G2 pooooolss 8G). We note that [17] found it helpful to also utilize a model-based return-conditioned policy, whereby the future state is modelled part of the joint distribution being learned. This is shown to provide useful regularizing signal for the policy, even though the future state prediction is not directly used at inference time. In this work, we also found it helpful to regularize the learned policy by predicting the full sequence of future states. The final distribution we are aiming to model thus given by po($c+1.T, az, Gr(st, = pe ( SG, G., a,) (e) &G, G.)pe(Gele sa).
At inference time, obtain actions by first sampling returns G. (G. sc) and then sam- pling actions aę (a $t, sc, Gt). This sampling procedure corresponds to the imitative pol- icy since the sampled returns are obtained from the learned density that models the data distri- bution. Following prior work in offline RL [16, 17, 18). we can also sample actions from an exponentially- policy distribution. This is done by sampling the returns from the tilted distri- bution G Po (G, (st, 8G) exp(1:G)) with G; being the tilted return-to-go and where represents the inverse temperature; higher values of и concentrate more density around the best outcomes or higher returns, while negative values of x concentrate less favourable outcomes lower returns.
We are interested in modelling and controlling the individual components of the reward function rather than maximizing their weighted sum. For example, we would like to model an agent's ability to reach its goal, drive on the road, and avoid collisions. In general, given reward components, our objective is to learn policies that are conditioned on all factored dimensions as this would grant us control over each one test time. This entails modelling separate return components Gi pe( G se, SG) for each return component Applying this factorization, reformulate the learned policy to explicitly account for the conditioning on all return components (p |St: &G, G!....Gy At test time, each return component will be accompanied by its own inverse temperature nº to
Note that omit the additional context for brevity.
3

---

enable control over each return component, which enables sampling actions that adhere to different behaviours specified by [x',...,,'" ) as shown in Algorithm 2 in Appendix B.
To implement our framework for behaviour simulation, we extend the approach presented above to the multi-agent setting. Across all agents we have sets for the joint states St, goal states Sa, actions Ae, and returns-to-2 Gr The final multi-agent joint distribution we model is:
po (0 A. G. = po SS, Sa, Gr Ar) (A Sa. GG), Sc), (1)
where the returns and actions from the previous timesteps are shared across agents, while at the present timestep they are masked out so one can only observe one's own return and action.
2.2 Multi-Agent Behaviour Simulation Architecture
In this section, we introduce the proposed architecture for multi-agent behaviour simulation within the CtRL-Sim framework that parameterizes the multi-agent joint distribution presented in Equation (1). We propose encoder-decoder Transformer architecture [19], as illustrated in Figure 2, where the encoder encodes the initial scene and the decoder autoregressively generates the trajectory rollout for all agents in the scene.
Encoder To encode the initial scene, we first process the initial agent states and goals (Sn, sG) and the map context m, where So is the joint initial state of all agents and sg the joint goal state of all agents. Each agent i's initial state information sy, which includes the position, velocity, heading, and agent type, is encoded with an MIP. Similarly, each agent's goal s.g, which is represented as the ground-truth final position, velocity, and heading, is also encoded with an MLP. We then concatenate the initial state and goal embedding of each agent and embed them with a linear layer to get per- agent embeddings of size d. We additionally apply an additive Iearnable embedding to encode the agents' identities across the sequence of agent embeddings. The map context is encoded using a polyline map encoder, detailed more fully in Appendix F, which yields L road segment embeddings of size d. The initial agent embeddings and road segment embeddings are then concatenated into sequence of length N + L and processed by a sequence of E' Transformer encoder blocks.
The proposed decoder architecture models the joint distribution in Equation (1) as a se- quence modelling problem, where we model the probability of the next token in the sequence con- ditioned on all previous tokens po( [15]. In this work, we consider trajectory sequenees of the form: x = (.... (s) s(): (G.
Decoder
a. (s a (G.... .......... G..., aaM..... These sequences are an extension the sequences considered in the Multi-Game Decision Trans- former [16] to the multi-agent goal-conditioned setting with factorized returns. Unlike Decision Transformer [15], model predicts the return distribution and samples from it at inference time, which enables flexible control over the agent behaviours and circumvents the need specify an expert return-to-go. We obtain state-goal tuple (sf, s'c) embeddings in the same way that (s),8c) sc) are processed in the encoder. Following recent work that tokenizes driving trajectories [20, 9], we discretize the actions and return-to-gan into uniformly quantized bins. We then embed the action and return-to-ge go tokens with a linear embedding. To each input token, we additionally add two learnable embeddings representing the agent identity and timestep, respectively. The tokenized sequence is then processed by D Transformer decoder layers with a temporally causal mask that is modified to ensure that the model is permutation equivariant the agent ordering (see Appendix F for details).
Training Given a dataset of offline trajectories (Section 3), we train our model by sampling se- quences of length return-to-go, and action token embeddings output by the decoder are used to predict the next return token, action token, and future state sequence, respectively. We train the return-to-go and action headers with the standard cross-entropy loss function and the future state sequence header with an L2 regression loss function. The final loss function is of the form: L = Luction Cremmm-o--o a
N x 3, where is the number of timesteps in the context. The state,

---

0- 1 +
40. 20
Figure 3: Qualitative results of multi-agent simulation with CtRL-Sim. The teal agents are controlled by CtRL-Sim, and other agents pink are set log-replay through physics.
Goal Success
Collision (%) 4
Road
Per Soene
Method
(m)
(m) +
Rate (%) f
(x10 2)
(%) 4
Gen. Time
Replay-Physics"
0.47
0.97
87.3
7.6
2.8
10.7
1.1
Actions-Only [9]
1.81a052
11 11.8
32, 7=14
10. 10.4
19.9=12
27.6+110
3.3
Imitation Learning
1.24g00s
1.95
77.4
5.8102
12.14a2
DT (Max Return)
1.562004
3.07=046
63.3-08
8.4
5.3203
11,0a22
20.7
CTG++ [11]
1.7320.0
4.02
38.8254
7,4g02
5.9g24
15.0
44.0
CtRL-Sim (No State Prediction)
1.32
2.21=006
72.4=03
8,2g02
6.1404
12.0+03
CiRI-Sim (Base)
1.2992004
2.13200s
73.0a1a
8.1g0a
11.8102
8.2
CtRL-Sim (Positive Tilting)
1.252003
2.04=005
72.9.15
7.9
5.3502
11.0a2
DT (GT Retur)
1.10
77.5=15
8,4.01
5.3403
11 11.9
CtRL--Sim (GT Initial Return)
1.09
1.602m06
77.2
8.1g02
5.6220
12.2191
20.8
Table 1: Multi-agent simulation results 1000 test scenes. We report mean across 5 seeds. CtRL-Sim achieves good balance between reconstruction performance, common sense, realism, and efficiency. indicates privileged models requiring GT future. indicates reimplementation.
3 Experiments
3.1 Experimental Setup
Offiine RL Dataset We curate an offline RI. dataset derived from the Waymo Open Motion Dataset (WOMD) [14]. We extend Nocturne by integrating a physics engine based on the Box2D library for enabling realistic vehicle dynamics and collisions, detailed in Appendix C.I. Each scene in the Waymo dataset is fed through the physics-enhanced Nocturne simulator to compute the per-tine per-timestep actions and factored rewards for each agent. The factored rewards comprise of goal position, vehicle-vehicle collision, and vehicle-road-etee collision reward, detailed in Appendix C.2.
Evaluation We evaluate CtRL-Sim on its ability to replicate the driving behaviours found in the Waymo Open Motion Dataset (imitation) and generate counterfactual scenes that are consistent with specified tilting coefficients (controllability. For both modes of evaluation, of history and simulate 8 second future rollout. For imitation, agents per scene that we control with CtRL-Sim, where the remaining agents are set to log replay through physics. We evaluate on 1000 random test scenes in both modes of evaluation. Following recent work [7]. we use three types metrics for imitation evaluation: reconstruction metrics, such as Final Displacement Error (FDE), Average Displacement Error (ADE), and Goal Success Rate; distributional realism metric (JSD) defined by the mean of the Jensen-Shannon Distances computed on linear speed, angular speed, acceleration, and distance to nearest vehicle features between real and simulated scenes; and common sense metrics measured by Collision and Offroad rate.
use second evaluate up to 8 moving
For controllability evaluation, we evaluate on selected "interesting" interactive agent that con- trolled by CtRL-Sim, defined as an agent who is moving and whose goal is within metres of another moving agent. All agents except for the CiRL-SSi--onntrollled interesting agent are set to log replay through physics. We evaluate the model's controllability through metrics aligned with the specified reward dimensions: report the goal success rate for the goal reward control, collision rate for the vehicle-vehicle reward control, and offroad rate for the rchicle-road-edee reward control.
5

---

0.30
DT mox rotr D7 nauun
0.225
OT neum citidin
Rate
0.25
0,200
CIRL-Sm IT
Goal Success Mate
0.75
6.175
0.20
0.15
DT rau metm Dr min reta
0.125
0.65
0.100
0.60
-25
-10 -5 0 Thking Neticce-etiiee
5 10
23 -25
-10
10
25
-25
-10
30-3
10
25
Tiking (Véhicle Edge)
Titting (Goal)
Figure 4: Effects of exponential tilting. Comparison of CtRL-Sim base model (magenta) and fine-tuned model (purple) across different reward dimensions. Rewards range from -25 to 25 for vehicle-vehicle collision (left), vehicle-edge collision (middle), and goal reaching (right). Results show smooth controllability, with fine-tuning enhancing this effect. We report mean over 5 seeds.
Methods under Comparison For imitation evaluation, we compare CtRL-Sim against several rel- evant baselines: 1. Repluy-Physics employs inverse bicycle model to obtain the ground-truth log-replay actions and executes through the simulator. 2. Aactions-Only is an encoder-decoder model inspired by [9] where the decoder trajectory sequences only contain actions. 3. Imitation Learning (IL) is identical to the architecture in Section 2.2 except with the removal of returns and the future state prediction. 4. Decision Transformer (DT): The GT Initial Return variant specifies the initial ground-truth return-to-go from the offline RL dataset, with the goal of acting an imitative policy. Max Return follows the standard DT approach of selecting the maximum observable return in the dataset. The DT architecture is identical to that of CtRL-Sim except the return token precedes the state token, and the returns and future states are not predicted by the decoder. 4. CTG++ is a reim- plementation of [11], a competitive Transformer-
diffusion model for behaviour simulation.
We evaluate the following variants of CtRL-Sim: 1. CtRL-Sim (Base) is the CtRL-Sim model trained on the offline RL dataset. 2. CtRL-Sim (No State Prediction) the base model trained without the state prediction task. 3. CtRL-Sim (Positive Tilting) applies Ke 10 tilting all components of the base model. 4. Instead of predicting the return-to-go at each timestep, CIRL-Sim (GT Initial Return) uses the ground-truth initial return-to-go and, at each timestep, decrements the reward from the initial return-to-go until the episode terminates. For controllability evaluation, we evaluate on the base model and a finetuned CtRL-Sim model (CtRL-Sim FT). The finetuned model takes a trained base model and finetunes it on a dataset of simulated long-tail scenarios that we collect using an existing simulated collision generation method CAT [21]. This allows CtRL-Sim be exposed to more long-tail collision scenarios during training, as the WOMD mainly contains nominal driving. We refer readers to Appendix H for details of CAT and our proposed finetuning procedure.
3.2 Results
Table presents the multi-agent imitation results comparing the CtRL-Sim model and its variants with imitation baselines. The CtRL-Sim models perform competitively with the imitation baselines, with the CtRL-Sim (Positive Tilting) model achieving good balance between distributional realism (2nd in JSD), reconstruction performance (2nd in FDE, ADE), common sense (Tied 1st in Collision and Offroad Rate), and efficiency (5.4x faster than CTG++). We note that while II. faster than CtRL-Sim due to fewer tokens to decode and offers similar performance, IL not controllable. Although DT (Max Return) attains equal collision and offroad rates CtRL-Sim, the reconstruction performance is substantially worse. We further validate the importance of the future state prediction task, with CtRL-Sim (Base) outperforming CtRL-Sim (No State Prediction) across all metrics. The CtRL-Sim (Positive Tilting) model attains the best collision rate and offroad rate, demonstrating the effectiveness of exponential tilting for steering the model towards good driving behaviours.
A distinctive feature of CtRL-Sim is that it enables intuitive control over the agent behaviours through exponential tilting of the return distribution. This contrasts with DT, which, although pable of generating suboptimal behaviours by specifying low initial return-to-gos lacks intuitive

---

Planner Metries
Adversary Realism
Progress (m)
Coll. w/ Adv.
ISD
Coll. Speed (m/s) +
Adv. Method
Tilt
Reactive?
Control?
(%)
(x (x10-2)4
CAT
53.3
61.4
18.7
6.9
CtRL-Sim
10
/
57.5
10.01aa
10.8103
7.4105
10
57.7.0.1
8.720.
10.3+05
8.3
10
56. 56.1102
33.8119
17.4106
6.310.2
CtRL-Sim FT
10
57.1+0.1
18.5+16
12.6+07
6. 6.1
50
57.4+0.2
12.8+02
15.6+1.2
6.0+03
Table 2: Adversarial scenario generation results over 1000 test scenes. We report the mean=sd over 5 seeds for the CtRL-Sim models. Finetuning CtRL-Sim on CAT data improves ability to generate adversarial scenarios compared with base CtRL-Sim model. Compared with CAT, CtRL Sim reactive and controllable, while exhibiting better collision realism.
control due to the prerequisite knowledge about the returnttogo values and an absence of an in- terpretable mechanism for behaviour modulation. By constrast, the exponential tilting employed in CtRL-Sim has a clear interpretation: negative exponential tilting yields behaviours that are worse than the average behaviours learned from the dataset, while positive exponential tilting yields better- than-average behaviours. We show the results of our controllability evaluation in Figure 4. For each reward dimension corresponding metric of interest. We also show the results of DT when conditioning on the mini- mum and maximum possible return. For both the base and finetuned CtRL-Sim models, we observe a relatively monotonic change in each metric of interest as the tilting coefficient is increased. As the finetuned model exposed to collision scenarios during finetuning. it demonstrates significant improvements over the base model in generating bad driving behaviours. Specifically,, at -25 tilting, the finetuned model able generate 2.1 as many collisions and 1.8x as many offroad violations as the base model. Figure 5 (and 7, 8 in Appendix J) shows qualitatively the effects of tilting.
exponentially tilt K. between -25 and 25 and observe how this affects the
Table evaluates CtRL-Sim's ability to produce adversarial agents that collide with a data-driven planner. We evaluate on a held-out test set of two-agent interactive scenarios from the Waymo interactive dataset, where one interacting agent controlled by the adversary and the other is con- trolled by the planner. We use positively-tile CtRL-Sim base model as our planner, due to its demonstrated ability to produce good driving behaviours in Table 1. For adversarial scenario gen- eration, we compare the base CtRL-Sim model against the CtRL-Sim FT model. With -10 tilting applied, the finetuned model generates 238 more collisions with the planner than the base model over 1000 scenes, which we attribute to its exposure to simulated collision scenarios during fine- tuning. Notably, CtRL-Sim FT was finetuned in only 30 minutes on 1 NVIDIA A100-Large GPU and only 3500 CAT scenarios. This underscores CtRL-Sim's capability to flexibly incorporate data from various sources through finetuning, thereby enabling the generation of new kinds of driving behaviours. Importantly, after finetuning, CtRL-Sim FT largely retains its ability produce good driving behaviours. This is evidenced by 21.1 percentage point decrease in the planner's 's collision rate when using a +50 positively tilted finetuned model as the adversary. We also compare CtRL- Sim against a state-of- tatt-offtheear collision generation method, CAT [21]. which uses a motion prediction model to select plausible adversarial trajectories that overlap with the ego plan. Although CAT gen- erates more collisions, CAT is not controllable as it can't 't control how adversarial the agents are, and CAT agents are non-reactive to the ego's actions as the trajectory fixed prior to the simulation, limiting its realism. This is evidenced by larger adversary collision speed than all finetuned CtRL- Sim models and is also validated qualitatively in the supplementary video. We further conduct user study to confirm that CiRL-Sim adversarial scenarios are indeed more realistic than CAT adversarial scenarios, with details and results reported in Appendix I.
4 Related Work
Agent behaviour simulation involves modelling the behaviour of other agents in simulation. such as vehicles and pedestrians. Agent behaviour simulation methods can be categorized into rule-based
7

---

Vehidle-Vhhicle Tilting
Vehicle-Edge Tilting
Negative Tilting
Positive Tilting
Negative Tilting
Positive Tilting
Qualitative results of vehicle-vehicle and vehicle-edge tilting, Two traflic scenes com- paring positive tilting of the CuRL-Sim-controlle the same agent. Bounding boxes in red indicate traffic violations. Other agents log-replay through physics, with interacting agents in pink. Goals are marked by small circles.
Figure 5:
agent (shown in teal) with negative tilting for
and data-driven methods. Rule-based methods rely human-specified rules to produce plausible agent behaviours, such as adhering strictly to the center of the lane [2, 3]. These methods often yield unrealistic behaviours that fail to capture the full spectrum of driving behaviours. To address these limitations, prior work has proposed learning generative models that aim to replicate agent behaviours found in real-world driving trajectory datasets [22, 23, 24, 4, 5, 25, 9]. These approaches draw inspiration from methods for the task of joint motion prediction [26, 27, 28, 29, 30]; however, it's crucial to distinguish that, unlike the open-loop nature of joint motion prediction, behaviour sim- ulation operates closed-loop [31]. To improve the realism of the learned behaviours, other work has proposed using adversarial imitation learning [32] to minimize the behavioural discrepancy between expert and model rollouts [33, 34, 6] or RL to improve traffic rule compliance [35, 36]. While such methods demonstrate improved realism over rule-based methods, they lack the necessary control over the behaviours to enable the generation targeted simulation scenarios for AV testing.
More recent work has proposed more controllable behaviour simulation models by learning con- ditional models [10, 37, 7, 38, 8, 11, 12] that enable conditioning
high-level latent variables [10, 37], route information [7]. or differentiable constraints [8, 11, 39, 12, 40]. More recently, [41] used retrieval augmented generation to generate controllable traffic scenarios. However, these methods either lack interpretable control over the generated behaviours [37] or require costly test- time optimization procedures to steer the generated behaviours, such as latent variable optimiza- tion [10], Bayesian optimization [42, 43, 7], or the simulation of expensive diffusion processes [8, 44, 11, 39, 12, 40]. In contrast, CtRL-Sim offers more efficient alternative and learns a ditional multi-agent behaviour model that conditions interpretable factorized returns, thereby eliminating the need for costly test-time optimization. By exponentially tilting the predicted return distribution [16] at test time, CtRL-Sim enables efficient, interpretable, and one-grained control over agent behaviours while being grounded in real-world data.
5 Conclusion
We presented CtRL-Sim, a novel framework applying offline RL for controllable and reactive be- haviour simulation. Our proposed multi-agent behaviour Transformer architecture allows CtRL- Sim to employ exponential tilting at test time to simulate a wide range of agent behaviours. We present experiments showing the effectiveness of CtRL-Sim producing controllable and reactive behaviours, while maintaining competitive performance the imitation task compared to baselines.
Limitations The learned policies produced by CtRL-Sim in its current form may make large-scale RL training on CtRL-Sim scenarios prohibitively expensive. However, we believe that further op- timizations of the model and simulator such as exploring more compact models, utilizing model quantization or Flash Attention [45], reducing the planning frequency, or making the simulator GPU-accelerated can further improve the inference latency. We leave this for future work. Fur- thermore, CtRL-Sim enables control over the behaviours, but not the initial agent placements. We believe that a generative model that supports controlling both the initial agent placements and the agent behaviours is promising future work to explore.

---

Acknowledgments
LP and CP are supported by CIFAR under the Canada CIFAR AI Chair program and by NSERC under the Discovery Grants program.
References
[1] K. D. Kusano, J. M. Scanlon, Y. Chen, T. L. McMurry, R. Chen, T. Gode, and T. Victor. Comparison of waymo rider-only crash data to human benchmarks at 7.1 million miles. arXiv preprint ar.iv. 27771,232..112 2023.
[2] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
[3] A. Kesting, M. Treiber, and D. Helbing. General lane-changing model mobil for car-following models. Transportation Research Record, 1999(1):86 94, 2007.
[4] S. Suo, S. Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[5] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. BITS: bi-level imitation for traffic simulation. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2023.
[6] M. Igl, D. Kim, A. Kuefler, P. Mougin, P. Shah, K. Shiarlis, D. Anguelov, M. Palatueci, B. White, and S. Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In Proceedings of the International Conference on Robotics and Automa- tion (ICRA), 2022.
[7] S. Suo, K. Wong, J. Xu, J. Tu, A. Cui, S. Casas, and R. Urtasun. MIXSIM: A hierarchical framework for mixed reality traffic simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[8] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2023.
[9] J. Philion, X. B. Peng, and S. Fidler. Trajeglish: Learning the language of driving scenarios. arXiv preprint 2312v2311..4555. 2023.
[10] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[11] Z. Zhong, D. Rempe, Y. Chen, B. Ivanovic, Y. Cao, D. Xu, M. Pavone, and B. Ray. Language- guided traffic simulation via scene-level diffusion. arXiv preprint arXiv.2306.06344 06344, 2023.
[12] W. Chang, F. Pittaluga, M. Tomizuka, W. Zhan, and M. Chandraker. Controllable safety- critical closed-loop traffic simulation via guided diffusion. arXiv preprint 2xX0v.2401.00391, 2024.
[13] E. Vinitsky, N. Lichtlé, X. Yang. B. Amos, and J. Foerster. Nocturn: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems (NeurIPS). 2022.
[14] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, Ngiam, Vasudevan, A. McCauley, Shlens, and D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, (ICCV), 2021.

---

[15] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Ad- vances in Neural Information Processing Systems (NewrIPS), 2021.
[16] K. Lee, O. Nachum, M. Yang, L. Lee, Freeman, S. Guadarrama, Fischer, W. Xu, E. Jang, H. Michalewski, and L. Mordatch. Multi-gama decision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[17] N. Gontier, R. López, H. Laradji, D. Vázquez, and C. Pal. Language decision transform- ers with exponential tilt for interactive text environments. arXiv preprint arXin:3302.55507, 2023.
[18] A. Piché, R. Pardinas, D. Vazquez, and C. Pal. A probabilistic perspective reinforcement learning via supervised learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
[19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[20] A. Seff, B. Cera, D. Chen, M. Ng. A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[21] L. Zhang, Z. Peng, Q. Li, and B. Zhou. CAT: closed-loop adversarial training for safe end-to- end driving. In Conference Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pages 2357-2372. PMLR, 2023.
[22] L. Bergamini, Y. Ye. O. Scheel, L. Chen, C. Hu, L. D. Pero. B. Osinski, H. Grimmett, and Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations. In Proceedings of the International Conference Robotics and Automation (ICRA), 2021.
[23] A. Kamenev, L. Wang, O. B. Bohan, I. Kulkarni, B. Kartal, A. Molchanov, S. Birchfield, D. Nistér, and N. Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic predic- tion for planning. control, and simulation. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2022.
[24] A. Scibior, V. Lioutas, D. Reda, P. Bateni, and F. Wood. Imagining the road ahead: Multi-agent trajectory prediction via differentiable simulation. In Proceedings of the IEEE International Intelligent Transportation Systems Conference (ITSC), 2021.
[25] Y. Wang, T. Zhao, and Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. arXiv preprint arXiv. 23X.h.2306.11866 2023.
[26] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[27] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D'Souza, S. E. Kahou, F. Heide, and C. J. Pal. Latent variable sequential set transformers for joint multi-agent motion pre- diction. In International Conference Learning Representations, 2021. URL https: //api. semannicscchollar hwwphhholloor.ro/oooooooo Barg/CorpusID.... 246824069.
[28] Ngiam, Vasudevan, B. Caine, Z. Zhang,
Chiang, J. Ling, Roelofs, Bewley, C. Liu, A. Venugopal, D. J. Weiss, B. Sapp. Z. Chen, and J. Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference Learning Representations (ICLR), 2022.
10

---

[29] Y. Chen, B. Ivanovic, and M. Pavone. Scept: Scene-consistent, policy- bolee trajectory pre- dictions for planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[30] L. Rowe, M. Ethier, E.-H. Dykhne, and K. Czarnecki. FJMP: factorized joint multi-agent motion prediction over learned directed acyclic interaction graphs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[31] H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. M. Wolff, A. H. Lang, L. Fletcher, O. Beijbom, and S. Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv: 1a..21006//8100 2021.
[32] Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2016.
[33] R. P. Bhattacharyya, D. J. Phillips, B. Wulfe, J. Morton, A. Kuefler, and M. J. Kochenderfer. Multi-agent imitation learning for driving simulation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2018.
[34] G. Zheng, H. Liu, K. Xu, and Z. Li. Objective-aware traffic simulation via inverse reinforce- ment learning. In Proceedings of the International Joint Conference Artificial Intelligence (IJCAI), 2021.
[35] Y. Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. White- son, D. Anguelov, and S. Levine. Imitation not enough: Robustifying imitation with rein- forcement learning for challenging driving scenarios. JEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.
[36] C. Zhang. J. Tu, L. Zhang, K. Wong. S. Suo, and R. Urtasun. Learning realistic traffic agents in closed-loop. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedingy of Machine Learning Research, pages 800-821. PMLR, 2023.
[37] Z. Zhang. A. Liniger, D. Dai, F. Yu, and L. Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In Proceedings of the International Conference Robotics and Automation (ICRA), 2023.
[38] W. Chang, C. Tang, C. Li, Y. Hu, M. Tomizuka, and W. Zhan. Editing driver character: Socially behavior generation for interactive traffic simulation. IEEE Robotics Au- tom. Lett., 2023.
[39] C. Xu, D. Zhao, A. Sangiovanni-Vincene
and B. Li. Diffscene: Diffusion-basee safety- critical scenario generation for autonomous vehicles. In The Second Workshop on New Fron- fiers in Adversarial Machine Learning, 2023.
[40] Z. Guo, X. Gao, J. Zhou, X. Cai, and B. Shi. Scenedm: Scene-level multi-agent trajectory generation with consistent diffusion models. arXiv preprint: drXiv2311.15736, 2023.
[41] W. Ding, Y. Cao, D. Zhao, C. Xiao, and M. Pavone. Realgen: Retrieval augmented generation for controllable traffic scenarios. arXiv preprint arXiv22312.13303 2023.
[42] Y. Abeysirigonnawarden,, F. Shkurti, and G. Dudek. Generating adversarial driving scenarios in high-fidelity simulators. In International Conference on Robotics and Automation (ICRA), 2019.
[43] Wang, A. Pun, Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
11

---

[44] C. M. Jiang, A. Commann, C. Park, B. Sapp, Y. Zhou, and D. Anguelov. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[45] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory--fficient exact attention with io-awareness. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[46] R. K. Srivastava, Shyam, F. W. Mutz, W. Jaškowski, and Schmidhuber. Training agents using upside-down reinforcement learning. ArXiv, abs/1912.02877,, 2019. URL https :// api semanticscholar org/CorpusID 208857468.
[47] A. Kumar, X. B. Peng, and S. Levine. Reward-conditinedd policies. arXiv:1912.13465 2019.
arXiv preprint
[48] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What essential for offline via leaminng ArXiv, ab5/211 2021. org/CorpusID: 245334837.
[49] L. Meng, M. Wen, Y. Yang. C. Le, X. Li, W. Zhang, Y. Wen, H. Zhang, J. Wang, and B. Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tack- les smac tasks. ArXiv, abs/2112.02844) 2021. URL https: http:///ap..mmmmticcccoo..... semanticscholar. org/CorpusID 245335360.
[50] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-based permutation-invarinl neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[51] A. Ibrahim, B. Thérien, K. Gupta, M. L. Richter, Q. Anthony, T. Lesort, E. Belilovsky, and I. Rish. Simple and scalable strategies to continually pre-train large language models. arXiv 2024.
preprint
12

---

Approach
Density Estimation
Action Sampling Density
Decision Transformers (DTs)
log lo, G.)
pe(a.la,,.))
Reward Weighted Regression (RWR)
expor G.) log paloolle (og)oo)
py(ai(o))
Reward Conditioned Policies
log pol lo,, (G.jo)
po(axlo,, Grlpo [ttpp/GG/oeexpeeeeeee
((K)
Reweighted Behavior Cloning (RBC)
log Po (Gg [plo))))
Pu(G, [o) (alo:/aaa//aaa (oplaa (o) exp(xG] 7(6))
Implicit RL (IRvS)
log te (iy Gglor)
Po(ar, Gelo, exp[KG *(x])
Model-Based RCPs (MB-RCP)
log Po(o Ge)polaelo (G.lo)
polaxlor, (Gllo) exp(xG. m(N))
RCP with Future Rollout (CiRL-Sim)
log pol 10 (0,, Ce) (a)pe C. (G, pola. G2 Jpa (p exp(xG, o(x))
Table 3: Offline policy modelling approaches in prior work. We can see that methods differ in the decomposition of the joint distribution over actions and returns, with some approaches utilizing state prediction as regularizer. We note that this table is adopted from prior work (18, 17].
A Offline RL Approaches
We position CtRL-Sim within the field of offline RL. Table presents the different ways explored in the literature for learning policies in offline RL, and how these methods sample return- maximizing actions at test time. Prior methods in offline RI differ in how the policy is modelled during training and how inference is performed; we refer the reader to Table 3 in the Appendix A for a breakdown of the different approaches. As CtRL-Sim employs return-condinn return-conditioned policy for controllable simulation, we briefly present a class of related methods that learn return-conditioned policies (RCPs) [15, 16, 46, 47, 48, 18, 49] for offline RL. RCPs are concerned with learning the joint distribution of actions and returns, such that action sampling is conditioned the return dis- tribution. Instead of modelling the return distribution, the Decision Transformer [15] conditions the learned policy on the maximum observed return, Gmax in the dataset. That is, at inference time, the actions are sampled from Po(a; (s, Gmax). Lee et al. [16] instead propose to employ the learned return distribution, combined with exponential tilting, in order to sample high-retumn actions while remaining close to the empirical distribution. CtRL-Sim adopts exponential tilting of the pre- dicted return distribution for each agent to finely control agent behaviour. Additionally, CtRL-Sim explicitly models the future sequences of states.
B Action Sampling Algorithm
Algorithm 2 describes the proposed action sampling procedure for controllable behaviour generation with factorized exponential tilting.
Algorithm 2 The action sampling algorithm used by CtRL-Sim to allow for factorized tilting of the exhibited behaviour.
1: Input: (s*,...,,
The specified inverse temperature for each return-to-go component.
2: for c to C do
G (o sc) Gx)
4: end for
5: ag ~ (aels sa, Gt ,t....... G(C)
6: return 0t
C Nocturne Physics Simulator and Offline RL Dataset
C.1 Physics-based Nocturne Simulator
CtRL-Sim extends the Nocturne simulation environment [13]. Nocturne is lightweight 2D driving simulator that built on real-world driving trajectory data from the Waymo Open Motion Dataset [14]. scene in Nocturne represented by a set of dynamic objects such as vehicles, pedestrians, and cyclists and the map context, which includes lane boundaries, lane markings, traffic signs, and crosswalks. Each dynamic object is prescribed goal state, which is defined as the final waypoint in the ground-trut -ruth trajectory from the Waymo Open Motion Dataset. If there exist missing timesteps
13

---

in the ground-trute truth trajectory, we re-define the goal as the waypoint immediately preceding the first missing timestep. By default, the dynamic objects track its 9 second trajectory from the Waymo Open Motion Dataset at 10 Hz. The Nocturne Simulator is originally designed for the development of RL driving policies, where the first 10 simulation steps (Is) of context is provided and the RL agent must reach the prescribed goal within the next 80 simulation steps (8s).
We extend Nocturne by integrating physics engine based on the Box2D library for enabling real- istic vehicle dynamics and vehicle collisions. We model the vehicle's 's dynamics using basic physics principles, where forces applied to the vehicle are translated into acceleration, influencing its speed and direction, and with frictional forces applied to simulate realistic sliding and adherence behav- iors. This extension additionally ensures that an agent's 's acceleration, braking, and turn radius are bound by plausible limits and that vehicles can physically collide with each other. Such improve- ments open the possibility of more accurately simulating complex conditions, such as emergency braking maneuvers, slippery roads, and multi-vehicle collisions.
C.2 Offline RL Dataset Collection
The actions are defined by the acceleration and steering angle and the reward function decom- posed into three components: goal position reward, vehicle to vehicle collision reward, and vehicle to road edge collision reward. We confirm that the trajectory rollouts obtained by feeding Waymo scenes through the simulator attain reasonable reconstruction of the ground-truth Waymo trajectories (see Table 1). Following Nocturne [13], we omit bicyclist and pedestrian trajectories from the Waymo Open Motion Dataset and we omit scenes containing traffic lights. This yields a training, validation, and test set containing 134150, 9678, and 2492 scenes.
For each agent, to obtain the action a, at timestep t, we compute the acceleration and steering value using an inverse bicycle model computed from the agent's current state in the simulator & and the ground-truth next state from the trajectory driving log 8t+1. We clip accleration values between -10 and 10 and steering values between -0.7 and 0.7 radians. We then execute a₂ with our proposed forward physics dynamics model to obtain the agent's updated state $ and we repeat until the agent has completed the full rollout. Table 1 confirms that this approach to offline RL trajectory data collection yields a reasonable reconstruction of the ground-truth -ruth driving trajectories.
We compute rewards at each timestep, where our reward function is factored into three rewards components: goal position reward, vehicle-vehicle collision reward, and vehicle-roal-edge col- lision reward. We chose the reward functions based on the information provided by the Nocturne simulator, which includes indicators for goal success, vehicle-vehicle collisions, and vehicle-edge collisions. For each agent, their goal set to the final state (i.e., position, heading, and velocity) of the ground-truth logged trajectory. The goal position reward is defined by:
Ry(se (G) 1 goal achieved St, sG);
where goal achieved(.) is 1 if the agent ever reaches within metre of the ground-truth goal, and otherwise. The vehicle-vehicle collision reward defined by:
Ro(s S. {st}) -10 x 1 vehicee-vehicc collision (st, S, {st})
min
ve {s: 15)
15
where dist-neerest--xhhillee)) computes the distance between the agent of interest and its nearest agent in the scene. Finally, the vehicle-road-edes collision reward is defined by:
min dcdis--nearrst--rdd-dddedeed m),
(st, m) = 10 x I vehicle-rond-cdge collision (St, m)
where dist-nearest-rooa--ddge computes the distance between the agent and the nearest road edge.
D Evaluation Metrics
The goal success rate is the proportion of evaluated agents across the evaluated test scenes that get within I metre of the ground-truth goal position at any point during the trajectory rollout. The
14

---

JSD (x10-3)
Method
Lin. Speed
Ang. Speed
Accel.
Nearest Dist.
Meta-JSD
Heplay-Physice
0.1
11.5
17.4
1.2
7.6
Actions-Oule Oaooos-OOly/ia 191
4.1 4.1 0.7
16,8 0.3
15.6 + 0.5
5.1 + 0.5
10.4 + 0.3
Imitation Leariing
1.0 0.1
1.3
16.9 0.4
2.2
8,3 0.1
DT (Max Retur) [15]
2.7 0.1
1.24
15.4
+5.4 0.5
2.2
8.4
CTG++ [11]
3.2+
11.9
12.4 0.4
2.2
7,4 0.2
CtRL-Sim (No State Prediction)
1.2
13.7
15,9 + 0.7
2.0
8.2 2 0.2
CtRL-Sim (Base)
1.1 0.2
13.8
15.6 0.5
0.3
8.1 0.2
CtRL-Sim (Positive Tilting)
1.4 0.1
13.6 0.2
14.8 + 0.5
1,8 + 0.2
7.9 0.1
DT (GT Initial Return)
1.1 +: 0.2
13.4 0.2
16.8 1 0.6
2.1 2.1
8.4 + 0.1
CIRI-Sim' (GT Initial Return)
1.1 + 0.2
13.8 + 0.3
15.3 + 0.6
2.2
2.2
8.1 + 0.2
Table 4: Breakdown of Meta-JSD in Table 1. For each metric, the best unprivileged method is bolded and second-bes is underlined. denotess a privileged method requiring the ground-truth future trajectory.
final and average displacement errors are calculated for evaluated agents across the test scenes and averaged. For specific scene s, the collision rate and offroad rate of are the proportion of evaluated agents in that collide with another agent road edge, respectively. These rates are then averaged across all tested scenes define the overall collision and offroad rates.
We compute the Jensen Shannon Distance (JSD) between the distributions of features computed from the real and simulated rollouts. The Jensen Shannon Distance between two normalized his- tograms p and q is computed as:
DKI. + DKL. (kt.(allm)
2
where m is the pointwise mean of and and DKL. is the KL-divergence. Unlike prior works that compute the Jensen Shannon Divergence [6. 7]. we compute its square root - the Jensen Shannon Distance so that values are not too close to We compute the JSD over the following feature distributions: linear speed, angular speed, acceleration, and nearest distance. Since the acceleration values discrete, for the acceleration JSD, define one histogram bin for each valid acceleration value, yielding 21 evenly spaced bins between -10 and 10. For the linear speed histogram, 200 uniformly spaced bins between O and 30. For the angular speed ISD, we use 200 uniformly spaced bins between -50 and 50. For the nearest distance JSD, we use 200 uniformly spaced bins between 0 and 40.
E Individual JSD Results
In Table 4, we report the per-feature JSD results for Table 1.
F CtRL-Sim Training and Inference Details
The CtRL-Sim behaviour simulation model trained using randomly subsampled se- quences of length of x acceleration and steering into 20 and 50 uniformly quantized bins, respectively, yielding 1000 action tokens. For the return-to-gos, we discretize each return-to-go component G into 350 uniformly quantized bins. All agents and the map context are encoded in global frame as in [28, 9] where we center and rotate the scene on a random agent during training. The map context represented as a set of road segments m := {ri)=r> where each road segment is defined by a sequence of points rı := (pt (pt....pff) where
Training:
where H! 32 and 24. For the actions, we discretize the
the number of road segments and P is the number of points
per road segment. We apply per-point MLP the points of each road segment rı. To produce road segment-level embeddings, we then apply attention-bann based pooling [50] on the embeddings of
the points within each road segment, yielding I. road segment embeddings of size d. We select the
15

---

L = 200 closest lane segments within 100 metres of the centered agent as the map context, and select up to N = 24 closest agents within 60 metres of the centered agent as social context for the model. For each lane segment, we subsample P = 100 points. We use a hidden dimension size = 256, where we use E = Transformer encoder blocks and = 4 Transformer decoder blocks, and we set a = 10o agents. We found it useful to employ goal dropout whereby the embeddings for 10% of agent goals are randomly set to 0 to prevent the model from overrelying on the goal information. We found goal dropout useful for learning an informative map representation. The state, return, and action embed- dings for the missing timesteps are set to O. To ensure that the model is permutation equivariant to the agent ordering [28, 27], we modify the standard temporally causal mask by additionally enforc- ing that each agent can only attend to its own action and return-to-go tokens at the present timestep while allowing access to all agents' state tokens at the present timestep and all agents' tokens in the past timesteps. The CtRL-Sim model is trained using a linear decaying learning rate schedule from 5e-4 for 200k steps using the AdamW optimizer and batch size of 64. At inference, sample actions with temperature of 1.5. The CtRL-Sim architecture comprises 8.3 million parameters that we train in 20 hours with 4 NVIDIA A100 GPUs.
_ in the loss function. We supervise our model only on the trajectories of moving
Inference: CtRL-Sim supports scenes with an arbitrary number of agents. As CtRL-Sim trained with up to N 24 agents, when the number of CiRL-Sim-controllle agents at inference time exceeds N 24, we iteratively select 24-agent subsets at each timestep for processing until all agents have been processed. We first randomly select a CiRL-Sim-controlled agent, we normalize the scene to this agent and select the 23 closest context agents to the CiRI--Si--onttrolledd agent to comprise the first set of 24 agents. We then iteratively continue centering on a CIRL-Sim--omtroolled agent that has not been processed in the previous sets of 24 agents and select its 23 closest agents for context until all CCRL--Sim-controlle agents have been included a 24-agent subset. If an agent belongs to multiple 24-agent subsets, we use the model's first prediction of that agent. At inference time, the context length set training context length H 32. At each timestep, we select H = 32 most recent timesteps context and we found it useful to always center and rotate the scene on the centered agent at the oldest timestep in the context. For the first 10 timesteps (1s) of the simulated rollout, the states and actions are fixed to the ground-truth states and actions from the offline RL dataset, whereas the return-to-go is predicted at every timestep of the simulated rollout.
G Baseline Details
In this section, we describe the design decision of each baseline employed in our work. We note that for all models below, we scaled them in order for all models to have approximately the same number of learnable parameters as CtRL-Sim's architecture.
Actions Only The actions-only baseline is encoder-decoder architecture implemented in exactly the same way CtRL-Sim with few ablations. These include removal of states and returns from the decoder sequence, and no state rollout predictions. This model was inspired by [9] but differs in that the model also has access to the agents' goals.
Imitation Learning The imitation learning baseline is also based on the CtRL-Sim multi-agent behaviour simulation architecture but lacks factorized return information. It is step better than the actions-only baseline since it considers the states in the decoder sequence, and this is corroborated by its improved performance on multi-agent simulation results of Table 1.
DT The Decision Transformer baseline is based on the seminal work [15]. We adopt an identical architecture to CtRL-Sim's 's with some minor difference based on the algorithm. One such decision is the lack of return prediction based on states, and instead returns are chosen at inference time based on domain knowledge. Returns are the first token fed to the decoder, followed by states in order to predict actions. We make the strong argument that this is suboptimal for controllability (results in Figure 4) and does not provide intuitive mechanism for selecting the return values to target.
16

---

ADE
FDE
Goal Success
JSD (x10 9)
Collision (%) 4
Off Road
Per Scene Gen. Time (8) +
Method
(m) 4
(m)
Rate (%) 1
(%) 4
CTG++
1.72
3.97
41.7
7.7
6.4
17.4
44.0
CTG+ (128 hidden dim)
1.83
4.32
37.8
7.6
7.0
17.7
25.0
CTG+ (100 diffusiin steps)
1.87
3.98
43.0
8.6
7.9
16.8
140.0
Table 5: Ablations of CTG++ on Multi-agent simulation results over 1000 test scenes. We report the results across all metrics of different configurations of the CTG++ baseline [11]. f indicates the original model results, with 256 hidden dimension and 50 diffusion steps at inference time.
CTG++ The CTG++ baseline a diffusion model reimplementation of the recent work by [11]. We attempted follow the architecture as closely as possible with a few minor differences. One such difference is that we diffuse over both states and actions, rather than diffusing over only actions and using an unicycle dynamics model derive the states. We chose this approach because the underlying dynamics of the physics-enhanced Nocturne simulator is not necessarily governed by unicycle dynamics model, and thus using unicycle dynamics model would induce small errors in the derived states during training. We further note that we cannot replace the unicycle dynamics model with the Nocturne physics dynamics model as this forward model not differentiable. We also condition on the present timestep and goal, to ensure fair comparison with CtRL-Sim. We note that at scene generation time, we diffuse over actions and states at a rate of 2 Hz which is consistent with the original CTG++ model. In addition, although we train with 100 diffusion steps, we run evaluations with 50 diffusion steps. As showing in Table 5, the difference in performance is insignificant. As we do not have information on the size of the network used in the original manuscript, we explored different hidden dimension sizes of the transformer architecture of the diffusion model, also shown in Table A final difference is the future relative encoding. While CTG++ use the ground-truth to compute the relative encoding during training and a constant velocity model test time, we opted use the final historical timestep's 's relative encoding. We found this approach be more stable.
H
CAT Simulated Data Collection and CtRL-Sim Finetuning
The Waymo Open Motion dataset largely contains nominal driving scenes. To enhance control over the generation of safety-critical scenarios, we finetune CtRL-Sim on a simulated dataset of safety- critical scenarios generated by CAT [21]. CAT is state-of-the the-art collision generation method that involves fixing the agent's future trajectory to a trajectory predicted by Dense trajectory pre- dictor. CAT searches for trajectory that has high likelihood of colliding with the log-replay future trajectory of the ego vehicle, while having high probability under the behaviour prior (Dense'TNT)) For more details, we refer readers to [21]. We note that a limitation of CAT is that the agent non- reactive to the ego as the agent's trajectory is fixed the beginning of the simulation. Moreover, unlike CtRL-Sim, CAT does not have control over the degree which the agent is adversarial.
To collect the simulated safety-critical dataset, we run CAT on subset of the interactive validation split of the Waymo Open Motion Dataset, which involves two interacting agents. Following CAT, we select one of the two interacting agents to be the ego (whose trajectory is fixed to the log-replay trajectory) and the other interacting agent to the CAT adversary. In total, we collect 3577 CAT scenarios for finetuning, of which around 60% contain ego-adversary collisions.
To encourage CtRL-Sim to learn how to generate safety-critical scenarios without forgetting how to generate good driving behaviour, we adopt a continual pre-training strategy for finetuning [51] where we randomly sample 3577 real training scenarios from the offline RL dataset in each training epoch, or a 50% replay ratio. We rewarm the learning rate to the maximum learning rate of 5e-4 over 500 steps and follow a linear decay learning rate schedule to over 20 epochs. We expect that the finetuned CtRL-Sim model will be more capable of generating long-tail scenarios as it is more exposed to such scenarios during finetuning. Finetuning takes roughly 30 minutes 1 NVIDIA A A100-Large GPU.
17

---

I Adversarial Scenario Generation User Study
We conduct user study that contained a total of 24 par- ticipants to evaluate which method (CtRL-Sim vs. CAT) generates more plausible adversarial behaviours. We did not record any identifying information of the partici- pants, and participants were invited on voluntary ba- sis. The user study contained total of 271 paired sce- narios, where each paired scenario consisted of two inter- acting agents, one controlled by a positively-tilted CtRL- Sim planner and the other controlled by an adversary. The scenario conditions of the paired scenarios are iden- tical, except one scenario employed CtRL-Sim adver- sarial agent and the other employed CAT adversarial agent. The adversarial agent in each paired scenario is highlighted in pink that users can distinguish the adversarial agent from the remaining agents in the scene. We do not identify the planner agent as we found this to detract users' attention from the behaviour of the adversarial agent. Each participant was tasked with evaluating randomly selected set paired scenarios from the pool of 271 paired scenarios. For each paired scenario, users selected in which scenario the adversarial agent was more realistic, with the option to select "Tie" the two scenarios were sufficently similar. Users were presented with the user interface shown in Figure 6, where randomize the order between both videos. We use the same positively tilted CtRL-Sim planner in both of the paired scenarios. Table 6 shows the tally of the user study, The results indicate convincingly that CtRL-Sim produces more plausible adversarial behaviours than CAT based on the participants' preferences, as CtRL-Sim was preferred 54 more times than CAT over the 271 paired scenarios.
Method
Times Preferred
CtRL-Sim
123
CAT
69
Tie
79
Table 6: The tally of votes for the larger study. We show the breakdown of the votes for the conducted larger study. We observe that the results of this study are less conclusive than the pilot study.
J Additional Qualitative Results
Figure shows more qualitative examples demonstrating the effects of positive exponential tilting on each of the three reward components. In the left panels, CtRL-Sim with no tilting produces a vehicle-vehicle collision between two interacting agents at a left-turn. With positive vehicle-vehicle tilting, the CtRL-Sim-coom -ontoollllld agent moves more to the right-hand side of the lane to avoid the collision. In the middle panels, CtRL-Sim with tilting produces vehicle-edge collision the bus pulls into the curb. With positive vehicle-edge tilting, the CuRL--iin-controlled agent pulls into the curb at safer distance from the curb. In the right panels, CtRL-Sim with no tilting reaches the goal. With positive goal tilting, the CCRL-Sim-controlles agent reaches the goal much faster and nearly avoids collision with the turning vehicle. In Figure 8, we show two more examples of adversarial collision scenarios generated with negative vehicle-vehicke tilting. We refer the interested reader to the supplementary video for more examples.
K Multi-Agent Simulation Results with Higher Temperature Sampling
In Table 7, we report results from the same experiments as Table 1 except with a higher action sampling temperature, set 1.5.
L Fine-tuning CtRL-Sim on CtRL-Sim Scenarios
Instead of finetuming on CAT scenarios, we explore finetuning CtRL-Sim on adversarial scenar- ios generated by CtRL-Sim. We first collect simulated dataset of scenes either containing vehicle-vehicle collision or an offroad infraction. Specifically, we generate rollouts of single agent with the negatively tilted base CtRL-Sim model where the other agents set to log play through physics, and we save the scenario only if the generated rollout yields vehicle-vchicle
18

---

Survey
In which video does the pink vehicle behave more like human driver? If neither video is preferred, select the "Tie" option.
Figure 6: User Study Example Scenario. We show an example pair of scenarios along with the question users are asked to answer.
Reconstraction ADE (m)
Distributional Realism
Common Sense
Method
FDE (m)
Goal Suc. Rate (%)
Meta JSD
Collision (%)
Road (%)
Rsplay-Physicss Physics"
0.97
0.47
87.3
7.6
2.8
10.7
Actions-Only [9]
11,70 = 1.12
4.78 0.42
34.4
14.3
22.8 0.7
29,7 1.7
Initation Leamiing
0.12
0.07
73.8 + 1.2
12.3 0.5
7.3 0.6
13.1 + 0.4
Retur) [15]
0.11
1.67
1.2
12.3
6.1
11.6
CtRL-Sim (No State Prodiction)
2.57 0.16
1.52 1.52:000 0.07
66.2 1.0
1.3
7.6=00
13.1 0.3
CtRL-Sim (Base)
2.49 0:100
1,50
67.9
12.2.+0.2
7.6 0.3
13.1 0:5
CiRI.-Sim (Positive Tilting)
2.38 0.08
1.44 0.03
67.2 4: 1.0
12.1 0.1.1.011
6.7 04
12,3 0.3
DT" (GT Initial Retur)
1.94 0.07
1.28
0.02
73.7 1.5
12.2 0.3
666 04
1..6
CiRL-Sim" (GT Initial Return)
1.97
0.08
1,30
0.03
71.1
12.2
1.12
7.2
13.1
Table 7: Multi-agent simulation results over 1000 test scenes with action temperature 1.5 over seeds. This table presents the results from the same experiments as Table 1, but with an action sampling temperature of 1.5 instead of 1.0. This allows for a comparison of the impact of the temperature hyperparameter. Overall, an action sampling temperature of 1.0 yields better results.
collision or rehicle-road-edge collision. For tilting, we uniformly sample Kiveh-veh ~ 24(2(225 0) and Ngoal Ld(25,, 0) when generating vehicle-vehicle collision scenarios, and uniformly sample Kveh-odge ud(-55, 0) and Kggaal 2d-25 0) when generating vehicle-road-edge collision scenar- ios. By additionally negatively tilting the goal, this grants the model more flexibility when generat- ing traffic violations as the agents are not trying to reach its prescribed goal. We collect 5000 sce- narios of each type of traffic violation derived from the training set, which comprises the simulated dataset of safety-critical scenarios. To encourage CtRL-Sim to learn how to generate safety-critical scenarios without forgetting how to generate good driving behaviour, we adopt the same finetuning strategy as Appendix H, except we randomly sample 90000 real training scenarios from the of- fline RL dataset each training epoch, 90% replay ratio. We find useful to use larger replay ratio when finetuning CtRL-Sim scenarios. The controllability results shown in Figure 9, demonstrating similar control over adversarial behaviours the CAT-finctunee CtRL-Sim model.
19

---

Vehicle-Vehicle Tilting
Vehicle-Edge Tilting
Goal Tilting
No Tilting
Positive Tilting
Figure 7: Qualitative results of the effects of positive tilting. We show the evolution of three traffic scenes with the top panels applying no exponential tilting to the CiRL-Sim-controlled agent (shown in teal) and the bottom panels applying positive tilting to the same CiRL-Sim-controlled agent. Bounding boxes outlined in red contain traffic violation. All other agents are set to log- replay through physics, with the agent interacting with the CtRL-Sim-contres controlld agent denoted in pink. Goals are denoted by small circles.
Tilting
Goal Success Rate (%)
Tilting
Collision Rate (%)
Tilting
Offfroad Rate (%)
0
69.0 0.7
14.5 1.5
15.2 0.9
(Late)
64.1 ::11::0 1.3
-10 (Late)
18.2 + 2.3
-10 (Late)
17.4 + 1.0
10
63.1 2.6
10
28.9 + 1.8
10
21.6 = 0.6
(a) Goal Tilting
(b) Schicle-Vehicle Tilting
(c) Vehicle-Edge Tilting
Table 8: We report the results of the CtRL-Sim FT model over 1000 test scenes applying (a) Goal Tilting, (b) Sehicle-Vehicle Tilting, and (c) Vehicle-Edge Tilting. We either apply tilting, -10 tilting, or 0 tilting for the first half of the rollout and -10 tilting for the second half of the rollout (i.e., Late). We report the mean + std over seeds.
M Dynamic Exponential Tilting
In this section, we experiment with dynamic exponential tilting, where the tilting value varies de- pending the simulation timestep. We specifically experiment with late filting to test if tilting is still effective at later timesteps. We applied -10 tilting to the CtRL-Sim FT model only during the last half (4 seconds) of the generated rollout. We evaluated over 5 seeds on 1000 scenes, as in Figure 4. We evaluate the tilting of each reward component independently. The results are shown in Table 8, where we report the md over 5 seeds.
As expected, with late tilting, we observe higher goal success rate and a lower collision/offroad rate compared to the model that applies negative tilting all timesteps. Conversely, goal success rate and higher collision/offroad rate compared to the model with no tilting. This
lower
20

---

Vehicle-Vehidle Tilting
Positive Tilting
Negative Tilting
Figure 8: Qualitative results of vehicle-vehicle tilting. We show the evolution of two traffic scenes with the top panels applying positive exponential tilting to the CrRL-Sim--ontrooleed agent (shown in teal) and the bottom panels applying negative tilting the same CiRL-Sim--controleed agent. Bounding boxes outlined in red contain a traffic violation. All other agents are set to log-replay through physics, with the agent interacting with the CiRL-Sim-controlled agent denoted in pink. Goals are denoted by small circles.
return
refor
Success Maaza
Rore
Rinetmne
Fineturod
0.20
0.75
OPfiadd
0.15
0.15
0.70
retum
0.10
0.10
-10
-25
Tibing SethiceWWeiidee
Tiking (Veticle Edpe)
Tilbing (Goall
Figure 9: Effects of exponential tilting. Comparison of CtRL--Sm base model (magenta) and CtRL-Sim model fine-tuned on adversarial CtRL-Sim scenarios (purple). As opposed to Figure 4, this fine-tuned model does not involve using CAT to select the adversarial scenarios. Rewards range from -25 to 25 for vehicle-vehicle collision (left), vehicle-edge collision (middle), and goal reaching (right). Results show smooth controllability, with fine-tuning enhancing this effect. Mean and std are reported over 5 seeds.
shows that tilting remains effective at the later timesteps. An interesting observation that late tilting performs similarly to full tilting of the goal reward, likely because avoiding the goal can be effectively achieved within the last 4 seconds of the rollout, even if the agent was initially traveling towards the goal. On the other hand, it may be more difficult for the agent to collide with another agent road edge in the last seconds if it hasn't 't planned for that in the first 4 seconds.
21

---

N
Failure Cases
Figure 10: Failure cases. Visualization of two failure cases of CtRL-Sim. The agent controlled by CtRL-Sim is depicted in teal. On the left, the agent drifts off the road, likely due to imitation drift. On the right, the agent's behaviour is erratic and implausible due extreme tilting values (-50).
In Figure 10, we visualize two failure cases generated by CtRL-Sim. In the scenario on the left, we control one agent using CtRL-Sim with zero tilting, indicated in green. However, the agent drifts off the road, exhibiting a failure mode likely caused by imitation drift-a common issue in imitation learning methods when rolled out over time. This drift likely results from the model's lack of exposure to sufficient examples of agents successfully recovering their position when approaching the edge of the road. In the scenario the right of Figure 10, control agent using CtRL-Sim with -50 vehicle-vehicle tilting. The agent's 's behaviour is erratic and implausible, which is caused by selecting large negative tilting value that pushes the model out of distribution. This failure mode can be mitigated by choosing tilting values that are not too extreme. We find empirically that tilting CtRL-Sim with values between -25 to 25 generally produce more plausible outputs.
22


Reviews:
[{"review_id": "6ARPyEPvnp", "reviewer": "Reviewer_GaAn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a return conditioned offline reinforcement learning (RL) approach, called \"CtRL-Sim\" to address the autonomous driving (AD) controllable simulated agents problem for the evaluation of autonomous vehicles (AV) planners. Compared to decision transformers (DT), CtRL-Sim predicts state sequences, and allows for modifying (\"tilting\") the rewards-to-go during the autoregressive inference process. The paper shows favorable quantitative results as well as qualitative examples to demonstrate the effectiveness of the proposed method. The paper is well-written, and easy to read with excellent clarity. Experiments are properly designed.", "summary_of_recommendation": "I recommend a weak accept. I may raise the rating depending on how well the questions are addressed in rebuttal.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "* Line 77, what’s the forward transition model? It is worth describing in the main text.\n* Line 99: “we also found it helpful to regularize the learned policy by predicting the full sequence of future states”, why predicting the full sequence regularizes the learned policy?\n* Line 152, \"Discretize the actions and return-to-gos into uniformly quantized bins.\" have the authors considered alternative tokenization methods?\n* Table 1, why is the per-scene gen. time of CtRL-Sim w/ GT initial return so high compared to w/o GT initial return? The Line 292 says “efficient” but 20s is very slow compared to imitation learning (IL), 3.4s.\n* Line 196, “The DT architecture is identical to that of CtRL-Sim except the return token precedes the state token” How does the order of return token and the state token matter?\n* Table 1, w/o access to privileged GT futures, IL seems to perform really well and also very fast. This result seems to suggest that the main benefit of CtRL-Sim is controllability. Is that correct?\n* Table 1, is it a fair comparison between methods w/ privileged GT futures and w/ privileged GT futures? Imitative metrics (ADE, FDE and Goal Success Rate) are closely related to privileged GT future information.\n* Line 240, how to choose the two agents in the two-agent interactive scenarios?\n* Fig 4, it seems that FT model generally performs worse than non-FT model. Why is that?\n* Limitations of the proposed method in this paper is not sufficiently discussed.\n* Overall, I'd be curious how does this reactive sim agents model compare to log-playback sim agents wrt evaluating AD planners.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NoThi45waj", "reviewer": "Reviewer_xpFT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on the generation of scenarios in driving simulation. Different from existing works that rely on heuristics or generative models of real-world data, the proposed method leverages return-conditioned offline reinforcement learning to efficiently generate reactive and controllable traffic agents. By controlling the desired returns for the various reward components, their method can generate a wide range of driving behaviors beyond the scope of the initial dataset. In addition, the authors also show that they can generate diverse and realistic safety-critical scenarios using adversarial generation.", "summary_of_recommendation": "My initial score is weak reject as I have some questions to be answered. Most of them are clarification questions so my final score will depend on the new information provided by the authors.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "1.\tThe definition of reward-to-go is not mentioned in the main context. I find the definition in the appendix, but I think the authors should at least mention what reward they used during the training. I think Figure 1 already shows all three rewards used. A justification for the selection of them will be better. The vehicle-to-vehicle and vehicle-to-edge factors seem to be carefully designed for the collision ratio and offroad ratio metrics. However, they are not sufficient conditions to show the realism of the scenario.\n2.\tIn Figure 3, some of the agents are controlled by CtRL-Sim and others are not. Will that cause collisions between controlled agents and replayed agents? It seems the scenario is not fully reactive. If all agents are controlled by CtRL-Sim, the computation could be a burden and it is also hard to select desired goals.\n3.\tIn Table 1, the result of Replay-Physics still has a 2.8% collision rate and 10.7% offroad rate. Does that mean the vehicle dynamics are not good enough?\n4.\tIt is not clear what kind of initial RTG is used for CtRL-Sim (Base). I assume it is using the max RTG. Then according to the results in Table 1, CtRL-Sim ∗ (GT Initial Return) looks better than CtRL-Sim (Base). Does this mean the RTG influences the results a lot and the RTG prediction in CtRL-Sim is not accurate?\n5.\tAccording to Table 2, CtRL-Sim achieves a low collision ratio without finetuning, which might be evidence to show that the controllability is largely limited by the dataset. In addition, after fine-tuning, CtRL-Sim FT has a higher collision ratio than the model without finetune even with tilt=50. Does that mean finetuning on adversarial scenarios sacrifices the quality of the model?\n6.\tIn DT, the input reward-to-go is decided by a linear scheduler. However, in the proposed method, it seems that the input reward-to-go is always the predicted one from the last step. Will the prediction error accumulate during the generation? Then does the tilting still work in later time steps? Some experimental results of the predicted error of RTG vs. timesteps could be helpful. \n7.\tIt would be interesting to show more tilting examples as in Figure 5.\n8.\tIf I understand correctly, the tilting is applied to the predicted RTG. This makes sense since some scenario initialization will not easily lead to collision and the predicted RTG provides a baseline to tune. However, this also raises a question. Should I tune the tilting value according to the specific scenario? In addition, the value may depend on the distribution of this initialization in the dataset rather than the fact. For example, some initialization may easily lead to collision but the collected samples do not contain such scenarios.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bb9xds4Eny", "reviewer": "Reviewer_KJao", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes Decision Transformer-based agent with variable weightings of objectives at test time for autonomous vehicle simulation. The core part of the proposal is to introduce factorized return-to-go training and inverse temperature weights to modify agent behavior at test time.", "summary_of_recommendation": "From an application point of view, this paper proposes a useful methodology in autonomous driving community. But, the manuscript needs more discssion to clarify the novelty of the proposed method in deep RL community.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "My main concern is that whether the proposed idea has novelty, compared to existing multi-objective deep RL research while I acknowledge its novelty as an application work. I'd like authors to add this discussion to Section 4. It'd be even nice if authors could add additional experimental results to include baselines based on an existing multi-objetive deep RL approach.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: This paper leverages return-conditioned offline reinforcement learning to generate reactive and controllable traffic agents from real-world driving data, providing fine-grained control and enabling diverse, realistic, and safety-critical scenarios in simulation.

Strength:
* The importance of generating reactive and controllable scenarios for simulation is well articulated.
* Detailed and thorough analysis of experimental results, including quantitative and qualitative examples.
* Demonstrates a practical way to apply decision transformer-style offline RL methods to autonomous driving simulation.

Weakness:
* The controllability of the generator is limited by the quality of the dataset.
* The design of the reward in the proposed method is not comprehensive enough.
* Details such as the definition of reward-to-go and goal selection during generation, are missing or insufficiently explained.
* The method's controllability is limited without fine-tuning, as evidenced by experiments with safety-critical scenario generation.

----

Post rebuttal:
The reviewers are happy with the authors' replies and have reached a consensus on accepting the paper.
Decision: Accept