CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning

Authors: luke  rowe
roger  girgis
anthony  gosselin
bruno  carrez
florian  golemo
felix  heide
liam  paull
christopher  pal
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-16 23:00:41

Abstract: Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through a physics-enhanced Nocturne simulator to generate a diverse offline reinforcement learning dataset, annotated with various reward terms. We then train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including adversarial behaviours. We demonstrate that CtRL-Sim can generate diverse and realistic safety-critical scenarios while providing fine-grained control over agent behaviours.

Main paper: 
 introduction: 
 Recent advances in autonomous driving has enhanced their ability to safely navigate the complexities of urban driving [1] . Despite this progress, ensuring operational safety in long-tail scenarios, such as unexpected pedestrian behaviours and distracted driving, remains a significant barrier to widespread adoption. Simulation has emerged as a promising tool for efficiently validating the safety of autonomous vehicles (AVs) in these long-tail scenarios. However, a core challenge in developing a simulator for AVs is the need for other agents within the simulation to exhibit realistic and diverse behaviours that are reactive to the AV, while being easily controllable. The traditional approach for evaluating AVs in simulation involves fixing the behaviour of agents to the behaviours exhibited in pre-recorded driving data. However, this testing approach does not allow the other agents to react to the AV, which yields unrealistic interactions between the AV and the other agents.To address the issues inherent in non-reactive log-replay testing, prior work has proposed rule-based methods [2, 3] to enable reactive agents. However, the behaviour of these rule-based agents often lacks diversity and is unrealistic. More recently, generative models learned from real-world data have been proposed to enhance the realism of simulated agent behaviours [4, 5, 6, 7, 8, 9] . While these methods produce more realistic behaviours, they are either not easily controllable [4, 5, 9] or require costly sampling procedures to control the agent behaviours [10, 8, 7, 11, 12] . Figure 1 : CtRL-Sim allows for controllable agent behaviour from existing datasets. This allows users to create interesting edge cases for testing and evaluating AV planners.In this paper, we propose CtRL-Sim to address these limitations of prior work. The CtRL-Sim framework utilizes return-conditioned offline reinforcement learning (RL) to enable reactive, closed-loop, controllable, and probabilistic behaviour simulation within a physics-enhanced Nocturne [13] environment. We process scenes from the Waymo Open Motion Dataset [14] through Nocturne to curate an offline RL dataset for training that is annotated with reward terms such as "vehicle-vehicle collision" and "goal achieved". We propose a return-conditioned multiagent autoregressive Transformer architecture [15] within the CtRL-Sim framework to imitate the driving behaviours in the curated dataset. We then leverage exponential tilting of the predicted return distribution [16] as a simple yet effective mechanism to control the simulated agent behaviours. While [16] exponentially tilts towards more optimal outcomes for the task of reward-maximizing control, we instead propose to tilt in either direction to provide control over both good and bad simulated driving behaviours.We show examples of how CtRL-Sim can be used to generate counterfactual scenes when exponentially tilting the different reward axes in Figure 1 . For controllable generation, CtRL-Sim simply requires specifying a tilting coefficient along each reward axis, which circumvents the costly iterative sampling required by prior methods. CtRL-Sim scenarios are simulated within our physics-extended Nocturne environment. We summarize our main contributions: 1. We propose CtRL-Sim, which is, to the best of our knowledge, the first framework applying return-conditioned offline RL for controllable and reactive behaviour simulation. Specifically, CtRL-Sim employs exponential tilting of factorized reward-to-go to control different axes of agent behaviours. 2. We propose an autoregressive multi-agent encoder-decoder Transformer architecture within the CtRL-Sim framework that is tailored for controllable behaviour simulation. 3. We extend the Nocturne simulator [13] with a Box2D physics engine, which facilitates realistic vehicle dynamics and collision interactions.We demonstrate the effectiveness of CtRL-Sim at producing controllable and realistic agent behaviours compared to prior methods. We also show that finetuning our model in Nocturne with simulated adversarial scenarios enhances control over adversarial behaviours. CtRL-Sim has the potential to serve as a useful framework for enhancing the safety and robustness of AV planner policies through simulation-based training and evaluation. 
ctrl-sim: 
 In this section, we present the proposed CtRL-Sim framework for behaviour simulation. We first introduce CtRL-Sim in the single-agent setting, and subsequently show how it extends to the multiagent setting. Given the state of an agent s t at timestep t and additional context (e.g., the road structure, the agent's goal), the behaviour simulation model employs a driving policy π(a t |s t , m, s G ) and a forward transition model P(s t+1 |s t , a t ) to control the agent in the scene. Note that a t is the action, m is the map context, and s G is the prescribed goal state. Using the physics-extended Nocturne simulator, we have access to a physically-realistic forward transition model P. In this work, we are interested in modelling the policy π(a t |s t , m, s G ) such that we can both imitate the real distribution of driving behaviour and control the agent's behavior to generate long-tail counterfactual scenes.We consider the common offline RL setup where we are given a dataset D of trajectories τ i = {. . . , s t , a t , r t , . . .}, with states s t ∈ S, actions a t ∈ A and rewards r t . These trajectories are generated using a (suboptimal) behaviour policy π B (a t |s t ) executed in a finite-horizon Markov [9] . Trajectories are arranged first by agents, then by timesteps, embedded, and fed through the decoder. For each agent, we encode (s t , G t , a t ) (i.e. state, return-to-go, action) and we predict from these (G t , a t , s t+1 , . . . , s T ). 2b (right) At inference time, the state predicts the return-to-go. The return-to-go is tilted (i.e., reweighed to encourage specific behaviors) and is used to predict the action, which in turn is used to predict the next states. decision process. The return-to-go at timestep t is defined as the cumulative sum of scalar rewards obtained in the trajectory from timestep t, G t = T t ′ =t r t ′ . The objective of offline RL is to learn policies that perform as well as or better than the best agent behaviours observed in D.The primary insight of this work is the observation that offline RL can be an effective way to perform controllable simulation. That is, the policy distribution over actions can be tilted at inference time towards desirable or undesirable behaviors by specifying different values of return-to-go G t . This requires a different formulation of the policy such that it is conditioned on the return π(a t |s t , G t , s G ) 2 . In Table 3 , we outline how different approaches in offline RL have learned return-conditioned policies. In this work, we adopt an approach that learns the joint distribution of returns and actions of an agent in a given dataset. Specifically, p θ (a t , G t |s t , s G ) = π θ (a t |s t , s G , G t )p θ (G t |s t , s G ). We note that [17] found it helpful to also utilize a model-based return-conditioned policy, whereby the future state is modelled as part of the joint distribution being learned. This is shown to provide a useful regularizing signal for the policy, even though the future state prediction is not directly used at inference time. In this work, we also found it helpful to regularize the learned policy by predicting the full sequence of future states. The final distribution we are aiming to model is thus given by pθ (s t+1:T , a t , G t |s t , s G ) = p θ (s t+1:T |s t , s G , G t , a t )π θ (a t |s t , s G , G t )p θ (G t |s t , s G ).At inference time, we obtain actions by first sampling returns G t ∼ p θ (G t |s t , s G ) and then sampling actions a t ∼ π θ (a t |s t , s G , G t ). This sampling procedure corresponds to the imitative policy since the sampled returns are obtained from the learned density that models the data distribution. Following prior work in offline RL [16, 17, 18] , we can also sample actions from an exponentially-tilted policy distribution. This is done by sampling the returns from the tilted distributionG ′ t ∼ p θ (G t |s t , s G ) exp(κG t ), with G ′ t being the tilted return-to-go and where κ represents the inverse temperature; higher values of κ concentrate more density around the best outcomes or higher returns, while negative values of κ concentrate on less favourable outcomes or lower returns.We are interested in modelling and controlling the individual components of the reward function rather than maximizing their weighted sum. For example, we would like to model an agent's ability to reach its goal, drive on the road, and avoid collisions. In general, given C reward components, our objective is to learn policies that are conditioned on all its factored dimensions as this would grant us control over each one at test time. This entails modelling separate return components as G c t ∼ p θ (G c t |s t , s G ) for each return component c. Applying this factorization, we reformulate the learned policy to explicitly account for the conditioning on all return components π θ (a t |s t , s G , G 1 t , . . . G C t ). At test time, each return component will be accompanied by its own inverse temperature κ c to enable control over each return component, which enables sampling actions that adhere to different behaviours specified by {κ 1 , . . . , κ C }, as shown in Algorithm 2 in Appendix B.To implement our framework for behaviour simulation, we extend the approach presented above to the multi-agent setting. Across all agents we have sets for the joint states S t , goal states S G , actions A t , and returns-to-go G t . The final multi-agent joint distribution we model is:EQUATIONwhere the returns and actions from the previous timesteps are shared across agents, while at the present timestep they are masked out so one can only observe one's own return and action.In this section, we introduce the proposed architecture for multi-agent behaviour simulation within the CtRL-Sim framework that parameterizes the multi-agent joint distribution presented in Equation (1) . We propose an encoder-decoder Transformer architecture [19] , as illustrated in Figure 2 , where the encoder encodes the initial scene and the decoder autoregressively generates the trajectory rollout for all agents in the scene.Encoder To encode the initial scene, we first process the initial agent states and goals (s 0 , s G ) and the map context m, where s 0 is the joint initial state of all agents and s G is the joint goal state of all agents. Each agent i's initial state information s i 0 , which includes the position, velocity, heading, and agent type, is encoded with an MLP. Similarly, each agent's goal s i G , which is represented as the ground-truth final position, velocity, and heading, is also encoded with an MLP. We then concatenate the initial state and goal embedding of each agent and embed them with a linear layer to get peragent embeddings of size d. We additionally apply an additive learnable embedding to encode the agents' identities across the sequence of agent embeddings. The map context is encoded using a polyline map encoder, detailed more fully in Appendix F, which yields L road segment embeddings of size d. The initial agent embeddings and road segment embeddings are then concatenated into a sequence of length N + L and processed by a sequence of E Transformer encoder blocks.Decoder The proposed decoder architecture models the joint distribution in Equation (1) as a sequence modelling problem, where we model the probability of the next token in the sequence conditioned on all previous tokens p θ (x t |x <t ) [15] . In this work, we consider trajectory sequences of the form:x = ⟨. . . , (s 1 t , s 1 G ), (G 1,1 t , . . . , G C,1 t ), a 1 t , ..., (s N t , s N G ), (G 1,N t , . . . , G C,N t), a N t , . . . ⟩. These sequences are an extension of the sequences considered in the Multi-Game Decision Transformer [16] to the multi-agent goal-conditioned setting with factorized returns. Unlike Decision Transformer [15] , our model predicts the return distribution and samples from it at inference time, which enables flexible control over the agent behaviours and circumvents the need to specify an expert return-to-go. We obtain state-goal tuple (s i t , s i G ) embeddings in the same way that (s 0 , s G ) are processed in the encoder. Following recent work that tokenizes driving trajectories [20, 9] , we discretize the actions and return-to-gos into uniformly quantized bins. We then embed the action and return-to-go tokens with a linear embedding. To each input token, we additionally add two learnable embeddings representing the agent identity and timestep, respectively. The tokenized sequence is then processed by D Transformer decoder layers with a temporally causal mask that is modified to ensure that the model is permutation equivariant to the agent ordering (see Appendix F for details).Training Given a dataset of offline trajectories (Section 3), we train our model by sampling sequences of length H × N × 3, where H is the number of timesteps in the context. The state, return-to-go, and action token embeddings output by the decoder are used to predict the next return token, action token, and future state sequence, respectively. We train the return-to-go and action headers with the standard cross-entropy loss function and the future state sequence header with an L2 regression loss function. The final loss function is of the form: L = L action +L return-to-go +αL state . 3 Experiments 
experimental setup: 
 Offline RL Dataset We curate an offline RL dataset derived from the Waymo Open Motion Dataset (WOMD) [14] . We extend Nocturne by integrating a physics engine based on the Box2D library for enabling realistic vehicle dynamics and collisions, detailed in Appendix C.1. Each scene in the Waymo dataset is fed through the physics-enhanced Nocturne simulator to compute the per-timestep actions and factored rewards for each agent. The factored rewards comprise of a goal position, vehicle-vehicle collision, and a vehicle-road-edge collision reward, detailed in Appendix C.2.Evaluation We evaluate CtRL-Sim on its ability to replicate the driving behaviours found in the Waymo Open Motion Dataset (imitation) and generate counterfactual scenes that are consistent with specified tilting coefficients (controllability). For both modes of evaluation, we use 1 second of history and simulate an 8 second future rollout. For imitation, we evaluate on up to 8 moving agents per scene that we control with CtRL-Sim, where the remaining agents are set to log replay through physics. We evaluate on 1000 random test scenes in both modes of evaluation. Following recent work [7] , we use three types of metrics for imitation evaluation: reconstruction metrics, such as Final Displacement Error (FDE), Average Displacement Error (ADE), and Goal Success Rate; a distributional realism metric (JSD) defined by the mean of the Jensen-Shannon Distances computed on linear speed, angular speed, acceleration, and distance to nearest vehicle features between real and simulated scenes; and common sense metrics measured by Collision and Offroad rate.For controllability evaluation, we evaluate on 1 selected "interesting" interactive agent that is controlled by CtRL-Sim, defined as an agent who is moving and whose goal is within 10 metres of another moving agent. All agents except for the CtRL-Sim-controlled interesting agent are set to log replay through physics. We evaluate the model's controllability through metrics aligned with the specified reward dimensions: we report the goal success rate for the goal reward control, collision rate for the vehicle-vehicle reward control, and offroad rate for the vehicle-road-edge reward control. show smooth controllability, with fine-tuning enhancing this effect. We report mean±std over 5 seeds.For imitation evaluation, we compare CtRL-Sim against several relevant baselines: 1. Replay-Physics employs an inverse bicycle model to obtain the ground-truth log-replay actions and executes through the simulator. 2. Actions-Only is an encoder-decoder model inspired by [9] where the decoder trajectory sequences only contain actions. The finetuned model takes a trained base model and finetunes it on a dataset of simulated long-tail scenarios that we collect using an existing simulated collision generation method CAT [21] . This allows CtRL-Sim to be exposed to more long-tail collision scenarios during training, as the WOMD mainly contains nominal driving. We refer readers to Appendix H for details of CAT and our proposed finetuning procedure. control due to the prerequisite knowledge about the return-to-go values and an absence of an interpretable mechanism for behaviour modulation. By constrast, the exponential tilting employed in CtRL-Sim has a clear interpretation: negative exponential tilting yields behaviours that are worse than the average behaviours learned from the dataset, while positive exponential tilting yields betterthan-average behaviours. We show the results of our controllability evaluation in Figure 4 . For each reward dimension c, we exponentially tilt κ c between -25 and 25 and observe how this affects the corresponding metric of interest. We also show the results of DT when conditioning on the minimum and maximum possible return. For both the base and finetuned CtRL-Sim models, we observe a relatively monotonic change in each metric of interest as the tilting coefficient is increased. As the finetuned model is exposed to collision scenarios during finetuning, it demonstrates significant improvements over the base model in generating bad driving behaviours. Specifically, at -25 tilting, the finetuned model is able to generate 2.1× as many collisions and 1.8× as many offroad violations as the base model. Figure 5 (and 7, 8 in Appendix J) shows qualitatively the effects of tilting.Table 2 evaluates CtRL-Sim's ability to produce adversarial agents that collide with a data-driven planner. We evaluate on a held-out test set of two-agent interactive scenarios from the Waymo interactive dataset, where one interacting agent is controlled by the adversary and the other is controlled by the planner. We use a positively-tilted CtRL-Sim base model as our planner, due to its demonstrated ability to produce good driving behaviours in Table 1 . For adversarial scenario generation, we compare the base CtRL-Sim model against the CtRL-Sim FT model. With -10 tilting applied, the finetuned model generates 238 more collisions with the planner than the base model over 1000 scenes, which we attribute to its exposure to simulated collision scenarios during finetuning. Notably, CtRL-Sim FT was finetuned in only 30 minutes on 1 NVIDIA A100-Large GPU and only 3500 CAT scenarios. This underscores CtRL-Sim's capability to flexibly incorporate data from various sources through finetuning, thereby enabling the generation of new kinds of driving behaviours. Importantly, after finetuning, CtRL-Sim FT largely retains its ability to produce good driving behaviours. This is evidenced by a 21.1 percentage point decrease in the planner's collision rate when using a +50 positively tilted finetuned model as the adversary. We also compare CtRL-Sim against a state-of-the-art collision generation method, CAT [21] , which uses a motion prediction model to select plausible adversarial trajectories that overlap with the ego plan. Although CAT generates more collisions, CAT is not controllable as it can't control how adversarial the agents are, and CAT agents are non-reactive to the ego's actions as the trajectory is fixed prior to the simulation, limiting its realism. This is evidenced by a larger adversary collision speed than all finetuned CtRL-Sim models and is also validated qualitatively in the supplementary video. We further conduct a user study to confirm that CtRL-Sim adversarial scenarios are indeed more realistic than CAT adversarial scenarios, with details and results reported in Appendix I. 
related work: 
 Agent behaviour simulation involves modelling the behaviour of other agents in simulation, such as vehicles and pedestrians. Agent behaviour simulation methods can be categorized into rule-based and data-driven methods. Rule-based methods rely on human-specified rules to produce plausible agent behaviours, such as adhering strictly to the center of the lane [2, 3] . These methods often yield unrealistic behaviours that fail to capture the full spectrum of driving behaviours. To address these limitations, prior work has proposed learning generative models that aim to replicate agent behaviours found in real-world driving trajectory datasets [22, 23, 24, 4, 5, 25, 9] . These approaches draw inspiration from methods for the task of joint motion prediction [26, 27, 28, 29, 30] ; however, it's crucial to distinguish that, unlike the open-loop nature of joint motion prediction, behaviour simulation operates closed-loop [31] . To improve the realism of the learned behaviours, other work has proposed using adversarial imitation learning [32] to minimize the behavioural discrepancy between expert and model rollouts [33, 34, 6] or RL to improve traffic rule compliance [35, 36] . While such methods demonstrate improved realism over rule-based methods, they lack the necessary control over the behaviours to enable the generation of targeted simulation scenarios for AV testing.More recent work has proposed more controllable behaviour simulation models by learning conditional models [10, 37, 7, 38, 8, 11, 12] that enable conditioning on a high-level latent variables [10, 37] , route information [7] , or differentiable constraints [8, 11, 39, 12, 40] . More recently, [41] used retrieval augmented generation to generate controllable traffic scenarios. However, these methods either lack interpretable control over the generated behaviours [37] or require costly testtime optimization procedures to steer the generated behaviours, such as latent variable optimization [10] , Bayesian optimization [42, 43, 7] , or the simulation of expensive diffusion processes [8, 44, 11, 39, 12, 40] . In contrast, CtRL-Sim offers a more efficient alternative and learns a conditional multi-agent behaviour model that conditions on interpretable factorized returns, thereby eliminating the need for costly test-time optimization. By exponentially tilting the predicted return distribution [16] at test time, CtRL-Sim enables efficient, interpretable, and fine-grained control over agent behaviours while being grounded in real-world data. 
conclusion: 
 We presented CtRL-Sim, a novel framework applying offline RL for controllable and reactive behaviour simulation. Our proposed multi-agent behaviour Transformer architecture allows CtRL-Sim to employ exponential tilting at test time to simulate a wide range of agent behaviours. We present experiments showing the effectiveness of CtRL-Sim at producing controllable and reactive behaviours, while maintaining competitive performance on the imitation task compared to baselines.Limitations The learned policies produced by CtRL-Sim in its current form may make large-scale RL training on CtRL-Sim scenarios prohibitively expensive. However, we believe that further optimizations of the model and simulator such as exploring more compact models, utilizing model quantization or Flash Attention [45] , reducing the planning frequency, or making the simulator GPU-accelerated can further improve the inference latency. We leave this for future work. Furthermore, CtRL-Sim enables control over the behaviours, but not the initial agent placements. We believe that a generative model that supports controlling both the initial agent placements and the agent behaviours is promising future work to explore. Table 3 : Offline policy modelling approaches in prior work. We can see that methods differ in the decomposition of the joint distribution over actions and returns, with some approaches utilizing state prediction as a regularizer. We note that this table is adopted from prior work [18, 17] . 
 
 Appendix: We position CtRL-Sim within the field of offline RL. Table 3 presents the different ways explored in the literature for learning policies in offline RL, and how these methods can sample returnmaximizing actions at test time. Prior methods in offline RL differ in how the policy is modelled during training and how inference is performed; we refer the reader to Table 3 in the Appendix A for a breakdown of the different approaches. As CtRL-Sim employs a return-conditioned policy for controllable simulation, we briefly present a class of related methods that learn return-conditioned policies (RCPs) [15, 16, 46, 47, 48, 18, 49] for offline RL. RCPs are concerned with learning the joint distribution of actions and returns, such that action sampling is conditioned on the return distribution. Instead of modelling the return distribution, the Decision Transformer [15] conditions the learned policy on the maximum observed return, G max in the dataset. That is, at inference time, the actions are sampled from a t ∼ p θ (a t |s t , G max ). Lee et al. [16] instead propose to employ the learned return distribution, combined with exponential tilting, in order to sample high-return actions while remaining close to the empirical distribution. CtRL-Sim adopts exponential tilting of the predicted return distribution for each agent to finely control agent behaviour. Additionally, CtRL-Sim explicitly models the future sequences of states.Algorithm 2 describes the proposed action sampling procedure for controllable behaviour generation with factorized exponential tilting.The action sampling algorithm used by CtRL-Sim to allow for factorized tilting of the exhibited behaviour.1: Input: {κ 1 , . . . , κ C } ▷ The specified inverse temperature for each return-to-go component. 2: for c = 1 to C do 3:G ′c t ∼ p θ (G c t |s t , s G ) exp(κ c G c t ) 4: end for 5: a t ∼ π θ (a t |s t , s G , G ′1 t , . . . G ′C t ) 6: return a tCtRL-Sim extends the Nocturne simulation environment [13] . Nocturne is a lightweight 2D driving simulator that is built on real-world driving trajectory data from the Waymo Open Motion Dataset [14] . A scene in Nocturne is represented by a set of dynamic objects -such as vehicles, pedestrians, and cyclists -and the map context, which includes lane boundaries, lane markings, traffic signs, and crosswalks. Each dynamic object is prescribed a goal state, which is defined as the final waypoint in the ground-truth trajectory from the Waymo Open Motion Dataset. If there exist missing timesteps in the ground-truth trajectory, we re-define the goal as the waypoint immediately preceding the first missing timestep. By default, the dynamic objects track its 9 second trajectory from the Waymo Open Motion Dataset at 10 Hz. The Nocturne Simulator is originally designed for the development of RL driving policies, where the first 10 simulation steps (1s) of context is provided and the RL agent must reach the prescribed goal within the next 80 simulation steps (8s).We extend Nocturne by integrating a physics engine based on the Box2D library for enabling realistic vehicle dynamics and vehicle collisions. We model the vehicle's dynamics using basic physics principles, where forces applied to the vehicle are translated into acceleration, influencing its speed and direction, and with frictional forces applied to simulate realistic sliding and adherence behaviors. This extension additionally ensures that an agent's acceleration, braking, and turn radius are bound by plausible limits and that vehicles can physically collide with each other. Such improvements open the possibility of more accurately simulating complex conditions, such as emergency braking maneuvers, slippery roads, and multi-vehicle collisions.The actions are defined by the acceleration and steering angle and the reward function is decomposed into three components: a goal position reward, a vehicle to vehicle collision reward, and a vehicle to road edge collision reward. We confirm that the trajectory rollouts obtained by feeding Waymo scenes through the simulator attain a reasonable reconstruction of the ground-truth Waymo trajectories (see Table 1 ). Following Nocturne [13] , we omit bicyclist and pedestrian trajectories from the Waymo Open Motion Dataset and we omit scenes containing traffic lights. This yields a training, validation, and test set containing 134150, 9678, and 2492 scenes.For each agent, to obtain the action a t at timestep t, we compute the acceleration and steering value using an inverse bicycle model computed from the agent's current state in the simulator ŝt and the ground-truth next state from the trajectory driving log s t+1 . We clip accleration values between -10 and 10 and steering values between -0.7 and 0.7 radians. We then execute a t with our proposed forward physics dynamics model to obtain the agent's updated state ŝt+1 , and we repeat until the agent has completed the full rollout. Table 1 confirms that this approach to offline RL trajectory data collection yields a reasonable reconstruction of the ground-truth driving trajectories.We compute rewards at each timestep, where our reward function is factored into three rewards components: a goal position reward, vehicle-vehicle collision reward, and vehicle-road-edge collision reward. We chose the reward functions based on the information provided by the Nocturne simulator, which includes indicators for goal success, vehicle-vehicle collisions, and vehicle-edge collisions. For each agent, their goal is set to the final state (i.e., position, heading, and velocity) of the ground-truth logged trajectory. The goal position reward is defined by: R g (s t , s G ) = 1 goal achieved (s t , s G ), where goal achieved(•) is 1 if the agent ever reaches within 1 metre of the ground-truth goal, and 0 otherwise. The vehicle-vehicle collision reward is defined by: R v (s t , S t -{s t }) = -10 × 1 vehicle-vehicle collision (s t , S t -{s t })EQUATION,where dist-nearest-vehicle(•) computes the distance between the agent of interest and its nearest agent in the scene. Finally, the vehicle-road-edge collision reward is defined by: R e (s t , m) = -10 × 1 vehicle-road-edge collision (s t , m) + min(dist-nearest-road-edge(s t , m), 5) 5 ,where dist-nearest-road-edge(•) computes the distance between the agent and the nearest road edge.The goal success rate is the proportion of evaluated agents across the evaluated test scenes that get within 1 metre of the ground-truth goal position at any point during the trajectory rollout. The final and average displacement errors are calculated for all evaluated agents across the test scenes and averaged. For a specific scene s, the collision rate and offroad rate of s are the proportion of evaluated agents in s that collide with another agent or road edge, respectively. These rates are then averaged across all tested scenes to define the overall collision and offroad rates.We compute the Jensen Shannon Distance (JSD) between the distributions of features computed from the real and simulated rollouts. The Jensen Shannon Distance between two normalized histograms p and q is computed as:D KL (p||m) + D KL (q||m) 2 ,where m is the pointwise mean of p and q and D KL is the KL-divergence. Unlike prior works that compute the Jensen Shannon Divergence [6, 7] , we compute its square root -the Jensen Shannon Distance -so that values are not too close to 0. We compute the JSD over the following feature distributions: linear speed, angular speed, acceleration, and nearest distance. Since the acceleration values are discrete, for the acceleration JSD, we define one histogram bin for each valid acceleration value, yielding 21 evenly spaced bins between -10 and 10. For the linear speed histogram, we use 200 uniformly spaced bins between 0 and 30. For the angular speed JSD, we use 200 uniformly spaced bins between -50 and 50. For the nearest distance JSD, we use 200 uniformly spaced bins between 0 and 40.In Table 4 , we report the per-feature JSD results for Table 1 .Training: The CtRL-Sim behaviour simulation model is trained using randomly subsampled sequences of length of H × N × 3, where H = 32 and N = 24. For the actions, we discretize the acceleration and steering into 20 and 50 uniformly quantized bins, respectively, yielding 1000 action tokens. For the return-to-gos, we discretize each return-to-go component G c,i t into 350 uniformly quantized bins. All agents and the map context are encoded in global frame as in [28, 9] where we center and rotate the scene on a random agent during training. The map context is represented as a set of road segments m := {r l } L l=1 , where each road segment is defined by a sequence of points r l := (p 1 l , . . . p P l ), where L is the number of road segments and P is the number of points per road segment. We apply a per-point MLP to the points of each road segment r l . To produce road segment-level embeddings, we then apply attention-based pooling [50] on the embeddings of the points within each road segment, yielding L road segment embeddings of size d. We select the L = 200 closest lane segments within 100 metres of the centered agent as the map context, and select up to N = 24 closest agents within 60 metres of the centered agent as social context for the model. For each lane segment, we subsample P = 100 points. We use a hidden dimension size d = 256, where we use E = 2 Transformer encoder blocks and D = 4 Transformer decoder blocks, and we set α = 1 100 in the loss function. We supervise our model only on the trajectories of moving agents. We found it useful to employ goal dropout whereby the embeddings for 10% of agent goals are randomly set to 0 to prevent the model from overrelying on the goal information. We found goal dropout useful for learning an informative map representation. The state, return, and action embeddings for the missing timesteps are set to 0. To ensure that the model is permutation equivariant to the agent ordering [28, 27] , we modify the standard temporally causal mask by additionally enforcing that each agent can only attend to its own action and return-to-go tokens at the present timestep while allowing access to all agents' state tokens at the present timestep and all agents' tokens in the past timesteps. The CtRL-Sim model is trained using a linear decaying learning rate schedule from 5e-4 for 200k steps using the AdamW optimizer and a batch size of 64. At inference, we sample actions with a temperature of 1.5. The CtRL-Sim architecture comprises 8.3 million parameters that we train in 20 hours with 4 NVIDIA A100 GPUs.Inference: CtRL-Sim supports scenes with an arbitrary number of agents. As CtRL-Sim is trained with up to N = 24 agents, when the number of CtRL-Sim-controlled agents at inference time exceeds N = 24, we iteratively select 24-agent subsets at each timestep for processing until all agents have been processed. We first randomly select a CtRL-Sim-controlled agent, we normalize the scene to this agent and select the 23 closest context agents to the CtRL-Sim-controlled agent to comprise the first set of 24 agents. We then iteratively continue centering on a CtRL-Sim-controlled agent that has not been processed in the previous sets of 24 agents and select its 23 closest agents for context until all CtRL-Sim-controlled agents have been included in a 24-agent subset. If an agent belongs to multiple 24-agent subsets, we use the model's first prediction of that agent. At inference time, the context length is set to training context length H = 32. At each timestep, we select H = 32 most recent timesteps as context and we found it useful to always center and rotate the scene on the centered agent at the oldest timestep in the context. For the first 10 timesteps (1s) of the simulated rollout, the states and actions are fixed to the ground-truth states and actions from the offline RL dataset, whereas the return-to-go is predicted at every timestep of the simulated rollout.In this section, we describe the design decision of each baseline employed in our work. We note that for all models below, we scaled them in order for all models to have approximately the same number of learnable parameters as CtRL-Sim's architecture.The actions-only baseline is encoder-decoder architecture implemented in exactly the same way as CtRL-Sim with a few ablations. These include removal of states and returns from the decoder sequence, and no state rollout predictions. This model was inspired by [9] but differs in that the model also has access to the agents' goals.The imitation learning baseline is also based on the CtRL-Sim multi-agent behaviour simulation architecture but lacks factorized return information. It is a step better than the actions-only baseline since it considers the states in the decoder sequence, and this is corroborated by its improved performance on multi-agent simulation results of Table 1 .DT The Decision Transformer baseline is based on the seminal work [15] . We adopt an identical architecture to CtRL-Sim's with some minor difference based on the algorithm. One such decision is the lack of a return prediction based on states, and instead returns are chosen at inference time based on domain knowledge. Returns are the first token fed to the decoder, followed by states in order to predict actions. We make the strong argument that this is suboptimal for controllability (results in Figure 4 ) and does not provide intuitive mechanism for selecting the return values to target. Table 5 : Ablations of CTG++ on Multi-agent simulation results over 1000 test scenes. We report the results across all metrics of different configurations of the CTG++ baseline [11] . † indicates the original model results, with 256 hidden dimension and 50 diffusion steps at inference time.CTG++ The CTG++ baseline is a diffusion model reimplementation of the recent work by [11] .We attempted to follow the architecture as closely as possible with a few minor differences. One such difference is that we diffuse over both states and actions, rather than diffusing over only actions and using an unicycle dynamics model to derive the states. We chose this approach because the underlying dynamics of the physics-enhanced Nocturne simulator is not necessarily governed by a unicycle dynamics model, and thus using a unicycle dynamics model would induce small errors in the derived states during training. We further note that we cannot replace the unicycle dynamics model with the Nocturne physics dynamics model as this forward model is not differentiable. We also condition on the present timestep and goal, to ensure fair comparison with CtRL-Sim. We note that at scene generation time, we diffuse over actions and states at a rate of 2 Hz which is consistent with the original CTG++ model. In addition, although we train with 100 diffusion steps, we run evaluations with 50 diffusion steps. As showing in Table 5 , the difference in performance is insignificant. As we do not have information on the size of the network used in the original manuscript, we explored different hidden dimension sizes of the transformer architecture of the diffusion model, also shown in Table 5 . A final difference is the future relative encoding. While CTG++ use the ground-truth to compute the relative encoding during training and a constant velocity model at test time, we opted to use the final historical timestep's relative encoding. We found this approach to be more stable.The Waymo Open Motion dataset largely contains nominal driving scenes. To enhance control over the generation of safety-critical scenarios, we finetune CtRL-Sim on a simulated dataset of safetycritical scenarios generated by CAT [21] . CAT is a state-of-the-art collision generation method that involves fixing the agent's future trajectory to a trajectory predicted by a DenseTNT trajectory predictor. CAT searches for a trajectory that has high likelihood of colliding with the log-replay future trajectory of the ego vehicle, while having high probability under the behaviour prior (DenseTNT).For more details, we refer readers to [21] . We note that a limitation of CAT is that the agent is nonreactive to the ego as the agent's trajectory is fixed at the beginning of the simulation. Moreover, unlike CtRL-Sim, CAT does not have control over the degree to which the agent is adversarial.To collect the simulated safety-critical dataset, we run CAT on a subset of the interactive validation split of the Waymo Open Motion Dataset, which involves two interacting agents. Following CAT, we select one of the two interacting agents to be the ego (whose trajectory is fixed to the log-replay trajectory) and the other interacting agent to be the CAT adversary. In total, we collect 3577 CAT scenarios for finetuning, of which around 60% contain ego-adversary collisions.To encourage CtRL-Sim to learn how to generate safety-critical scenarios without forgetting how to generate good driving behaviour, we adopt a continual pre-training strategy for finetuning [51] where we randomly sample 3577 real training scenarios from the offline RL dataset in each training epoch, or a 50% replay ratio. We rewarm the learning rate to the maximum learning rate of 5e-4 over 500 steps and follow a linear decay learning rate schedule to 0 over 20 epochs. We expect that the finetuned CtRL-Sim model will be more capable of generating long-tail scenarios as it is more exposed to such scenarios during finetuning. Finetuning takes roughly 30 minutes on 1 NVIDIA A100-Large GPU.Method Times Preferred CtRL-Sim 123 CAT 69 Tie 79Table 6 : The tally of votes for the larger study. We show the breakdown of the votes for the conducted larger study. We observe that the results of this study are less conclusive than the pilot study.We conduct a user study that contained a total of 24 participants to evaluate which method (CtRL-Sim vs. CAT) generates more plausible adversarial behaviours. We did not record any identifying information of the participants, and participants were invited on a voluntary basis. The user study contained a total of 271 paired scenarios, where each paired scenario consisted of two interacting agents, one controlled by a positively-tilted CtRL-Sim planner and the other controlled by an adversary. The scenario conditions of the paired scenarios are identical, except one scenario employed a CtRL-Sim adversarial agent and the other employed a CAT adversarial agent. The adversarial agent in each paired scenario is highlighted in pink so that users can distinguish the adversarial agent from the remaining agents in the scene. We do not identify the planner agent as we found this to detract users' attention from the behaviour of the adversarial agent. Each participant was tasked with evaluating a randomly selected set of 30 paired scenarios from the pool of 271 paired scenarios. For each paired scenario, users selected in which scenario the adversarial agent was more realistic, with the option to select "Tie" if the two scenarios were sufficently similar. Users were presented with the user interface shown in Figure 6 , where we randomize the order between both videos. We use the same positively tilted CtRL-Sim planner in both of the paired scenarios. Table 6 shows the tally of the user study. The results indicate convincingly that CtRL-Sim produces more plausible adversarial behaviours than CAT based on the participants' preferences, as CtRL-Sim was preferred 54 more times than CAT over the 271 paired scenarios.Figure 7 shows more qualitative examples demonstrating the effects of positive exponential tilting on each of the three reward components. In the left panels, CtRL-Sim with no tilting produces a vehicle-vehicle collision between two interacting agents at a left-turn. With positive vehicle-vehicle tilting, the CtRL-Sim-controlled agent moves more to the right-hand side of the lane to avoid the collision. In the middle panels, CtRL-Sim with no tilting produces a vehicle-edge collision as the bus pulls into the curb. With positive vehicle-edge tilting, the CtRL-Sim-controlled agent pulls into the curb at a safer distance from the curb. In the right panels, CtRL-Sim with no tilting reaches the goal. With positive goal tilting, the CtRL-Sim-controlled agent reaches the goal much faster and nearly avoids collision with the turning vehicle. In Figure 8 , we show two more examples of adversarial collision scenarios generated with negative vehicle-vehicle tilting. We refer the interested reader to the supplementary video for more examples.In Table 7 , we report results from the same experiments as Table 1 except with a higher action sampling temperature, set to 1.5.Instead of finetuning on CAT scenarios, we explore finetuning CtRL-Sim on adversarial scenarios generated by CtRL-Sim. We first collect a simulated dataset of scenes either containing a vehicle-vehicle collision or an offroad infraction. Specifically, we generate rollouts of a single agent with the negatively tilted base CtRL-Sim model where the other agents are set to log replay through physics, and we save the scenario only if the generated rollout yields a vehicle-vehicle 1 , but with an action sampling temperature of 1.5 instead of 1.0. This allows for a comparison of the impact of the temperature hyperparameter. Overall, an action sampling temperature of 1.0 yields better results.collision or vehicle-road-edge collision. For tilting, we uniformly sample κ veh-veh ∼ U(-25, 0) and κ goal ∼ U(-25, 0) when generating vehicle-vehicle collision scenarios, and we uniformly sample κ veh-edge ∼ U(-25, 0) and κ goal ∼ U(-25, 0) when generating vehicle-road-edge collision scenarios. By additionally negatively tilting the goal, this grants the model more flexibility when generating traffic violations as the agents are not trying to reach its prescribed goal. We collect 5000 scenarios of each type of traffic violation derived from the training set, which comprises the simulated dataset of safety-critical scenarios. To encourage CtRL-Sim to learn how to generate safety-critical scenarios without forgetting how to generate good driving behaviour, we adopt the same finetuning strategy as in Appendix H, except we randomly sample 90000 real training scenarios from the offline RL dataset in each training epoch, or a 90% replay ratio. We find it useful to use a larger replay ratio when finetuning on CtRL-Sim scenarios. The controllability results are shown in Figure 9 , demonstrating similar control over adversarial behaviours as the CAT-finetuned CtRL-Sim model. shows that tilting remains effective at the later timesteps. An interesting observation is that late tilting performs similarly to full tilting of the goal reward, likely because avoiding the goal can be effectively achieved within the last 4 seconds of the rollout, even if the agent was initially traveling towards the goal. On the other hand, it may be more difficult for the agent to collide with another agent or a road edge in the last 4 seconds if it hasn't planned for that in the first 4 seconds.Positive Tilting Negative TiltingNote that we omit the additional context m for brevity. 


Decision: Summary: This paper leverages return-conditioned offline reinforcement learning to generate reactive and controllable traffic agents from real-world driving data, providing fine-grained control and enabling diverse, realistic, and safety-critical scenarios in simulation.

Strength:
* The importance of generating reactive and controllable scenarios for simulation is well articulated.
* Detailed and thorough analysis of experimental results, including quantitative and qualitative examples.
* Demonstrates a practical way to apply decision transformer-style offline RL methods to autonomous driving simulation.

Weakness:
* The controllability of the generator is limited by the quality of the dataset.
* The design of the reward in the proposed method is not comprehensive enough.
* Details such as the definition of reward-to-go and goal selection during generation, are missing or insufficiently explained.
* The method's controllability is limited without fine-tuning, as evidenced by experiments with safety-critical scenario generation.

----

Post rebuttal:
The reviewers are happy with the authors' replies and have reached a consensus on accepting the paper.

Reviews: review: title: title: Good paper. Solid technical contributions. Well-written. Properly designed experiments.
paper_summary: paper_summary: The paper proposes a return conditioned offline reinforcement learning (RL) approach, called "CtRL-Sim" to address the autonomous driving (AD) controllable simulated agents problem for the evaluation of autonomous vehicles (AV) planners. Compared to decision transformers (DT), CtRL-Sim predicts state sequences, and allows for modifying ("tilting") the rewards-to-go during the autoregressive inference process. The paper shows favorable quantitative results as well as qualitative examples to demonstrate the effectiveness of the proposed method. The paper is well-written, and easy to read with excellent clarity. Experiments are properly designed.
main_review: main_review: The paper proposes a return conditioned offline reinforcement learning (RL) approach, called "CtRL-Sim", to address the autonomous driving (AD) controllable simulated agents problem for the evaluation of autonomous vehicles (AV) planners. Compared to decision transformers (DT), CtRL-Sim predicts state sequences, and allows for modifying ("tilting") the rewards-to-go during the autoregressive inference process. The paper shows favorable quantitative results as well as qualitative examples to demonstrate the effectiveness of the proposed method. The paper is well-written, and easy to read with excellent clarity. Experiments are properly designed. 

* Originality.
Neither the problem or the method is new. However, the paper demonstrates a practical way to apply DT style offline RL methods to AD simulated agents, which is valuable contribution to the autonomous driving research community. The method is very practical and is easily reproduced. 

* Quality.
The paper is technically sound. Most of the claims are supported (except for claiming the method is "efficient") with experimental results. The method is appropriate. The paper is complete. The weaknesses of the method is not discussed in the paper. It'd be great to show some qualitative examples where the method is not performing well and explain why that's the case. 

* Clarity.
Perhaps the biggest strength of the paper is its clarity. The paper is well organized and enjoyable to read. 

* Significance.
The results are expected and supportive of most of the claims in the paper, however, according to the reported results, the proposed method doesn't show significant improvements over IL. I'd encourage the authors to clearly state that and explain why in the corresponding paragraph in the Experiments section. The method itself is an extension of existing methods, but it likely be used by other researcher and practitioners in the field. 

* Relevance.
The paper addresses simulated agents modeling, which is an important problem in AD. The method is evaluated in the Nocturne simulator. The AD community would be interested in reading this paper.

* Limitations.
Limitations of the proposed method in this paper is not sufficiently discussed.
questions: questions: * Line 77, what’s the forward transition model? It is worth describing in the main text.
* Line 99: “we also found it helpful to regularize the learned policy by predicting the full sequence of future states”, why predicting the full sequence regularizes the learned policy?
* Line 152, "Discretize the actions and return-to-gos into uniformly quantized bins." have the authors considered alternative tokenization methods?
* Table 1, why is the per-scene gen. time of CtRL-Sim w/ GT initial return so high compared to w/o GT initial return? The Line 292 says “efficient” but 20s is very slow compared to imitation learning (IL), 3.4s.
* Line 196, “The DT architecture is identical to that of CtRL-Sim except the return token precedes the state token” How does the order of return token and the state token matter?
* Table 1, w/o access to privileged GT futures, IL seems to perform really well and also very fast. This result seems to suggest that the main benefit of CtRL-Sim is controllability. Is that correct?
* Table 1, is it a fair comparison between methods w/ privileged GT futures and w/ privileged GT futures? Imitative metrics (ADE, FDE and Goal Success Rate) are closely related to privileged GT future information.
* Line 240, how to choose the two agents in the two-agent interactive scenarios?
* Fig 4, it seems that FT model generally performs worse than non-FT model. Why is that?
* Limitations of the proposed method in this paper is not sufficiently discussed.
* Overall, I'd be curious how does this reactive sim agents model compare to log-playback sim agents wrt evaluating AD planners.
limitations: limitations: 2
review_summary: review_summary: I recommend a weak accept. I may raise the rating depending on how well the questions are addressed in rebuttal.
title: title: Review of Submission27
paper_summary: paper_summary: This paper focuses on the generation of scenarios in driving simulation. Different from existing works that rely on heuristics or generative models of real-world data, the proposed method leverages return-conditioned offline reinforcement learning to efficiently generate reactive and controllable traffic agents. By controlling the desired returns for the various reward components, their method can generate a wide range of driving behaviors beyond the scope of the initial dataset. In addition, the authors also show that they can generate diverse and realistic safety-critical scenarios using adversarial generation.
main_review: main_review: **Strength:**

1.	The motivation is clear. Generating reactive and controllable scenarios is important to simulation. Existing work has not explored enough in this direction.
2.	The paper is well-written and well-organized. Most of the concepts and backgrounds are clearly stated.
3.	The analysis of experimental results is detailed and thorough.

**Weakness:**

1.	One potential issue I can think about is that the controllability of the generator is limited by the quality of the dataset. This is actually the limitation of all offline RL methods. I think the experiment of safety-critical scenario generation also supports this point. Without fine-tuning, the controllability is quite limited.
2.	The design of the reward in the proposed method seems not comprehensive enough. The three sub-rewards used seem to be particularly designed for the metrics of realism (collision and offroad). 
3.	How to select the goal during generation? I think the realism heavily depends on the selection of the goal. Does it require a lot of human effort to select the goal? If so, how to generate large-scale scenarios?
4.	Some details are missing. Check my questions.
5.	Missing evaluation of the improvement of downstream tasks. Simulation is mainly used to improve driving models. It would be better if the authors could show improvement of agents using the proposed scenarios for training or evaluation.
questions: questions: 1.	The definition of reward-to-go is not mentioned in the main context. I find the definition in the appendix, but I think the authors should at least mention what reward they used during the training. I think Figure 1 already shows all three rewards used. A justification for the selection of them will be better. The vehicle-to-vehicle and vehicle-to-edge factors seem to be carefully designed for the collision ratio and offroad ratio metrics. However, they are not sufficient conditions to show the realism of the scenario.
2.	In Figure 3, some of the agents are controlled by CtRL-Sim and others are not. Will that cause collisions between controlled agents and replayed agents? It seems the scenario is not fully reactive. If all agents are controlled by CtRL-Sim, the computation could be a burden and it is also hard to select desired goals.
3.	In Table 1, the result of Replay-Physics still has a 2.8% collision rate and 10.7% offroad rate. Does that mean the vehicle dynamics are not good enough?
4.	It is not clear what kind of initial RTG is used for CtRL-Sim (Base). I assume it is using the max RTG. Then according to the results in Table 1, CtRL-Sim ∗ (GT Initial Return) looks better than CtRL-Sim (Base). Does this mean the RTG influences the results a lot and the RTG prediction in CtRL-Sim is not accurate?
5.	According to Table 2, CtRL-Sim achieves a low collision ratio without finetuning, which might be evidence to show that the controllability is largely limited by the dataset. In addition, after fine-tuning, CtRL-Sim FT has a higher collision ratio than the model without finetune even with tilt=50. Does that mean finetuning on adversarial scenarios sacrifices the quality of the model?
6.	In DT, the input reward-to-go is decided by a linear scheduler. However, in the proposed method, it seems that the input reward-to-go is always the predicted one from the last step. Will the prediction error accumulate during the generation? Then does the tilting still work in later time steps? Some experimental results of the predicted error of RTG vs. timesteps could be helpful. 
7.	It would be interesting to show more tilting examples as in Figure 5.
8.	If I understand correctly, the tilting is applied to the predicted RTG. This makes sense since some scenario initialization will not easily lead to collision and the predicted RTG provides a baseline to tune. However, this also raises a question. Should I tune the tilting value according to the specific scenario? In addition, the value may depend on the distribution of this initialization in the dataset rather than the fact. For example, some initialization may easily lead to collision but the collected samples do not contain such scenarios.
limitations: limitations: 1
review_summary: review_summary: My initial score is weak reject as I have some questions to be answered. Most of them are clarification questions so my final score will depend on the new information provided by the authors.
title: title: This paper proposes an offline deep RL-based methodology to increase variability of autonomous driving simulator.
paper_summary: paper_summary: This paper proposes Decision Transformer-based agent with variable weightings of objectives at test time for autonomous vehicle simulation. The core part of the proposal is to introduce factorized return-to-go training and inverse temperature weights to modify agent behavior at test time.
main_review: main_review: The proposed idea is well described and easy to follow. The paper provides quantitative and qualitative experimental results of the resulted agent.

Strength:
- The inverse temperature for the factorized return-to-go is an intuitive knob for users to manipulate behavior of agents at test time.
- Since the inverse temperature doesn't involve to the training process, the training algorithm remains almost as simple as the base algorithm.
- Both quantitative and qualitative analysis is provided with sufficient number of baselines to show the advantage of the proposed method.

Weakness:
- The paper doesn't provide this analysis, but to train Decision Transformer-based agent with a good coverage of variable behavior, training datasets might need to have sufficient diversity to interpolate return-to-go space. In this paper, it seems not a problem since they used an open dataset. But, it could be potentially an issue especially when the large-scale dataset is not available.
- From deep RL perspective, dynamically weighting multiple objectives is not novel. There are some papers practically doing similar things[1][2].  While I think this paper proposes a novel application in autonomous driving community, authors need to add descriptions to clarify how this paper is positioned in deep RL community. 


References:
- [1] Abels, Axel, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. "Dynamic weights in multi-objective deep reinforcement learning." In International conference on machine learning, pp. 11-20. PMLR, 2019
- [2] Yang, Runzhe, Xingyuan Sun, and Karthik Narasimhan. "A generalized algorithm for multi-objective reinforcement learning and policy adaptation." Advances in neural information processing systems 32 (2019).
questions: questions: My main concern is that whether the proposed idea has novelty, compared to existing multi-objective deep RL research while I acknowledge its novelty as an application work. I'd like authors to add this discussion to Section 4. It'd be even nice if authors could add additional experimental results to include baselines based on an existing multi-objetive deep RL approach.
limitations: limitations: 1
review_summary: review_summary: From an application point of view, this paper proposes a useful methodology in autonomous driving community. But, the manuscript needs more discssion to clarify the novelty of the proposed method in deep RL community.