Learning Realistic Trafﬁc Agents in Closed-loop
Authors: Chris Zhang
James Tu
Lunjun Zhang
Kelvin Wong
Simon Suo
Raquel Urtasun
Venue: CoRL 2023 Poster (2023)
Keywords: Trafﬁc simulation
Imitation learning
Reinforcement learning

Abstract: Realistic traffic simulation is crucial for developing self-driving software in a safe and scalable manner prior to real-world deployment. Typically, imitation learning (IL) is used to learn human-like traffic agents directly from real-world observations collected offline, but without explicit specification of traffic rules, agents trained from IL alone frequently display unrealistic infractions like collisions and driving off the road. This problem is exacerbated in out-of-distribution and long-tail scenarios. On the other hand, reinforcement learning (RL) can train traffic agents to avoid infractions, but using RL alone results in unhuman-like driving behaviors. We propose Reinforcing Traffic Rules (RTR), a holistic closed-loop learning objective to match expert demonstrations under a traffic compliance constraint, which naturally gives rise to a joint IL + RL approach, obtaining the best of both worlds. Our method learns in closed-loop simulations of both nominal scenarios from real-world datasets as well as procedurally generated long-tail scenarios. Our experiments show that RTR learns more realistic and generalizable traffic simulation policies, achieving significantly better tradeoffs between human-like driving and traffic compliance in both nominal and long-tail scenarios. Moreover, when used as a data generation tool for training prediction models, our learned traffic policy leads to considerably improved downstream prediction metrics compared to baseline traffic agents.

Learning Realistic Traffic Agents in Closed-loop
Chris Zhang James Tu
Lunjun Zhang
Kelvin Wong
Simon Suo"
Raquel Urtasun
Waabi University of Toronto
{czhang, jtu, 1tt,, kwong, urtasun urtaaun))xaabi..at @waabi.
Abstract: Realistic traffic simulation is crucial for developing self-driving soft- ware in a safe and scalable manner prior to real-world deployment. Typically, imitation learning (IL) is used to learn human-like traffic agents directly from real-world observations collected offline, but without explicit specification of traf- fic rules, agents trained from IL alone frequently display unrealistic infractions like collisions and driving off the road. This problem exacerbated out-of- distribution and long-tail scenarios. On the other hand, reinforcement learning (RL) can train traffic agents avoid infractions, but using RL alone results in unhuman-like like driving behaviors. We propose Reinforcing Traffic Rules (RTR), holistic closed-loop learning objective to match expert demonstrations under a traffic compliance constraint, which naturally gives rise to a joint IL + RL ap- proach, obtaining the best of both worlds. Our method learns in closed-loop sim- ulations of both nominal scenarios from real-world datasets as well as procedu- rally generated long-tail scenarios. Our experiments show that RTR learns more realistic and generalizable traffic simulation policies, achieving significantly bet- ter tradeoff's between human-like driving and traffic compliance in both nominal and long-tail scenarios. Moreover, when used as a data generation tool for train- ing prediction models, our learned traffic policy leads to considerably improved downstream prediction metrics compared to baseline traffic agents.
Keywords: Traffic simulation, Imitation learning, Reinforcement learning
Introduction
Simulation is critical component to safely developing autonomous vehicles. Designing realistic traffic agents is fundamental in building high-fidelity simulation systems that have a low domain gap to the real world. However, this can be challenging as we need to both capture the idiosyncratic nature of human-like driving and avoid unrealistic traffic infractions like collisions or driving off- road. Existing approaches used in the self-driving industry lack realism: they either replay logged trajectories in non-reactive manner [1, 2] behaviors. Using data-driven approaches to learn more realistic policies is promising alternative.
heuristic policies which yield rigid, unhuman-like
The dominant data-driver approach has been imitation learning (IL), where nominal human driving data used as expert supervision to train the agents. However, while expert demonstrations provide supervision for human-like driving, pure IL methods lack explicit knowledge of traffic rules and in- fractions which can result in unrealistic policies. Furthermore, the reliance expert demonstrations can be disadvantage, long-tail scenarios with rich interactions very rare, and thus learning is overwhelmingly dominated by more common scenarios with much weaker learning signal.
Reinforcement learning (RL) approaches encode explicit knowledge of traffic rules through hand- designed rewards that penalize infractions [3, 4, 5, 6, 7]. These approaches do not rely on expert demonstrations and instead learn to maximize traffic-compliance rewards through trial and error. In the context of autonomy, this allows training synthetic scenarios that do not have expert demon- strations in order to improve the robustness of learned policies [5]. However, traffic rules alone cannot describe all the nuances of human-like driving, and it is still an open question if one can manually design a reward that can completely capture those intricacies.
"Wor done at Waabi.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

hns Condition
Exert Damo
Closed-loop Simulation
Export
Nominal Scanarion
Multeeent
Polley
E, 2,[-log (t)
Inital Conditions
Hero Agent (iplfiyy definition Slanntegee oit, insppedd
Long-tail
Scunarica
Closed-loop Learning
E_[-R(9]
Figure I: Our multi-agent policy is trained in closed-loop to match expert demonstrations under traffic compliance constraint using both nominal offline data and simulated long-tail scenarios as a rich learning environment. This gives rise to an IL objective which supervises the policy using real-world expert demonstrations and an RL objective which explicitly penalizes infractions.
Towards learning human-like and traffic-compliant agents, we propose Reinforcing Traffic Rules (RTR), a holistic closed-loop learning method to match expert demonstrations under a traffic- compliance constraint using both nominal offline data and additional simulated long-tail scenarios (Figure 1). We show our formulation naturally gives rise to unified closed-loop IL RL objective, which efficiently optimize by exploiting differentiable dynamics and per-agent factorization. In contrast to prior works that combine IL and RL [1], closed-loop approach allows the model to understand the effects of its actions and suffers significantly less from compounding error. Fur- thermore, exploiting simulated long-tail scenarios improves learning by exposing the policy to more interesting interactions that would be difficult and possibly dangerous to collect from the real world at scale. Our experiments show that unlike a wide range of baselines, RTR learns realistic policies that better generalize to both nominal and long-tail scenarios unseen during training. The benefits carry forward downstream tasks such simulating scenarios train autonomy models; predic- tion models trained on data simulated with RTR have the strongest prediction metrics on real data, serving as further evidence that RTR has learned more realistic traffic simulation. We believe this serves crucial step towards more effective applications of traffic simulation for self-driving.
2 Related Work
Traditional traffic simulation: To generate general traffic flow, simulators [8. 9, 10, 11] typically use heuristic models [12, 13, 14] as models of human driving. While these heuristic models are use- ful in capturing high-level traffic characteristics like flow and density, they lacking in capturing the lower-level nuances of human driving, thus limiting their applicability in self-driving. For more realistic traffic models, we explore using machine learning as more promising approach.
Imitation learning: IL methods learn a control policy from expert demonstrations. In the context of autonomous vehicles, [15] pioneered the use of behavior cloning (BC) to learn driving policy in open-loop. Since then, open-loop methods have been explored for both autonomy [16, 17, 18] and traffic simulation [19, 20, 21]. Open-loop methods primarily suffer from distribution shift due to compounding error [22], and various techniques like data augmentation [23, 16], uncertainty- based regularization [24, 25], and augmentation with rules- based planner [21] have been proposed to alleviate the problem. Closed-loop imitation learning approaches [26, 27, 28, 29], which address distribution shift by exposing the policy to a self-induced state distribution during training, have also been explored in traffic simulation [30, 31, 32, 33). While exploits expert demonstrations, there is a lack of explicit knowledge on safety-critical aspects like avoiding infractions. Methods like differentiable common-sense penalties [30, 23), additional finetuning [34], and test-time guided sampling [35] have been proposed complement the standard IL approach. In this work, we use reinforcement learning to explicitly encode general non-differentiable traffic rules.
Reinforcement learning: RL methods [36, 37, 38] do not require expert demonstrations and instead learn through interacting with the environment and a reward function. In self-driving, knowledge of infractions can be encoded in the reward [3, 5, 6. 7]. Because RL does not require expert demonstrations, it is possible to train procedurally generated scenarios for improved infraction avoidance [5]. However, it is difficult to learn realistic driving behavior using reward alone. RL

---

methods can be sample inefficient [3, 4], and specifying human- like driving with a scalar reward difficult. In this work, we supplement RI. with IL to learn more human-like driving while still enjoying the explicit learning signal provided from the reward.
Combined IL + RL: Pretrained IL policies can be used as initialization guide exploration [39, 40] or regularize learning [41, 42, 43, 44], and offline data can be used bootstrap learning and help with sparse rewards [45, 46]. Offline RL methods also use IL for -otto---istribttion generalization and overestimation [47, 48]. In self-driving, IL has been used as a pre-training phase improve sample efficiency [49]. Recent work augments open-loop II. with RI. 50, 51 to learn more robust models. While promising, the open-loop nature of BC leaves the policy susceptible to distribution shift. In this work, we explore a holistic closed-loop IL + RL method for traffic simulation.
Long-tail Scenarios: Real data can be curated [1, 52, 53] for more interesting scenarios, but col- lecting these at scale can be unsafe and expensive. Alternatively, scenarios can be generated by maximizing an adversarial objective w.r.t to the ego [54, 55, 56), but incorporating factors like diversity for training scenarios is still an open problem. In this work, we use knowledge-bart baseed approaches [57, 58] to guide generation towards a large variety of difficult but realistic scenarios.
3 Learning Infraction-free Human-like Traffic Agents
To learn realistic infraction-fis -frece agents, we propose unified learning objective to match expert demonstrations under an infraction- based constraint. We show how our formulation naturally gives rise to a joint closed-loop IL RL approach which allows learning from both offline collected human driving data when possible, and additional simulated long-tail scenarios containing rich interactions that would otherwise be difficult or impossible collect in the real world.
3.1 Preliminaries
We model multi-agent traffic simulation as a Markov Decision Process M = (S, with state space, action space, reward function, transition dynamics, and discount factor respectively. As our focus is traffic simulation where we have access to all ground truth states, we opt for a fully observable and centralized multi-agent formulation where a single model jointly controls all agents. This enables efficient inference by sharing computation 2, and easier interaction modeling.
m) E S to be the joint states of N agents where N may vary across different scenarios, as well as an HD map m which captures the road and lane topology. We parameterize the state of the i-th agent g(ó) with its position, heading, and velocity over the past H history timesteps. The state also captures 2D bounding boxes for each agent. Likewise, a = (a(2) (...))... .(()) E A is the joint action which contains the actions taken by all the agents. The i-th agent's action (+)) is parameterized by its acceleration and steering angle. Agents are controlled by single centralized policy T(a (s) which maps the
State, action and policy: We define the state (,.
g(N),
joint state to joint action.
Trajectories and dynamics: We define a trajectory To:T = (So, ao..... ST-1, aT-1, ST) as a se- quence of state action transitions of length 7 for all agents. We use the kinematic bicycle model [59] as a simple but realistic model of transition dynamics [[st+1|8:, a.) for each agent. Trajectories can be sampled by first sampling from some initial state distribution Po before unrolling a policy π through the transition dynamics, i.c. pM (T) = (o(s)) I[ π T( (s) (s).
Reward: Let R(*) (s, ()) ) be a per-agent reward which is specific for the i-th agent, but dependent on the state of all agents, to model interactions such as collision. The joint reward is then R(s, a)
ΣN R(O) ()), with R()) = ET- T- (R(ss,, ar) as the 7-discounted return of a trajectory.
Policy learning: Both imitation learning (IL) and reinforcement learning (RL) can be described in this framework. IL can be described as an f-divergence minimization problem: "" = arg min, Dy (T) II (T))) where P is the expert-induced distribution. RL on the other hand aims find the policy which maximizes the expected reward n¹ arg max pp [[(r)).
In experiments, model easily scales to 50 agents and map ROI of 1000m 400m per simulation.
3

---

3.2 Learning
To learn a multiagent traffic policy that is as human-like as possible while avoiding infractions, we consider the reverse KL divergence to the expert distribution with an infraction- constraint
arg min DEL (PT(T) P (T))
R (s, (s, = {or
infraction
(I)
(2)
S.t. Ep- [R(T)] 2 0
otherwise,
where R is per-agent reward function that penalizes any infractions (collision and off-road events). For rich learning environment, we consider both dataset D of nominal expert trajec- tories - ~ _h collected by driving in the real world, and additional simulated long-tail scenarios. Unlike real world logs, these scenarios contain what we denote as hero agents, which induce in- teresting interactions like sudden cut-ins, etc. (details in Section 3.4). More precisely, let ny be our learner policy. Let a ~ p be the initial state sampled from the long-tail distribution and m represent the policy of the hero agent. The overall multiagent policy is given as
Th (at (1)) if agent i is hero (a(a (s)) otherwise.
π (s),
The overall initial state distribution is then given as pn (1- a)pg app where of corresponds to the offline nominal distribution, and E [0, 1] is hyperparameter that balances the mixture. Taking the Lagrangian of Equation 8 decomposes the objective into an IL and RL component,
= Ep-
log P (T) A R(T))
H(x) C AC H(x)
II.
RL.
where À is a hyperparameter balancing the two terms, and ( ) is an additional entropy regulariza- tion term [26]. Notably, we optimize II. and jointly in a closed-loop manner, as the expectation taken with respect to the on-policy distribution (T). Compared to open-loop behavior cloning, the dlosed-loop IL component allows the model to experience states induced by its own policy rather than only the expert distribution, increasing its robustness distribution shift. Furthermore, while the additional reward constraint may not change the optimal solution of the unconstrained problem (the expert distribution may be infraction-free), it can provide additional learning signal through RL.
The RL component Ep [R()) can be optimized using standard RL techniques and exploits both offline-coleeced nominal scenarios and simulated long-tail scenarios containing rich interactions. However, the imitation component _ is only well-defined when expert demonstrations are avail- able and thus only applied nominal data. We start from initial state sf of and have the policy нa control all agents in closed-loop simulation. The loss is the distance between the ground truth and policy-inne inlicy-induced trajectory*
CLL E,, (B-
[D(TE, .7]]
(5)
is difficult to obtain accurate action labels for human driving data in practice, so we only consider states our loss, i.e. D(7 7) Σi= d(s 8t) where d a distance function (e.g. Huber).
Optimization: To optimize Equation 4, we first note that the L component is differentiable by using the reparameterization trick [60] when sampling from the policy and differentiating through the transition dynamics (kinematic bicycle model). We refer the reader to the appendix for more details. To optimize the RR. component, we design a centralized and fully observable variant of PPO [36]. While it is possible directly optimize the policy with the overall scene reward
The causal entropy term is included as an entropy regularizer some learning algorithms such as PPO [36]. In our setting, we empirically found that it was not necessary to include.
1 "As we do not have access E directly to query log-likelihood, using distance is essentially making the assumption that p²(r) (T) x exp [-D(r". ())r
We found that directly using the mean action provides good results without the need for sampling.

---

a(s,a)= = Ei= (s, dl we instead optimize each agent individually with their respective individual reward R,(s,a,), a,). While this factorized approach may ignore second-order interaction effects, it considerably simplifies the credit assignment problem leading to more efficient learning. More precisely, we compute factorized value targets V(1) = Ei= 7 R2 (8t, a,"). and the factor- ized PPO policy loss is given as Craliey Σ min(r(9) (0() A clip(r(), the probability ratio is factorized, i.c. r()) ((a ss m)/ Tola (a |s, m) and A ( is a factorized
c, 1 + e)A(9) where
GAE [61] estimate. More details can be found in the appendix.
3.3 Model Architecture
Our traffic model ma architecture uses common ideas from SOTA traffic agent motion forecasting literature in order extract context and map fea- tures and predict agent actions (Figure 2). Recall that a state each individual agent's s states s( that contain the agent's 's kinematic state over a history horizon H, and an HD map m. From each agent's state his- tory. a shared ID CNN and GRU are used to ex-
Agani States
History Encoder
Interaction
Agent adtine
{s(2), ( m) consists of
Lane Graph
Graph
Encoder
Our multiagent policy architecture. The value network architecture is the same but regresses value targets instead.
Figure 2:
tract agent history context features (4) f(s()). At the same time, a GNN used to extract map features from a lane graph representation of the map input h. gm).. HeteroGNN [62] then jointly fuses all agent context features and map features before shared MLP decodes actions for each agent independently.
(
((W,,...(N))) (N)) (eteroGNN((1.))) ..... ,...,,,,)) MNN)) hm)
(6)
.....
(n (0))) = (L)
We use independent normal distributions to represent the joint agent policy, i.c. T(a N ,())), and thus x(a s) I (s) Note that agents only independent condi- tional on their shared past context, and thus important interactive reasoning is still captured. Our value model uses the same architecture but does not share parameters with the policy; we compute {hg...., .% in a similar fashion, and decode per-agent value estimates F7(() -MLP((h())
3.4 Simulated Long-tail Scenarios
Nominal driving logs can be monotonous and provide weak learning signal when repeatedly used for training. In reality, most traffic infractions can be attributed to rare and long-tail scenarios belonging to handful of scenario families [63] which can be difficult and dangerous to collect from the real world at scale. In this work, we procedurally generate long-tail scenarios to supplement nominal logs for training and testing. Following the self-driving industry standard, we use logical scenarios [64, 57] which vary in the behavioral patterns of particular hero agents with respect to an ego agent (e.g. cut-in, hard-braking, merging, etc.). Designed by expert safety engineers, each logical scenario is parameterized by 0 θ which controls lower-level aspects of the scenario such behavioral characteristics of the hero agent (e.g. time-to-collision or distance triggers, aggressiveness, etc.), exact initial placement and kinematic states, and geolocation. A concrete scenario can then be procedurally generated an automated fashion by sampling a logical scenario and corresponding parameters 9. While these scenarios cannot be used for imitation as they are simulated and do not have associated human demonstrations, they provide rich reinforcement learning signal due the interesting and rare interactions induced by the hero agents.
4 Experiments
Scenario sets: Our experiments use two datasets that represent nominal and long-tail scenarios respectively. The NOMINAL dataset consists of set of highway logs which capture varying traffic densities and road topologies while containing expert demonstrations. The dataset consists of 465
5

---

BC
RL
RL-S RL-Shaped
BC
8TR (ours)
Nomiina
Nominal
Long-tel
Long tail
0:30
0.10
Collsiinn
*
0.6
IDE (m))
Accell |SD Inats)
FDE (m)
Accel JSD (nats)
Figure 3: Metrics (lower is better) on held-out nominal and long-tail scenarios. Pareto frontier of baselines is shaded; RTR achieves the best tradeoff between infraction and other realism metrics.
Collision
Aggressive overtake
Safe lane change
marge
Unreactive
Braking and defensive
Controlled (start)
Controlled (end)
Hero (start)
Hero (end)
Other Controlled
Figure 4: Qualitative examples comparing the baseline IL model (left) and RTR (right). Scenarios with hero agents (blue) are from the long-tail set. All other agents are controlled; pink is used for visual emphasis. RTR avoids infractions while maintaining diverse, human-like driving behavior.
snippets for training and 115 for testing, where each snippet lasts for 20 seconds. We use LONGTAIL to denote the scenario set generated using the process outlined in Section 3.4 which contain rare actor maneuvers like sudden cut-ins. We use 25 logical scenarios to generate total of 333 concrete scenarios, where 167 concrete scenarios are used for training and 166 are held-out for evaluation. This evaluation set is held-out on the parameter level and measures in-distribution generalization. We also evaluate on an additional set of held out logical scenarios measure out-of-distribution generalization, with more details in Section 4.1.
Metrics: We evaluate our traffic models' ability to 1) match human-likk driving and 2) avoid in- fractions. For the former, we measure similarity to the demonstration data by computing the final displacement error (FDE) [33, 30], which measures the L2 distance between the agent's simulated and ground truth (GT) position after seconds. Furthermore, we use Jensen-Shannon Divergence (JSD) [33, 31] between histograms of scenario features (agent accelcration) in order to measure dis- tributional realism. Finally, to measure infraction rates, consider collision and driving off-road. We use bootstrap resampling over evaluation snippets to compute uncertainty estimates. Results with more extensive metrics (and their definitions) can be found in the appendix.
4.1 Benchmarking Traffic Models
Comparison to state-of-the-art: We evaluate RTR and several baselines both nominal and long-tail scenarios. For comparability, as described in Section 3.3 for all methods. Our first two baselines are representative of state-of- the-art imitation learning approaches for traffic simulation. BC is our single-step behavior cloning baseline following [19]. The IL baseline is trained using closed-loop policy unrolling [30, 31j. Next, RL is trained using our proposed factorized version of PPO [36] with the reward in Equation 20. The RL-Shaped baseline includes additional reward for driving the speed-limit to encourage more human- like driving. Finally, BC+RL an RL augmented BC baseline following [1].
the same input representation and model architecture

---

Rear-end collisin
Meth.
Col. (%)
Off. (%)
IL
11.8 + 2.1
1.0 + 0.1
atappinvsstoppinn
RTR
5.0 - 1.4
0.3 1 0.1
Figure 6: IL (top), RTR (bottom) on an out-of-distribution scenario where hero agent (blue) comes to complete stop the highway.
Table 1: Results on out-of-
distribution long-tail set.
Figures 3 and 4 show the results; full table be found in the appendix. Firstly, the BC model achieves poor realism because it suffers from distri- bution shift during closed-loop evaluation as it en- counters states unseen during training due to com- pounding error. Next, we see RI achieves low in- fraction rates but results in unhuman-like driving (Figure 5). This is because it is difficult for reward alone to capture realistic driving. Efforts in reward shaping result in improvements but are ultimately still insufficient. We see BC+RL improves upon BC infractions but still lacks realism. This is because BC an open-loop objective and only provides signal expert states, while only the RL signal is present in non-expert states. Thus, the policy still suffers from compounding еггоr with respect to imitation. On the other hand, closed-loop IL performs better it is more robust compounding error, but still struggles on the long-tail scenario set without explicit supervision. Finally, the holistic closed-loop IL and RL approach of RTR im- proves infraction rates while maintaining reconstruction and JSD metrics. We see RTR outperforms even pure RL in terms of infraction rate on long-tail scenarios, suggesting that including long-tail scenarios during training can help the model generalize to held-out evaluation long-tail scenarios.
Dzmonstratin AL Pollyy
Domonstraiin
JUR (ouns)
0.2
0.0
-4
0.0
Acceleration (mde
Acceleration Imis
Figure 5: RL policy naively decelerates to avoid infractions. RTR learns to avoid col- lision more naturally without slowing down.
Out-of-distributie generalization: Recall from Section 3.4 that logical scenarios define fam- ily of scenarios and concrete scenarios define variations within family. While have evaluated in-distribution generalization by using held-out concrete scenarios, we further evaluate on held-out logical scenarios. We use 11 held-out logical scenarios with new map topologies and behavioral patterns to procedurally generate an additional out-of-distribution set consisting of 84 concrete sce- narios. Our results show that RTR generalizes to this set better than basclines (Figure 6, Table 1).
4.2 Downstream Evaluation
One downstream application of traffic simulation is generating synthetic data for training autonomy models. We evaluate if the improved realism of RTR transfers in this context. Each
Method
FDE (m)
CTE (m)
BC
2.44 1: 0.05
0.90 1: 0.04
model is used to generate a synthetic dataset of 589 scenarios which we use to train a SOTA prediction model [62] before evaluating its performance on held-out real data. Besides FDE, the cross-track error (CTE) of predicted trajectories projected onto the GT are used as prediction metries. More experiment details can be found in the appendix. Table shows that using RTR generate training data results in the best prediction model. This provides evidence that RTR has learned more realistic behavior and has a lower domain gap compared to baselines, showing that our approach can improve the application of traffic simulation in developing autonomous vehicles.
IL.
1.75 0.06
0.28 0.01
RL
15.42 1: 1.21
0.32 1 0.02
RL-Shp
6.66 + 0.26
0.33 + 0.01
BC+RL
9.06 = 0.50
0.42 - 0.03
RTR
1.58 0.05
0.27 0.03
Table 2: Prediction model trained on synthetic, evaluated on real.
4.3 Additional Analysis
Long-tail scenarios: We evaluate our approach of using procedurally generated scenarios against the alternative of mining hard scenarios from data [1, 53] by curating set of logs from NOMINAL that the IL model commits infraction on. Figure shows that using only curated scenarios does not transfer well to the long-tail tail set, and in fact introduces regression in the nominal scenarios, sug-
7

---

Nom
Cur
Nom+Cur
Nom+ (ours)
6.0
Pactorazed lours)
Pactortzed (ours)
Untactortzed
2.0
Untactorized
Long Cal (50
0.08
Collsoo (S0
Offraed
0.06
2.0
2.0
0.9
0.0
0.004
0.005
5.0
5.5
6.0
0.15
1000
2000
2000
1000
2000
3000
Nominal Cal. (%6)
FDE (m)
Accel [SD (nats)
Step
Step
Figure 7: Using both nominal and long-tail yields the best tradeoff compared to baselines.
Figure 8: Our factorized PPO vs. standard PPO which uses single scene-level reward.
gesting the model is overfitting the curated scenarios. Up-sampling curated scenarios (Nom+Cur) also fails relying purely offline data may require prohibitively larger scale data collection.
Factorized multiagent RL; To ablate our factorized per-agent approach to multiagent PPO, we compare to standard PPO implementation where the scene-level reward R(s, s, a) is used as super- vision for the joint policy rather than each individual agent reward R(*) (s, al)). Figure shows that the factorized loss outperforms the alternative, likely due the fact that multiagent credit assign- ment is extremely difficult when using the scene-level reward, leading poor sample efficiency.
Balancing the trade-off:: Recall from Section 3.2 that RTR balances human-like driving and avoiding infrac- tions by weighting the IL vs. RL loss with À and nomi-
Nominal
Long-tail Col. (%)
A
Col. (%)
FDE (m)
0.0
0.0
0.89 i 0.39
4,50 :: 0.24
12.13
2.44
nal vs. long-tail training with (e.g. À = a 0 is the IL baseline). We found increasing the relative weight of RL and long-tail scenarios generally improves infraction avoidance while increasing the relative weight of II. and nominal training generally improves other realism met- rics as expected (Table 3). However, RTR is not particu- larly sensitive; many configurations are within noise and all configurations dominate the baseline Pareto frontier.
+
5.0
0.5
0.38 1: 0.20
5.16 2: 0.28
3.61 1: 1.35
0.52
1 0.17
5.10
- 0.20
3,82 + 1.12
0.35
+ 0.18
5.20
0.21
4.12 + 0.90
5.0
0.7
0.56
+ 0.21
6.10 = 0.23
3,51 + 1.32
Table 3: Balancing realism and infrac- tion avoidance.
Imitation learning signal: We consider the alternative of using a frozen pretrained IL policy as regularization [44] instead of our approach of using offline data. A frozen
Nominal
Long-tail
Method
Col. (%)
FDE
Col. (%)
KL-L
+: 0.20
25.68 1.14
policy potentially provides more accurate closed-loop su- pervision, as a Fuclidian-basee based distance loss with demon- stration data may inaccurate if the rollout has diverged. We evaluate two baselines: KL Reward and KL Loss, where the KL between the current and frozen policy is added to the reward or loss respectively. Our results in Table 4 show that using demonstration data is still the most performant, suggesting that the inaccuracy from an imperfect IL policy is larger than that of using a distance-base based loss.
KL.R
0.38 1: 0.22
15.19 1: 0.99
4.97 1: 1.29
3.61
Table 4: Comparing different alterna-
tives to proposed IL loss.
Conclusion and Limitations
We have presented RTR, a method for learning realistic traffic agents with closed-loop IL+RL using both real-world logs and procedurally generated long-tail scenarios. While we have shown substan- tial improvements over baselines in simulation realism and downstream tasks, we recognize some existing limitations. Firstly, while using logical scenarios framework for procedural generation exploits human prior knowledge and is currently an industry standard, manually designing scenar- ios can be difficult process, and ensuring an adequate coverage of all possible scenarios is an open problem. Exploring automated alternatives like adversarial approaches to scenario generation would be an interesting future direction. Secondly, while we have explored the downstream task of gen- erating an offline dataset to train prediction models, other applications like training and testing the entire autonomy stack end end in llosedllopp a promising future direction.

---

Acknowledgments
The authors would like thank Wenyuan Zeng for their insightful discussions throughout the project. The authors would also like to thank the anonymous reviewers for their helpful comments and suggestions to improve the paper.
References
[1] Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, B. Roelofs, B. Sapp, B. White, A. Faust, S. White- son, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. arXiv preprint arXiv:2212.114/11 2022.
[2] E. Vinitsky, N. Lichtlé, X. Yang. Amos, and J. Foerster. Nocturne: scalable driving benchmark for bringing multi-agent learning one step closer the real world. arXiv preprint arXiv:2206.09889 2022.
[3] D. Chen, V. Koltun, and Krähenbühl. Learning to drive from world on rails. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15590-15599, 2021.
[4] M. Toromanoff, E. Wirbel, and F. Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[5] C. Zhang, R. Guo, W. Zeng, Y. Xiong, B. Dai, R. Hu, M. Ren, and R. Urtasun. Rethinking closed-loop training for autonomous driving. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 27,27, 2022, Proceedings, Part XXXIX, pages 264-282. Springer, 2022.
[6] X. Pan, Y. You, Z. Wang, and C. Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint TrXXiv1704.039555 2017.
[7] Shalev-Shwartz, Shammah, and Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
[8] P. A. Lopez, M. Behrisch, L. Bicker-Walzz J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, Rummel, Wagner, and E. Wießner. Microscopic traffic simulation using sumo. In 20/8 2 1t international conference on intelligent transportation systems (ITSC), pages 2575-2582. IEEE, 2018
[9] M. Balmer, M. Rieser, K. Meister, D. Charypar, N. Lefebvre, and K. Nagel. Matsim-t: Archi- tecture and simulation times. In Multi-agent systems for traffic and transportation engineering, pages 57-78. IGI Global, 2009.
[10] J. Casas, J. L. Ferrer, D. Garcia, J. Perarnau, and A. Torday. Traffic simulation with aimsun. Fundamentals of traffic simulation, pages 173-232, 2010.
[11] M. Ben-Akiva, H. N. Koutsopoulos, T. Toledo, Q. Yang, C. Choudhury. C. Antoniou, and R. Balakrishna. Traffic simulation with mitsimlab. Fundamentals of traffic simulation, pages 233-268 2010.
[12] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations aug 2000, 1oi:10.11031 and microscopic simulations. Physical Review E, www3//2ppysrreeee.oommmm 62. 1805. ohysreve.62.1805. URL http:///ooo.orgg.....1//////////
[13] K. Kreutz and J. Eggert. Analysis of the generalized intelligent driver model (gidm) for un- controlled intersections. In 2021 IEEE International Intelligent Transportation Systems Con- ference (ITSC), pages 3223-3230, 2021. doi: 10:10.110 10001000/TS44499222001155555
[14] A. Kesting. Mobil General lane-changing model for car-following models. 2007.

---

[15] D. A. Pomerleau. Alvinm: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[16] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. Goyal, L. D. Jackel, M. Monfort, U. Muller, Zhang, al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.0736,, 2016.
[17] Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, End-to-end driving via con- ditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693-4700. IEEE, 2018.
[18] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 7077-7087, 2021.
[19] L. Bergamini, Y. Ye, O. Scheel, L. Chen, C. Hu, L. Del Pero, B. Osiński, H. Grimmett, and P. Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 5119- 5125. IEEE, 2021.
[20] L. Feng. Q. Li, Z. Peng, S. Tan, and B. Zhou. Trafficgen: Learning generate diverse and realistic traffic scenarios. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 3567-3575. IEEE, 2023.
[21] D. Xu, Chen, Ivanovic, and M. Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2929-2936. IEEE, 2023.
[22] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[23] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:/812.03077 2018.
[24] K. Brantley, W. Sun, and M. Henaff. Disagreememn--rggulaiizee imitation learning. In Interna- tional Conference Learning Representattins, 2020.
[25] M. Henaff, A. Canziani, and Y. LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic, arXiv preprint 1rXiv:1901.02000 2019.
[26] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[27] Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning, arXiv preprint arXiv:1710.11222 2017.
[28] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imita- tion learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
[29] L. Ke, S. Choudbury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pages 313-329, Springer, 2021.
[30] Suo, Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10400-10409, June 2021.
10

---

[31] M. Igl. D. Kim, A. Kuefler, P. Mougin, P. Shah, K. Shiarlis, D. Anguelov, M. Palatucci, B. White, and S. Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation, 2022. URL pttppp:///rxiix wrg..bgg///..ooo.oo.oo...o... 221./ 03195.
[32] A. Scibior, V. Lioutas, D. Reda, P. Bateni, and Wood. Imagining the road ahead: Multi- agent trajectory prediction via differentiable simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 720-725. IEEE, 2021.
[33] S. Suo, K. Wong, J. Xu, J. Tu, A. Cui, S. Casas, and R. Urtasun. Mixsim: A hierarchical framework for mixed reality traffic simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9622-9631, June 2023.
[34] V. Lioutas, A. Seibior, and F. Wood. Titrated: Learned human driving behavior without infrac- tions via amortized inference. Transactions on Machine Learning Research, 2022.
[35] Z. Zhong. D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided condi- tional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
[36] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint /rX.v:/707.06::: 2017.
[37] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 2018.
[38] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 509.02977, 2015.
[39] I. Uchendu, Xiao, Y. Lu, Zhu, M. Yan, Simon, M. Bennice, C. Fu, C. Ma, Jiao, al. Jump-start reinforcement learning, arXiv preprint arXiv:2204.02372. 2022.
[40] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0999 09359, 2020.
[41] N. Stienmon, I.. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[42] Y. Lu, K. Hausman, Y. Chebotar, M. Yan, E. Jang, A. Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, et al. Aw-opt: Learning robotic skills with imitation and reinforcement scale. In Conference on Robot Learning, pages 1078-1088. PMLR, 2022.
[43] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 8822099564 2018.
[44] I. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprini arXiv:2203.02/55. 2022.
[45] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[46] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint 170.001707.88177, 2017.
11

---

[47] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1 19:1199-199, 2020.
[48] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in newral information processing systems, 11-20132-20144 2021.
[49] X. Liang, T. Wang, L. Yang, and E. Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In Proceedings of the European conference on computer vision (ECCV), pages 584 009,9999220 2018.
[50] A. Kamenev, L. Wang, O. B. Bohan, Kulkarni, B. Kartal, A. Molchanov, S. Birchfield, D. Nistér, and N. Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In 2022 International Conference Robotics and Au- tomation (ICRA), pages 8936-8942. IEEE, 2022.
[51] Q. Zhang, Y. Gao, Y. Zhang, Y. Guo, D. Ding, Y. Wang. P. Sun, and D. Zhao. Trajgen: Generat- ing realistic and diverse trajectories with reactive and feasible agent behaviors for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(12):24474224487, 2022.
[52] N. Webb, D. Smith, C. Ludwick, T. Victor, Q. Hommes, F. Favaro, G. Ivanov, and T. Daniel. Waymo's safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054 2020.
[53] E. Bronstein, S. Srinivasan, S. Paul, A. Sinha, M. O'Kelly, P. Nikdel, and S. Whiteson. Em- bedding synthetic off off-policy experience for autonomous driving via zero-shot curricula, arXiv preprint
2022.
[54] D. Rempe, J. Philion, L. Guibas, Fidler, and O. Litany. Generating useful accident-prone driving scenarios via learned traffic prior. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[55] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In European Conference on Computer Vision, pages 335-352. Springer, 2022.
[56) J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9909-999999 2021.
[57] H. Weber, J. Bock, J. Klimke, C. Roesener, J. Hiller, R. Krajewski, A. Zlocki, and L. Eckstein. framework for definition of logical scenarios for safety assurance of automated driving. Traffic injury prevention, (OSSup)):S65-55 2019.
[58] Menzel, G. Bagschik, and M. Maurer. Scenarios for development, test and validation of automated vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1821-1827. IEEE, 2018.
[59] M. LaValle. Planning algorithms. Cambridge university press, 2006.
[60] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 131266 14, 2013.
arXiv preprint
[61] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02436. 2015.
[62] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint invariiat motion forecasting, arXiv preprint arXiv:2211.02544 2022.
12

---

[63] W. G. Najm, J. D. Smith, M. Yanagisawa, et al. Pre-crash scenario typology for crash avoidance research. Technical report, United States. National Highway Traffic Safety Administration, 2007.
[64] T. Menzel, G. Bagschik, and M. Maurer. Scenarios for development, test and validation of automated vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1821-1827.
IEEE, 2018.
[65] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 624-641. Springer, 2020.
[66] M. Liang, B. Yang, Hu, Y. Chen, Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, volume 12347 of Lecture Notes in Computer Science, pages 541-556. Springer, 2020. doi: 00011001007/978---00--555533
32. URL httptt//://do owg./org////
[67] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arX1::171.05101,220 2017.
13

---

A Additional Results
Metries: In order to measure the realism of our traffic models, we use a set of metrics which evaluate both the traffic models' ability to match human demonstration data in the nominal scenarios and avoid infractions both nominal and simulated long-tail scenarios.
Reconstruction: In nominal scenarios where expert demonstrations exist, we consider a set of metries which evaluate how close traffic model's s simulation to the real world conditioned the same initial condition. We measure the final displacement error (FDE) [65], defined as the L2 distance between agent's position in simulated scenario the ground truth scenario after 5s. We also measure the along-track error (ATE) and cross-track error (CTE) of an agent's simulated position projected onto the ground truth trajectory. This decomposition disentangles speed variability and lateral deviations respectively.
Distributional: While reconstruction metrics compare pains of real and simulated logs, we can compute distributional similarity metrics as an additional method to gauge realism. We compute the Jensen-Shannon Divergence (JSD) [31] between histograms of scenario features to compute their distributional similarity. Features include agent kinematics like acceleration and speed, pair- wise agent interactions like distance to lead vehicle, and map interactions like lateral deviation from lane centerline.
Infraction Rate: Finally, we measure the rate of traffic infractions made by agents controlled by a traffic model. Similar to prior work [30], we measure percentage of agents that end up in collision drive off-road. As this metric does require ground truth scenarios for pairing or computing statistics, it can be used in simulated long-tail scenarios that do not have ground truth.
Comparison to state-of-the-art: In main paper, we presented select results from com- parison to state-of- ther-rrtt-rtt traffic models both nominal and long-tail scenarios. Here, we include additional tradeoff plots for all metrics in Figure 9. We also include a table of detailed metrics for all methods in Table 5. Building on our observations in the main paper, we see that RTR outperforms and expands the existing Pareto frontier on all metrics and scenario sets. II. methods achieve strong reconstructonn/disttibbttooo realism metrics but suffer from high infraction rates, while RL meth- ods attain the opposite. RTR achieves the best both worlds-a testament to ability to learn human-liie driving while avoiding unrealistic traffic infractions.
Long-Tai Tail Scenarios: our main paper, we evaluated our approach of using procedurally gener- ated long-tail scenarios against the alternative of mining hard scenarios from data. Here, include additional tradeoff plots for all metrics in Figure 10, with the detailed metrics in Table 6. We see that training on both nominal and long-tail scenarios outperforms the alternatives in most cases.
In addition, we present slightly different view of Figure 3 of the main paper where the y-axis is in the same scale Figure 11. This view highlights the difference in difficulty between the different scenario sets.
Downstream Experiment: We provide additional details for our downstream experiment in Sec- tion 4.2. The evaluation data used is additional 118 snippets held out from the nominal dataset. The hyperparameters for training the prediction model (model size, number of epochs, learning rate schedule) were tuned on the training split of the nominal dataset and kept fixed and constant when training on the datasets generated by the methods in Table 2 in order to be fair. The synthetic dataset for cach method is also generated using the same 589 initial conditions to be fair. We report the min- imum over modes for our multimodal prediction model. We use 4 separate checkpoints to compute the uncertainty estimates.
Distributional Realism: In Figure 12, we include additional plots showing the histograms used to compute JSD distributional realism metrics on the nominal scenario set. We see that RL methods (RL, RL-Shaped, and BC RL) struggle to capture human-like driving, particularly in
14

---

BLShappd
SCTAL
20"1
Calizen (1g
10-4
Oftraad (8),
Longeia Cal (94
20-1
4x30
102 100 mm)
1Cc ATE 0m)
1.2 CTE (m)
0.2
0.4
0.6
0.3 0.4
3.20
0.20
Accel 1S0 (vata)
Speed (SD(wet))
Les dev. JSD (twit)
Lead dist. (nals)
Figure Additional plots comparing infraction realism tradeoff of RTR compared baseline models. We see that RTR outperforms and expands the existing Pareto frontier for all metrics.
Infraction (%)
Reconstruction (m)
JSD (nats)
LT-Inf. (%)
Method
Col.
Off Rd.
FDE
ATE
CTE
Acc.
Spd.
Lat
Ld.
Col.
BC
22.13 + 1.32
58.68
2,18
4,50 + 0.24
3,60
0.20
1.84
0.17
0.34
0.54
0.14
0.20
17:00 + 2.91
0.89
0.39
2.48
0.36
4.98 -: 0.23
4.75
0.23
0.66
0.15
0.14
0.07
12.13 + 2.44
RL
0.23
0.17
0.20
+ 0.13
56.92
0.87
56.91
+ 1.91
0.75
: 0.08
0.60
0.46
0.54
0.12
4.26
- 1.30
Stp.
1.50
0.36
1.01
1. 0.29
21.29
1: 1.13
21.17
1.12
0.97
1. 0.05
0.43
0.43
0.48
0.13
6.95
1.81
BC+RI.
3.08
0.49
1.88
0.32
47.30
0.49
47.26
+ 0.50
1.05 0.09
0.62
0.53
0.49
0.15
4.46
1.31
RTR
0.38 1. 0.20
0.20
1: 0.10
5.16
1: 0.28
4.97
0.28
0.61 1 0.04
0.16
0.33
0.14
0.07
3.61
1.35
Table 5: Detailed breakdown of metrics. Metrics on the left (resp. right) are computed on nominal scenarios (resp. long-tail scenarios). IL methods achieve strong reomsstuction//isiritttiooal realism metrics but suffer from high infraction rates, while RL methods attain the opposite. RTR achieves the best of both worlds, with high recomstaction/disiiibtttooaalll realism and low infraction rates.
speed and acceleration JSD where the RL methods tend to brake more often than humans. BC exhibits slightly better results overall, but has worse map interaction reasoning due distribution shift from compounding errors. In contrast, RTR captures human-like driving significantly better, closely matching IL in distributional realism while also improving on its infraction rate as seen in other results.
Qualitative Results: We include qualitative results comparing RTR against the baselines Fig- ures 13, 14, 15, and 16. Across fork, merge, and long-tail scenarios, we see that RTR exhibits the greatest realism of the competing methods.
B Learning
B.1 Loss Derivation
In this section, we will provide more details on the loss derivation using the Lagrangian. Recall that we begin with the following optimization problem
mig
DEL (PT(T) || (F)
a
(8)
s.t. Epr [R(T] 2 0
15

---

tamplet cat 8
0.08
0.07
0.00
0.00400.20 2040 00500 0253
2.202 0,003 0,004 2,000 3.0
5.5
5.0
2.55
0.50
0.05
0.70
Nominal Cal. (%)
Neminal Off Rosd. 196)
FDE (m]
ATE (m)
CTE pm)
0.00
Longret Cat (98
0.00
0.07
Nom Cur
0.06
(ours)
2.237 2128 0.119 0.340 0141 Speed (SDt/who
0.345
0.130 2,15 Accel ISD (nets)
0.26 0.20 3.30 0.22
2,072 0,073 0.274 0.375 Lead dist JSD matss
Latdewx /SD (nats)
Figure 10: Additional plots showing the tradeoff between infraction rate on the long-tail set and other realism metrics the nominal set, for models trained different scenario sets. We see that for most metrics, training on both nominal and long-tail scenarios obtain the best tradeoff.
Infraction (%)
Reconstruction (m)
(nats)
I.T.nnf. (%) Col.
Train
Col.
Off Rd.
FDE
ATE
CTE
Acc.
Spd.
Lai.
Nominal
0.38 0.20
0.49 + 0.17
5,05 1 0.25
4.86 1 0.24
0.65
0.04
0.14
0.14
0.33
0.07
9.60 2.17
Curated
0.45 0.21
0.14 = 0.08
5.34 0.23
5.34
5.11 + 0.23
0.76 0.05
0.15
0.14
0.29
0.08
9.42 = 2.17
Long-tail
0.23
0.56 0.04
0.25
L.37
Nom. + Cur
0.38 0.20
0.30 + 0.11
5.27 1: 0.24
5.06
1 0.30
0.67
1 0.05
0.15
0.14
0.34
0.08
9.04
2.14
Noo.
0.38 0.20
0.20
5.16 0.28
4.97 0.28
0.61 0.04
0.16
0.33
3.61 1.35
Table Detailed breakdown of realism and infraction metries training on different scenario sets.
We form the Lagrangian of the optimization problem
C(x, A) = DKL (PM (T) || (B(T)) + XEp= [R(T)]
(9)
(T) AR(T)
= Epi
log (
(10)
Epp [- log p pp( AR(T)] - (Tx).
(11)
where À is Lagrangian multiplier and
H(T) Epp [log pa (T)
(12)
T-1
= -p Pu(so) Σ
log r (a')
(13)
under deterministic dynamics is the causal entropy [26]. Using the Lagragian, the optimization problem is converted an unconstrained problem
arg min max L(T, A).
(14)
Equation 14 can be optimized in number of ways, such as iteratively solving the inner maximiza- tion over À and outer minimization over æ. We take simplified approximate approach where we simply set Aixed 2 0 as a hyperparameter, leading to what is ultimately a relaxed constraint or penalty method.
n arg min Ept [- log (F Aisea Aisea R(T) H())
(15)
The causal entropy term is included as an entropy regularization term in some learning algorithms such as PPO [36]. In practice, we found that it was not necessary to include.
16

---

BC
RL
Ri-Shaped
BC
RTR (ours)
Nominal
Nominal
Long tail
Long-tar tail
0.3200
0,0800
Collsiion
0.0200
26
0.0100
0.0050
0.0025
0.2
0.4
0.5
0.6
5
10
20
0.3 0.4 Accel 1S0 (nats]
0.6
FDE (m)
Accel JSD (nats)
(m)
Figure 11: Alternative view Figure 3, where now the y-axis on the same scale across the different scenario sets. We see that in the Long-tail scenario set is significantly harder than the nominal set.
B.2 Imitation Learning Loss
Recall that the imitation learning component of the loss is given as
CLL EYEND (EAPPT(11)) [D( []]
(16)
Fig .... sf.
Σ d(s k²)
(17)
where
ā ~ Ta)(:: 8c+1 8z + f(er, dt.
(18)
(19)
Because the dynamics function described in Section B.7 differentiable, Equation 17 com- pletely differentiable using the reparameterizatioo trick [60] when sampling from the policy. To compute the inner expectation in Equation 16, we simply sample single rollout. In practice we found that directly using the mean without sampling is also sufficient.
B.3 Reward Function
Sparse reward: Recall that we use the following reward function
{-1
-1 if an infraction occurs
R a (8)
(20)
otherwise.
In our experiments, we consider collisions events and driving off-road as infractions. Collisions are computed by checking for overlap between the bounding boxes of agents. Off-road is computed by checking if an agent's bounding box still intersects with the road polygon.
Note that when optimizing the reward, we apply early termination of the scenario in the event of infraction. We treat infractions terminal states in the MDP for few reasons. Regarding collision, it is unclear what the optimal behavior (or recovery) looks like after collision. Similarly, for driving off-road, the actor is likely in a state that it is physically impossible to recover from the real world, as an off-road event would imply the actor has driven off the shoulder into a divider. Finally, in early experiments, we found that continuing simulation for off-road events (and not modeling any shoulders or dividers, physics of off-road driving, etc.) would slow down training since in early phases the policy would drive off-road very early and very severely with no hope of recovering. Resetting in this prevents wasted simulation in very disttribbtti states where the policy is completely off the map, etc.
Early Termination:
17

---

Shaped reward: For the RL-Shaped baseline, use the same reward in Equation 20 with an addi- tional term which encourages driving at the speed limit.
R shaped ) (s, a) 0.5(C
(21)
where 8 = abs limit) and C 30, For the shaped reward, we additionally terminate the episode if 5 2 C.
B.4 Reinforcement Learning Loss
We describe our factorized approach to multiagent PPO [36] in more detail. Starting off we compute a per-agent probability ratio.
(0) (a (a())
(22)
Told [s
Our centralized value-function uses the same architecture as our policy, and computes per-agent value estimates (s). Details of the architecture are found in Section B.6. The value model
is trained using per-agent value targets, which are computed with per-agent rewards R(O)
R(0) (st, ai (8)
Σ (V76) 220)2
(23)
T
V
7 (R
(24)
1=0
We can obtain a per-agent GAE using the value model well.
A = GAE(R
R (8T))
(25)
The PPO policy loss is simply the sum of per-agent PPO loss,
N
Coolicy
min() ( clip(r!)), 1 c, + 1
())
(26)
Finally, the overall loss is the sum of the policy and value learning loss.
CRL
Croliey Cvalue
(27)
B.5 Input Parameterization
Agent history: Following [62], we adopt an viewpoint invariant representation of an agent's past trajectory. We encode the past trajectory as a sequence of pair-wise relative positional encodings between the past waypoints and the current pose. Each relative positional encoding consists of the sine and cosine of distance and heading difference of pair of poses. See [62] for details.
Lane graph: To construct our lane graph representation G (V, E), We first obtain the lane graph nodes by discretizing centerlines in the high-definition (HD) map into lane segments every 10m. We use length, width, curvature, speed limit, and lane boundary type (e.g., solid, dashed) as node features. Following [66], we then connect nodes with 4 different relationships: successors, predecessors, left and right neighbors.
B.6 Model Architecture
Briefly, the RTR model architecture is composed of three main building blocks: (1) context encoders for embedding lane graph and agent history inputs; (2) interaction module for capturing scene- interaction; and (3a) action decoder for parameterizing the per-agent policy and (3b) value decoder for the value model. Note that the policy model and the value model use the same architecture, but are trained completely separately and do not share any parameters. Early experiments found that not sharing parameters resulted in more stable training hypothesize that this is likely because this approach prevents updates the policy from interfering with the value function, and vice versa.
18

---

History encoder: The history encoder consists of a ID residual neural network (ResNet) followed by gated recurrent unit (GRU) that extracts agent features ha f(s(9) from sliding window of past agent states Intuitively, the ID CNN captures local temporal patterns, and the GRU aggregates them into global feature.
Lane graph encoder: The lane graph encoder is a graph convolutional network (GCN) [66] that extracts map features ln = g(m) from given lane-graph G of map m. We use hidden channel dimensions of [128, 128, 128, 128], layer normalization (LN), and max pooling aggregation.
Interaction module: To model scene-level interaction (i.e., agent-to-agent, agent-to-map. and map-to-map), we build heterogeneous spatial graph G" by adding agent nodes the original lane graph G. Besides the original lane graph edges, connect agent nodes to their closest lane graph nodes. All agent nodes also fully connected to each other. We use scene encoder parameter- ized by heterogeneous graph neural network (HeteroGNN) [62] to process map features and agent features into fused features,
(1(1) H H HeteroGNN (2) ,..., N,,),, (h. Nm)).
(28)
These fused features then provided input to the decoder.
Action decoder: Finally, we pass the fused features into a 4-layer MLP with hidden dimensions [128, 128, 128] to predict agent's 's acceleration and steering angle distributions (parameterized as Normals).
( = (L)(()()) T(al)ss) (2) o(1)
(29)
(30)
Value decoder: For the value model, 4-layer MIP instead regresses single scalar value repre- senting the value
T7(1) = MLP value (Tvalue)
(6)
(31)
B.7 Kinematic Bicycle Model
We use kinematic bicycle model [59] for our environment dynamics. The bicycle model state is given as
(r. y. 0 v))
(32)
where y is the position of the center of the rear axel, 0 is the yaw, and v is the velocity. The bicycle model actions are
a == (u; Φ)
(33)
where u is the acceleration, and is the steering angle. The dynamics function á =[(s, a) is then defined as
à cos (0)
(34)
Ú = e sin (0)
(35)
Ó U
Lan lan
(36)
T.
0 = u
(37)
where T. is wheelbase length, i.e. the distance between the rear and front axel. We can use simple finite difference approach to computing the next state
81+1 = + ((1, )dt
(38)
where dt is chosen to be 0.5 seconds in practice. We can apply the bicycle model to each agent individually to obtain the joint state dynamics function.
19

---

Algorithm RTR Closed-loop Learning
1:
for n = 1, ... ,N do
2: Set L <- 0.
3: Set LRL f- O.
4:
kork= k
K do
5:
Sample imitial state ~ (1 - p + app.
6:
Generate trajectory using policy Te (a|s) from Equation and simulator.
7:
Compute CRL. &- LRL R(T).
8:
if initial state Su from Nominal Dataset then
9:
_ + L D(T
10:
end if
11:
end for
Compute g t- Vo LRL using our factorized PPO.
13:
Compute gtl. + Vo flL using BPTT.
14:
Update 0 with g&L + nt using AdamW.
15:
end for
B.8
Training Details
We use A [67] as our optimizer, and decay the learning rate by factor 0.2 every 3 epochs, and train for total of 10 epochs. We provide additional training hyperparameters in Table 7, Our overall learning process is summarized in Algorithm 1.
Hyperparameter
Value
IL minibatch size
32
PPO batch size
192
PPO minibatch size
32
PPO num epochs
PPO clip
0.2
Discount factor
0.79
Learning rate
0.00001
Weight decay
0.0001
GAE X
1.0
Grad clip norm
1.0
Table 7: Training hyperparameters
20

---

204
112
111
0.4
03
6.1
4.3
5.0
-2
333
045
0.35
332
0.15
333
040
646
332
0.10
0:8
8.6
2.3
24
100
15
330
53
369
159
200
0.00
330
150
389
646
100
156
330
3.9
56
101
359
200
0.00
59
330
150
30
0.23
0.20
6.0
032
0.89
0.10
630
333
0.50
000
1-8
3.5
2.0
0.00
600
10 155
310
18
0.00
53
1.2
Figure 12: Histograms of scenario features for all methods used to compute JSD distributional realism metrics. We see that BC and RL methods often struggle with capturing the data distribution compared to II. and RTR. Notably, RTR closely matches II. performance in distributional realism, while greatly improving infraction rate in other results.
(a) BC
(b) IL
(c) RL
RL-Shaped
BC+RL
(f) RTR (ours)
Controlled (start)
Controlled (end)
Hero (start)
Hero (end)
Other Controlled
Figure 13: Qualitative results a fork scenario. BC drives off the road, II. results in collision while RL. and BC+RL slow down. RL-Shaped drives straight and loses the interesting lane change behavior.
21

---

(a) BC
(b) IL
(c) RL
(d) RL-Shaped
BC+RL
(f) RTR (ours)
Controlled (start)
Controlled (end)
Hero (start)
Hero (end)
Other Controlled
Figure 14: Qualitative results on a merge scenario. We see that RL methods slow down unrealisti- cally. IL results in collision while RTR maintains realism.
(a) BC
(b) IL
(c) RL
(d) RI--Shaped
BC+RI.
(f) RTR (ours)
Controlled (start)
Controlled (end)
Hero (start)
Hero (end)
Other Controlled
Figure 15: Qualitative results on procedurally generated merge scenario. collision. RTR maintains realism.
II. and BC result
(a) BC
(b) IL
(c) RL
(d) RL-Shaped
BC+RI.
(f) RTR (ours)
Cantrolled (start)
Controlled (and)
Hero (start)
Hara (end)
Other Controlled
Figure 16: Qualitative results on a procedurally generated cut-in scenario. BC+RI. drives off the road, while IL and RL-shaped result in a collision. RTR maintains realism.
22


Reviews:
[{"review_id": "P9g3XIt3-p", "reviewer": "Reviewer_ohnK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed a combined imitation learning (IL) and reinforcement learning (RL) approach, named Reinforcing Traffic Rules (RTR), to learn driving policies for traffic agents in closed-loop. RTR optimizes the IL component by minimizing the sum of distance between the expert states and policy states with huber loss. Given the differentiable transition dynamics, the policy can be directly optimized with the reparameterization trick. The RL component is optimized by standard on-policy methods, i.e., PPO with a factorized multi-agent reward function that penalizes infractions such as collisions and off-roads.\n\nThe paper also simulates long-tail scenarios based on logical scenarios to evaluate the out-of-distribution generalization of the RTR method. The paper conducts extensive experiments to demonstrate that the proposed method is effective for both learning realistic driving behavior and being able to avoid driving safety events. \n\n", "summary_of_recommendation": "The paper proposed a combined imitation learning (IL) and reinforcement learning (RL) approach, named Reinforcing Traffic Rules (RTR), to learn driving policies for traffic agents in closed-loop. RTR optimizes the IL component by minimizing the sum of distance between the expert states and policy states with huber loss. The RL component is optimized by standard on-policy methods, i.e., PPO with a factorized multi-agent reward function that penalizes infractions such as collisions and off-roads. Long-tail scenarios are generated based on logical scenarios to evaluate the out-of-distribution generalization of the method. The paper conducts extensive experiments to demonstrate that the effectiveness of the proposed method.\n\n- The paper is well written and easy to follow.\n- Although the technical novelty is limited, the results are relevant and the method is likely to be adopted. \n\nI recommend a weak accept.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Any comparisons with other long-tail scenario generation methods? No related work in long-tail scenario generation. \n- Could the authors provide the reward definition? How does reward shaping affect the performance?\n- The progress is not reported as a metric. I would be curious to know if the method makes sufficient progress while maintaining low infractions and high imitation fidelity. \n- Is the hero agent log-following for the nominal scenarios? Have the authors explored any other hero agent policies?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Quality\n- The paper is technically sound. The claims are supported by the empirical experimental results. The method is appropriate. \n\n#### Clarity\n- The paper is relatively clearly written and well organized. \n\n#### Originality\n- Neither the problem nor the method is new. The proposed method is a combination of GAIL (without the discriminator, but instead using a concrete distance function) and standard on-policy RL with factorized rewards in a multi-agent setting. The method is incremental to the previous similar methods such as BC+RL [1].\n\n\n#### Significance\n- The results are interesting and relevant to the field. Others are likely to use the idea. \n\n#### Strengths \n- The paper is relevantly well written\n- The proposed method is straightforward and effective - it’s likely to be adopted. \n- Extensive experiments (both qualitative and quantitative). Ablation results show the effects of a few important parameters.\n\n#### Weaknesses\n- The proposed method is a combination of existing well known methods. Technical novelty is limited. \n- The experimental results largely follow the similar results found in paper [1]. \n- Dataset is relatively small (~600 segments in total).\n- No related work in long-tail scenario generation. Although the paper mentioned adversarial approaches in the limitation paragraph.\n\n\n[1] Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8qqDTkqypV", "reviewer": "Reviewer_CEx1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Simulation is the very first step of deploying self-driving systems into the real world. The key to traffic simulation is ensuring the multi-agent system is realistic and able to react to other vehicles, i.e. ego-car with an autonomous driving system. This paper proposes a new method called RTR to simulate the traffic where simulated vehicles can not only retain the realistic driving behaviors in recorded driving logs but solve long-tail safety-critical scenarios. Extensive experiments are conducted to prove the effectiveness and advantages of RTR with both qualitative and quantitative results.", "summary_of_recommendation": "I recommend accepting this paper. But it is still likely to increase or decrease my score depending on the rebuttal and other reviewers' comments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "N/A", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strength:\n1. The motivation is good. Reactive traffic simulation is important to Autonomous Driving (AD) field, especially considering almost every company uses the recorded and fixed driving log to test their systems.  \n2. Technically, the method provides some interesting ideas. For example, combining recorded data and synthetic data together to build a mixed dataset enables the policy to solve interaction-rich safety-critical scenarios which are rare and impossible to record in the real world; The IL objective is learned through differential simulation via bicycle model; The centralized policy design encoding the whole context, avoiding computational overhead.\n3. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. Also, downstream tasks prove the potential of the traffic simulation model. \n\n### Suggestions:\n1. Some references can be included to provide recent advances in traffic simulation.\n>Zhong, Ziyuan, et al. \"Guided conditional diffusion for controllable traffic simulation.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.  \nFeng, Lan, et al. \"Trafficgen: Learning to generate diverse and realistic traffic scenarios.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.  \nRempe, Davis, et al. \"Generating useful accident-prone driving scenarios via a learned traffic prior.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. \nTwo of them, CTG and STRIVE, are related to safety-critical or long-tail scenario generation based on original scenarios. The other one, TrafficGen, focuses on simulating both initial conditions and subsequent trajectory generation given an empty map. The settings of these papers is not totally as same as RTR, but some overlaps do exist.\n\n2. A improvement of the method in my mind is to exempt the need for estimating the value function. Actually, one can directly optimize the RL objective with a differential simulation. Note I am not saying you should try this right now. I just want to give a potential improvement that can be adopted to make RTR better. With the differentiable bicycle model, the displacements between two frames can serve as dense rewards, and a termination reward can be raised by collision or driving off the road. In this way, both IL and RL would be unified into the RL framework whose goal is to follow a referenced trajectory. Besides, the dense reward signal can enjoy the TD, making the imitation episode-wise, which is not allowed by doing step-wise imitation learning through current IL loss. One feasible method is:\n>Mora, Miguel Angel Zamora, et al. \"Pods: Policy optimization via differentiable simulation.\" International Conference on Machine Learning. PMLR, 2021.\n\n3. Consider replacing long-tail with safety-critical, which is more accurate than long-tail and widely used/accepted\n4. typo line 241\n5. please report std in all tables\n6. I think one ablation result should be added where the $\\lambda$ is set to zero and thoroughly ablate the RL component. A performance drop in terms of collision/driving off the road can demonstrate the necessity to have RL loss.\n7. The conclusion of section 4.2 is that the generated scenarios can be used to train the trajectory prediction model. Indeed, the model trained with driving logs can generate a surrogate training dataset, while the performance of training a prediction model with the surrogate data is supposed to be worse than training the prediction model directly with the recorded scenario. This makes this application not interesting and promising enough. However, this experiment is actually good evidence demonstrating your agent model learns better data distribution and driving behaviors than other baselines according to the evaluation results on the held-out real-world test set. Thus I recommend using this experiment to support your main result: RTR can learn realistic driving behavior. \n\nBTW, I think this model-based motion prediction is an interesting application, as most of the motion prediction methods directly regress the velocity/position of the next several frames. Unlike these methods, RTR applies hard kinematics constraints by using the bicycle model and predicts actions that will be further processed by the model, which may produce better prediction results. Maybe authors can evaluate it on Waymo or nuScenes data to see if it can achieve SOTA.\n\n\n\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "3_AJwTlczXs", "reviewer": "Reviewer_F38n", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper investigates the traffic agent behavior modeling task using a joint IL and RL approach in a closed-loop environment. The reward for RL is designed with a sparse signal. Both nominal and long-tail scenarios are generated using the proposed methods. Further, the effectiveness of scenario generation is given for the downstream prediction metrics.", "summary_of_recommendation": "This paper investigates the traffic agent behavior modeling task which is  important  for high-level autonomous driving. However, the motivation and the experiments are both inadequate and incomplete.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "How about the  scaling and training efficiency of the proposed RTR?\nHow about the performances with the increasing number of sim agents?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There are some suggestions as follows:\n\n1. For the motivation and introduction, there is no doubt that the closed-loop is beneficial for modeling realistic behaviors of traffic agents. An existing work [1] also investigates this problem using the IL+RL framework with a closed-loop simulator, and the motivation and differences should be given.\n\n2. For scenario generation, another important attribute aside from being realistic is being controllable [2, 3]. However, this part is included in the proposed RTR.\n\n3. The biggest problem with RL-based closed loops is scaling and training efficiency. I guess it is one reason why the experiments are carried out on a small dataset with relatively simple highway scenarios. The open-sourced WOMD dataset with urban scenarios should be considered, and stronger baselines should be compared.\n\n4. How many agents are controlled for the nominal and long-tail scenarios? I see only two agents are controlled in the supplementary material. How about the performances with the increasing number of sim agents?\n\n5. The details about the downstream prediction are not given such as the evaluation data. Is it an open-sourced benchmark? I'm not sure about the credibility due to too many missing details.\n\n[1] TrajGEN: Generating realistic and diverse trajectories with reactive and feasible agent behaviors for autonomous driving IEEE Transactions on Intelligent Transportation Systems, 2022\n[2] Guided Conditional Diffusion for Controllable Traffic Simulation, arXiv, 2022.\n[3] MixSim: A Hierarchical Framework for Mixed Reality Traffic Simulation, CVPR, 2023.\n[4] TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios, ICRA, 2023.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "197TGUxNbvP", "reviewer": "Reviewer_gvqu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method of combining RL and Imitation Learning objectives to improve the realism of simulated traffic agents in application to AV. To this end, the authors combine pre-collected datasets from human driving together with scripted long-tail scenarios for closed-loop learning with RL objective. To overcome problems of credit assignment in RL objective, the paper introduces agent-factorized rewards, policy, and value estimation (together with the advantage estimator). \nThe method is evaluated in simulated and held-out pre-recorded human driving, showing improvements in collision avoidance and reduction in FDE metric measuring realism of traffic. ", "summary_of_recommendation": "The paper is enjoyable to read and introduces interesting contributions, tackles important problems, and considers a quite wide range of baselines for comparison. The overall results are quite valuable for RL practitioners in AV.\nOn the downside, it has a somewhat minimalistic scope and does not bring completely novel/ground-breaking ideas that could have a major impact on self-driving. Neither does it tackle the most intricate aspect of AV - long tail scenario generation. \n\nConsidering all that, I am inclined toward accepting the paper, but can't suggest a \"strong accept\" at this point.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The authors mentioned that the value network does not share parameters with the policy. Could the authors propose insights if this was detrimental to learning?\n- The Fig. 3 is worth bringing to the unified scale across domains with a much finer grid step to see jumps in performance across different datasets.\n- The authors mentioned early termination of episodes upon the infraction. Was it actually necessary? Shouldn't agents learn faster if allowed to continue simulation due to occasional recovery and, hence, receiving more detailed reward-based feedback?\n- The baseline BC + RL seems to be the worst at imitation. It is a bit surprising. Could the authors provide a more detailed analysis of why this happens?\n- How validation/testing is separated? It is especially interesting to see if testing was performed at the level of different logical scenarios vs just specific instances of the same logical scenarios.\n- Do the authors plan to release the source code? \n- Could the authors propose how their approach could be extended toward generating long-tail events automatically? \n- Some citations are definitely missing: \n  - PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation. This work has quite extensive validation, including real driving, and combines RL and IL objectives to retain realism and reduce infractions.\n  - BITS: Bi-level Imitation for Traffic Simulation. (I generally couldn't find any citations from M. Pavone group, although they have a lot of work toward traffic realism and prediction). This work also provides quite extensive ablation and goes beyond standard metrics for realism evaluation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Praise:\n- The paper tackles an important problem of improvement of traffic simulation realism.\n- The idea of factorizing value estimation together with the policy is interesting. \n- The work compares against a wide range of baselines. The findings of ablation studies in this work are important for RL practitioners in AV. \n- Demonstration in the improvement of downstream tasks (prediction) using data simulated with the trained policy is a somewhat surprising and valuable outcome. \n\nConcerns:\n- The infraction set considered in this work is very minimalistic and only considers basic events: road departure and collisions. Traffic violations, in reality, are much more complex.\n- Albeit FDE metric is a standard proxy for measuring realism, it is very indifferent toward the types of mistakes that are made by the traffic participants. In other words, different mistakes with the same metric might introduce drastically different outcomes for the behavior of surrounding agents. At this point, it would be interesting to see more sophisticated metrics.\n- Although the results are quite interesting the main idea is not particularly groundbreaking and more or less incremental improvement upon the work \"Imitation is not enough\". ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors' response has addressed most of the reviewers' concerns, and the general assessment is that this paper provides a good contribution to CoRL. In the final version of this paper, the authors should carefully revise the paper to address all reviewers' comments / suggestions.
Decision: Accept (Poster)