Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a method of combining RL and Imitation Learning objectives to improve the realism of simulated traffic agents in application to AV. To this end, the authors combine pre-collected datasets from human driving together with scripted long-tail scenarios for closed-loop learning with RL objective. To overcome problems of credit assignment in RL objective, the paper introduces agent-factorized rewards, policy, and value estimation (together with the advantage estimator). 
The method is evaluated in simulated and held-out pre-recorded human driving, showing improvements in collision avoidance and reduction in FDE metric measuring realism of traffic. 
review: Praise:
- The paper tackles an important problem of improvement of traffic simulation realism.
- The idea of factorizing value estimation together with the policy is interesting. 
- The work compares against a wide range of baselines. The findings of ablation studies in this work are important for RL practitioners in AV. 
- Demonstration in the improvement of downstream tasks (prediction) using data simulated with the trained policy is a somewhat surprising and valuable outcome. 

Concerns:
- The infraction set considered in this work is very minimalistic and only considers basic events: road departure and collisions. Traffic violations, in reality, are much more complex.
- Albeit FDE metric is a standard proxy for measuring realism, it is very indifferent toward the types of mistakes that are made by the traffic participants. In other words, different mistakes with the same metric might introduce drastically different outcomes for the behavior of surrounding agents. At this point, it would be interesting to see more sophisticated metrics.
- Although the results are quite interesting the main idea is not particularly groundbreaking and more or less incremental improvement upon the work "Imitation is not enough". 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is enjoyable to read and introduces interesting contributions, tackles important problems, and considers a quite wide range of baselines for comparison. The overall results are quite valuable for RL practitioners in AV.
On the downside, it has a somewhat minimalistic scope and does not bring completely novel/ground-breaking ideas that could have a major impact on self-driving. Neither does it tackle the most intricate aspect of AV - long tail scenario generation. 

Considering all that, I am inclined toward accepting the paper, but can't suggest a "strong accept" at this point.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposed a combined imitation learning (IL) and reinforcement learning (RL) approach, named Reinforcing Traffic Rules (RTR), to learn driving policies for traffic agents in closed-loop. RTR optimizes the IL component by minimizing the sum of distance between the expert states and policy states with huber loss. Given the differentiable transition dynamics, the policy can be directly optimized with the reparameterization trick. The RL component is optimized by standard on-policy methods, i.e., PPO with a factorized multi-agent reward function that penalizes infractions such as collisions and off-roads.

The paper also simulates long-tail scenarios based on logical scenarios to evaluate the out-of-distribution generalization of the RTR method. The paper conducts extensive experiments to demonstrate that the proposed method is effective for both learning realistic driving behavior and being able to avoid driving safety events. 


review: #### Quality
- The paper is technically sound. The claims are supported by the empirical experimental results. The method is appropriate. 

#### Clarity
- The paper is relatively clearly written and well organized. 

#### Originality
- Neither the problem nor the method is new. The proposed method is a combination of GAIL (without the discriminator, but instead using a concrete distance function) and standard on-policy RL with factorized rewards in a multi-agent setting. The method is incremental to the previous similar methods such as BC+RL [1].


#### Significance
- The results are interesting and relevant to the field. Others are likely to use the idea. 

#### Strengths 
- The paper is relevantly well written
- The proposed method is straightforward and effective - it’s likely to be adopted. 
- Extensive experiments (both qualitative and quantitative). Ablation results show the effects of a few important parameters.

#### Weaknesses
- The proposed method is a combination of existing well known methods. Technical novelty is limited. 
- The experimental results largely follow the similar results found in paper [1]. 
- Dataset is relatively small (~600 segments in total).
- No related work in long-tail scenario generation. Although the paper mentioned adversarial approaches in the limitation paragraph.


[1] Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposed a combined imitation learning (IL) and reinforcement learning (RL) approach, named Reinforcing Traffic Rules (RTR), to learn driving policies for traffic agents in closed-loop. RTR optimizes the IL component by minimizing the sum of distance between the expert states and policy states with huber loss. The RL component is optimized by standard on-policy methods, i.e., PPO with a factorized multi-agent reward function that penalizes infractions such as collisions and off-roads. Long-tail scenarios are generated based on logical scenarios to evaluate the out-of-distribution generalization of the method. The paper conducts extensive experiments to demonstrate that the effectiveness of the proposed method.

- The paper is well written and easy to follow.
- Although the technical novelty is limited, the results are relevant and the method is likely to be adopted. 

I recommend a weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: Simulation is the very first step of deploying self-driving systems into the real world. The key to traffic simulation is ensuring the multi-agent system is realistic and able to react to other vehicles, i.e. ego-car with an autonomous driving system. This paper proposes a new method called RTR to simulate the traffic where simulated vehicles can not only retain the realistic driving behaviors in recorded driving logs but solve long-tail safety-critical scenarios. Extensive experiments are conducted to prove the effectiveness and advantages of RTR with both qualitative and quantitative results.
review: ### Strength:
1. The motivation is good. Reactive traffic simulation is important to Autonomous Driving (AD) field, especially considering almost every company uses the recorded and fixed driving log to test their systems.  
2. Technically, the method provides some interesting ideas. For example, combining recorded data and synthetic data together to build a mixed dataset enables the policy to solve interaction-rich safety-critical scenarios which are rare and impossible to record in the real world; The IL objective is learned through differential simulation via bicycle model; The centralized policy design encoding the whole context, avoiding computational overhead.
3. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. Also, downstream tasks prove the potential of the traffic simulation model. 

### Suggestions:
1. Some references can be included to provide recent advances in traffic simulation.
>Zhong, Ziyuan, et al. "Guided conditional diffusion for controllable traffic simulation." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.  
Feng, Lan, et al. "Trafficgen: Learning to generate diverse and realistic traffic scenarios." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.  
Rempe, Davis, et al. "Generating useful accident-prone driving scenarios via a learned traffic prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 
Two of them, CTG and STRIVE, are related to safety-critical or long-tail scenario generation based on original scenarios. The other one, TrafficGen, focuses on simulating both initial conditions and subsequent trajectory generation given an empty map. The settings of these papers is not totally as same as RTR, but some overlaps do exist.

2. A improvement of the method in my mind is to exempt the need for estimating the value function. Actually, one can directly optimize the RL objective with a differential simulation. Note I am not saying you should try this right now. I just want to give a potential improvement that can be adopted to make RTR better. With the differentiable bicycle model, the displacements between two frames can serve as dense rewards, and a termination reward can be raised by collision or driving off the road. In this way, both IL and RL would be unified into the RL framework whose goal is to follow a referenced trajectory. Besides, the dense reward signal can enjoy the TD, making the imitation episode-wise, which is not allowed by doing step-wise imitation learning through current IL loss. One feasible method is:
>Mora, Miguel Angel Zamora, et al. "Pods: Policy optimization via differentiable simulation." International Conference on Machine Learning. PMLR, 2021.

3. Consider replacing long-tail with safety-critical, which is more accurate than long-tail and widely used/accepted
4. typo line 241
5. please report std in all tables
6. I think one ablation result should be added where the $\lambda$ is set to zero and thoroughly ablate the RL component. A performance drop in terms of collision/driving off the road can demonstrate the necessity to have RL loss.
7. The conclusion of section 4.2 is that the generated scenarios can be used to train the trajectory prediction model. Indeed, the model trained with driving logs can generate a surrogate training dataset, while the performance of training a prediction model with the surrogate data is supposed to be worse than training the prediction model directly with the recorded scenario. This makes this application not interesting and promising enough. However, this experiment is actually good evidence demonstrating your agent model learns better data distribution and driving behaviors than other baselines according to the evaluation results on the held-out real-world test set. Thus I recommend using this experiment to support your main result: RTR can learn realistic driving behavior. 

BTW, I think this model-based motion prediction is an interesting application, as most of the motion prediction methods directly regress the velocity/position of the next several frames. Unlike these methods, RTR applies hard kinematics constraints by using the bicycle model and predicts actions that will be further processed by the model, which may produce better prediction results. Maybe authors can evaluate it on Waymo or nuScenes data to see if it can achieve SOTA.





robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend accepting this paper. But it is still likely to increase or decrease my score depending on the rebuttal and other reviewers' comments.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper investigates the traffic agent behavior modeling task using a joint IL and RL approach in a closed-loop environment. The reward for RL is designed with a sparse signal. Both nominal and long-tail scenarios are generated using the proposed methods. Further, the effectiveness of scenario generation is given for the downstream prediction metrics.
review: There are some suggestions as follows:

1. For the motivation and introduction, there is no doubt that the closed-loop is beneficial for modeling realistic behaviors of traffic agents. An existing work [1] also investigates this problem using the IL+RL framework with a closed-loop simulator, and the motivation and differences should be given.

2. For scenario generation, another important attribute aside from being realistic is being controllable [2, 3]. However, this part is included in the proposed RTR.

3. The biggest problem with RL-based closed loops is scaling and training efficiency. I guess it is one reason why the experiments are carried out on a small dataset with relatively simple highway scenarios. The open-sourced WOMD dataset with urban scenarios should be considered, and stronger baselines should be compared.

4. How many agents are controlled for the nominal and long-tail scenarios? I see only two agents are controlled in the supplementary material. How about the performances with the increasing number of sim agents?

5. The details about the downstream prediction are not given such as the evaluation data. Is it an open-sourced benchmark? I'm not sure about the credibility due to too many missing details.

[1] TrajGEN: Generating realistic and diverse trajectories with reactive and feasible agent behaviors for autonomous driving IEEE Transactions on Intelligent Transportation Systems, 2022
[2] Guided Conditional Diffusion for Controllable Traffic Simulation, arXiv, 2022.
[3] MixSim: A Hierarchical Framework for Mixed Reality Traffic Simulation, CVPR, 2023.
[4] TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios, ICRA, 2023.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: This paper investigates the traffic agent behavior modeling task which is  important  for high-level autonomous driving. However, the motivation and the experiments are both inadequate and incomplete.
review_title: null

Meta-review: The authors' response has addressed most of the reviewers' concerns, and the general assessment is that this paper provides a good contribution to CoRL. In the final version of this paper, the authors should carefully revise the paper to address all reviewers' comments / suggestions.
Decision: Accept (Poster)