Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors build a traffic simulator in which each agent drives in order to maximize its reward as well as the mean reward of nearby agents. The extent to which the vehicle optimizes for the reward of other agents $c_\beta$ can be adjusted. The authors then show that if a new agent is trained in such a simulator and during training the parameter for controlling $c_\beta$ is adversarially optimized, the resultant policy transfers better than baselines to new traffic densities and test-time out-of-distribution NPC controllers (e.g. IDM).
strength_and_weaknesses: One strength of this paper is that it addresses multiple impactful problems in self-driving simulation. Generally speaking, adversarial NPCs exhibit unrealistic aggressive behavior. The idea of controlling the extent to which the NPCs optimize for the reward of nearby agents seems like a reasonable way to parameterize the space of plausible vehicle behaviors that an AV may interact with on real roads.

That being said, the main weakness for me reading this paper was that I found it hard to understand precisely what the authors claim as a contribution in this paper. My sense is that the main contribution is the use of $c_\beta$ as the variable to adversarially optimize. However, my sense is another contribution is maybe the construction of NPCs that have such a $c_\beta$? Or is the main contribution the policy that is ultimately trained using Algorithm 1 which they show can be transferred to new test-time simulators with out-of-distribution NPC controllers? I'm not seeing yet what is novel about Algorithm 1 - do the authors define what is different between "misunderstanding-based adversarial learning" and standard adversarial learning?

I list my other minor edits and questions below:

- state space - no notion of length or width?

- "Although IPL is prone to generate egoistic suboptimal behaviors" - is there a citation for this claim?

- Figure 3 - do the authors compare against the baseline of training the policy on a hard-coded uniform distribution for $c_\beta$ instead of the adversarially optimized distribution that they find for $c_\beta$?

- Section 5.1 - clarify why would the initial poses being close together imply that it's harder for the agents to coordinate?

- Figure 5 - I think it's easier to read if the same coordinate frame is used for each column

- Figure 6 - I find this figure difficult to read. I think it would be better if the authors use the same colormap for all grid cells instead of using blue only for the diagonal.

- Do the authors have videos they can share? It's hard to understand the effect of different $c_\beta$ from frame shots.

- Section 5.3 - I'm not sure what it means for the evaluation environment to be "IID with training environment"?

- Figures and Tables - I feel "ours" is being used to sometimes denote NPC policies, sometimes ego policies, and sometimes training environments. Is there a way the authors could disambiguate each of these contributions?
clarity,_quality,_novelty_and_reproducibility: I think the authors can improve the clarity of the submission substantially. Specifically, I'm having trouble identifying the main result that the authors achieve using their method. I'm also somewhat confused about Section 3.3 - is there a reason the authors use this architecture compared to prior work? Can the authors define formally what it means for their framework to be "misunderstanding-based"? Once I have a better sense of the main takeaway of the paper, I will be more confident in commenting on the originality of the paper.
summary_of_the_review: I'm currently unclear on how to interpret some of the main claims of the paper, namely whether the authors are claiming that they have a general-purpose driving policy, or a general-purpose simulator in which robust driving policies can be trained. I also think that qualitative videos would make it much easier to judge the quality of the contribution, since most of the metrics are designed by the authors. If the authors clarify their contribution, I am willing to increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel adversarial framework to train driving policies under learned traffic flow in simulation. It first extends traffic flow learned with reinforcement learning by exposing each other’s intrinsic social value orientations (SVO). Next, it uses this exposed SVO to train adversarial agents as well as a policy. In particular, the traffic flow accesses the driving policy’s SVO but not vice versa (misunderstanding). Experiments show that 1. The new traffic flow with exposed SVO to each other learns better behavior 2. The proposed adversarial training effectively reduces driving policy’s performance and 3. Driving policy has better zero-shot transferability across different traffic flows when trained with the proposed method. 

strength_and_weaknesses: ## Strengths 
+ The paper is well-written and the presentation is strong.
+ The proposed method demonstrates convincing empirical results.
+ The paper proposes a novel way to adversarially train driving policies in traffic flow, and actually demonstrates that agents trained with the proposed method drive (transfers) better, which is a big plus.

## Weaknesses
- I can’t seem to find sample complexity information on traffic flow training in the paper. I am curious to see whether exposing SVO improves/decreases sample complexity with respect to the metrics in Table 1, and how it compares to the baseline CoPO.
- Table 1 compares how the proposed method impedes the driving policy. I am curious to see whether this result generalizes across different driving policies/architectures. 
- I would like to see an ablation that justifies the “misunderstanding” part of the claim/story. For example, an ablation in which the driving policy also sees the SVO of the nearby traffic flow.
- The adversarial training currently strictly comes from the adversarial agent. Since the background traffic already coordinates with each other, I am wondering why not directly alternate between training between the driving policy and background traffic flow, and have the traffic flow perform multi-agent coordinated attacks to the driving policy.


clarity,_quality,_novelty_and_reproducibility: Clarity: Great; the paper is well written

Quality: Good

Originality: Good

Reproducibility: the method seems to extend upon prior work CoPO which is open-sourced; it hence seems to be reproducible based on information provided in the paper.
summary_of_the_review: The paper presents a novel framework to train driving policies under coordinated traffic flow. Experiments are thorough and demonstrate **constructive** usefulness of the presented approach. I personally think this paper is worth acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work builds a framework for traffic simulation and propose a form of adversarial training to improve driving policies.  
In their traffic flow, they assume the agents allowed to communicate *social value orientations* (SVOs) and introduce an adversarial agent that tries to minimize the overall reward for the driving agent by only modifying the SVO.
They show strong improvements on their coordinated traffic flow task on a variety of subtasks (intersection, bottleneck, merge, roundabout).
strength_and_weaknesses: ### Strengths

* Experiments show clear improvements over IDM, FLOW, COPO
* Overall writing is good, mathematical notion is for the most part clear

### Weaknesses

* Reproducibility - all experiments were performed on "our internal driving simulator". Can the authors give more details on this? This is a major factor in my decision.
* The experiments were all done in this one simulator and it is unclear if the effectiveness of this strategy translates to other domains.
* Writing/figure quality (more details below) - a couple figure/tables are quite hard to understand and could be presented better.

### Minor Issues / Questions

* What is the mathematical equation for the reward function? Were any other reward functions tried?
* Algorithm 1: in my opinion this is described well by Sec 4.2 and can be taken out.
* Table 1: Highlighted colors seem arbitrary - why is the "0.0" colored in "Ours + Wrong Lane"?
* Figure 6 took quite a while to understand what I was looking at. FailMaker seems bad and can be taken out and just described in text.
clarity,_quality,_novelty_and_reproducibility: **Clarity, Quality:** overall good quality and clarity if some of the figures/tables are edited.  
**Novelty:** their method for adversarially attacking SVO seems relatively novel.  
**Reproducibility:** the authors provide details about their architecture and learning algorithm, but I am unsure if this is reproducible without more training details and especially the simulator.
summary_of_the_review: This work introduces a fairly novel approach and overall shows improved performance across various scenarios in their simulator. The major factor in my current decision is on reproducibility.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a robust RL policy training framework which applies adversarial training using coordinated traffic flow. For building the traffic flow, it uses the so-called “Social Value Orientations (SVOs)” (which is a weight that balances the proportion of each agent’s own driving reward and its surrounding agents driving rewards in the agent’s aggregated optimization reward) and allows the communication of SVOs among the background agents in the traffic flow. Adversary emerges when the SVOs of the driving agent sent to the traffic flow agents are perturbed. The problem can then be turned into a min-max optimization problem where the driving policy maximizes its reward and a spurious adversarial policy minimizes it through changing the SVOs seen by the agents. The proposed method is evaluated in terms of the safety, speed, and success rate of the generated traffic flows, the effect of adversary based on coordinated traffic flow, and the zero-shot transfer of the driving policy.
strength_and_weaknesses: ## Pros:

1.Leveraging SVOs for adversarial attack are moderately novel and its effect on improving zero-shot transfer of the driving policy seems promising.

2.generalization of the training policy is an important problem

## Cons:

1.Communicating SVOs among agents seem to be a relatively incremental improvement over [Peng 2021].

2.One issue is that the realism evaluation of the settings is missing 
Are the generated trajectories realistic and initial positions realistic? One way to evaluate if the trajectories are realistic is to look into acceleration (both longitudinal and latitudinal) and jerk distributions and compare them against some similar scenarios in real world datasets (e.g., nuScenes and Waymo Open).

3.some details are missing
see “Clarity, Quality, Novelty And Reproducibility” below.

4.effect of adversary is not that large
Table 1 success rates are not influenced much percentage wise. One usually expects the attack to have a stronger effect.

clarity,_quality,_novelty_and_reproducibility: ## Clarity

Some parts are not very clear. For example, the usage of SVOs and the key difference with [Peng 2021] need to be further clarified and highlighted. Experiment setting details are also missing.


1.Can you further clarify the key difference (using the notations in Sec 3.1), between the current work and [Peng 2021] in terms of “agents in their traffic flow have no access to other agents’ SVOs, leading to conservative behaviors”? Is it basically the policy beta will only have O_i and C_i as the inputs but not the C_j for others? Are all the C_i for each agent i also learned during the training?

2.How many different settings were used for each scenario during the evaluation?

3.in (2), why the optimization for beta is to solve n separate optimization objectives rather than their sum?

## Quality

The quantitative and qualitative results are moderately decent. One drawback is the realism issue I mentioned earlier. Scalability is also suspicious given it only has very limited scenario settings. However, this is somewhat acceptable given previous works e.g., [Peng 2021] along this line also does not evaluate realism and has limited scenarios to show effectiveness.

## Reproducibility

It is hard to reproduce given code is not available, details of simulator and experiment settings are all missing.


## Minor Issues:

-Sec 3.2 - state space - last two lines, probably should not overload notation c_i here.

summary_of_the_review: Overall the approach is moderately interesting and the results seem to be promising. However, some clarification needs to be made regarding the novelty of the method and the details of evaluation for a better evaluation of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of designing adversarial agents to improve learned policies of AI driving agents. In general this is a neat idea, although the reviewers expressed some concerns that it is not clear what is the actual novel contribution here. More generally, the reviewers expressed concerns regarding the clarity of the current paper, as well as difficulty understanding the quality of the experimental results given the closed-source nature of the simulator. I appreciate the authors will be open-sourcing their simulator soon - this should make it easier for reviewers to understand the empirical significance of this work. 
justification_for_why_not_higher_score: Although this is exciting work, the reviewers will be in a better position to assess the strength of the simulated results once the simulator is accessible and can be validated by the community.
Decision: Reject