Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a coordinated policy optimization algorithm to simulate self-driven particles and showcases its power in the autonomous driving application of traffic simulation. The algorithm proposes both local and global coordination mechanisms to tradeoff cooperation and competitiveness at the scene-level. The paper benchmarks the efficiency and safety of their method against a few baselines on a newly proposed simulator of heavily interactive scenarios.
main_review: The paper is well motivated from the perspective of the challenges. It is clear that the key challenge in simulating microscopic traffic is the agents' coordination, while things like dynamic feasibility etc. at the agent level are easier. However, I missed a more detailed motivation in terms of potential applications. I think this type of multi-agent simulation (or planning) have multiple applications such as testing/training a separate autonomy stack or the more hypothetical scenario in which all autonomous cars are driving with the same, share autonomy system. While the latter is well covered by the proposed framework, I believe there are no studies or intuition about the suitability of the proposed algorithm on the former. Would be great if the authors can clarify during rebuttal.

In terms of related work, the paper has pretty good coverage but I found 2 rather relevant works missing:
[A] Suo, Simon, et al. "TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. --> This is based on Imitation Learning but very focused on the interaction modeling of realistic, scene-level trajectories.
[B] Tang, Yichuan. "Towards learning multi-agent negotiations via self-play." Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019. --> This one leverages RL and a zoo of models to iteratively achieve more complex and diverse simulated agents.
I would like to see the revision of the paper including a discussion of these, and it would also be great to include them as baselines in the future.

Regarding the method, I believe it is sound and intuitive. While the local coordination is a simple extension of previous approaches, I believe the global coordination mechanism and its learning is original. I like the fact that the proposed method is decentralized at inference despite learning being coordination-aware, and that the local coordination factor is modeled as a continuous degree between cooperation and competitiveness. The write-up is also pretty clear overall. Nonetheless, I have a few questions and comments:
- In the preliminaries, it says that "the policy graddit methods are the appropriate solution", but that seems like a big claim without much backup.
- In Eq. 1, would the intuition that optimizing the individual advantage is hard given that we only observe a single action from the rest of the agents and thus the effect of the agent action cannot be well identified?
- The radial neighborhood is overly simplistic. I believe this is probably the reason why there are still critical failures happening in intersection (as seen in Fig. 6), since the conflicting vehicles come from different branches that are far away.
- I am confused about the statement in L168 about a shared and fixed LCF sinc in Section 4.2 it seems that the LCF is optimized per actor, which makes more sense to me.
- Why is $\phi$ uniformly sampled between [0, 90] (L167) instead of [-90, 90] as stated in L164?
- In Eq. 6, should it be $r_{i, t}^C$ instead of $r_{i, t}$? If not, why?
- In L185, it says that $\gamma$ is removed because the start steps of agents are not aligned. Couldn't the discount factor just go inside the sum over actors? Or do the authors mean it is implemented this way but simplified in the Eq. 6?
- How is the output parameterized, more concretely?

Finally, regarding the experiments, I believe they provide good insights into the difference between the baselines and the proposed method, although more baselines would have made the submission stronger. Aside from the scarcity of baselines, I have the following questions:
- Why are the new traffic environments required? By looking at Figure 1, it looks like a similar environment such as HighwayEnv could be sufficient.
- Since the networks are shared across all agents, I wonder how different goals are encoded for different agents. For instance, in Parking Lot, the agents can navigate toward external roads or enter parking lots. Would be great if the authors could share some details about how this is achieved.
- Why is a basic LiDAR simulated? Is there realistic background? If not, I do not really understand how this contributes to better experiments as opposed to just having access to past tracks of other agents.
- In Fig. 4, why does CoPO achieve less efficiency?
- I missed an experiment where we train a simple planner on the simulated agents for the proposed method and baselines, and evaluate in a common set of scenarios/agents. This would be similar to Table 3 in TrafficSim [A].
- A bigger concern is the claim about greater diversity in the abstract. I believe this is not backed up by any metric or qualitative results. If anything, Fig. 6 suggests that CoPO generates __less__ diverse traffic than the baselines as the lines of traffic flow are "thinner".
- In the video, it seems that most CoPO collisions in roundabout happen before entering it, as opposed to when merging as in most of the cases for the baselines. Any intuitions on why?

Minor comments:
- L101: even if well-known, include the full name of the Dec-POMDP abbreviature.
- In the footnote at the end of page 4, how is the meaningful half range enforced? Clipping?
- L240: sentence is oddly phrased. Perhaps replace "being terminated" for "are terminated".

PS. Figures 6 and 7 are great and the visualizations on the video too! Really enjoyed looking at the results, although would have appreciated narration on the video while highlighting particular cases. Please address as many comments as possible during the rebuttal.

==========

Post rebuttal comments:
- Thanks for the new experiments where IDM is used to control a percentage of the agents. The results are interesting and do show better robustness for CoPO than IPO. However, the results are still concerning in that the effectiveness and applicability of this method seems to be a bit limited to the coordinated agents setting, and the variance of the impact across scenarios seems high. Also, I wanted to make a comment that Tang et. al, does not just use IDM. IDM is used only as an initialization, then a policy is fully learned over multiple training iterations where the other agents policy get picked from an increasing model zoo.
- Thanks for clarifying about the goal encoding. Please include this information in the final revision.
- Overall, the responses and other reviewer comments were aligned with my expectations and thus I maintain my score.

limitations_and_societal_impact: I did not see a discussion about the limitations or societal impact. Please comment on the rebuttal and include this in later revisions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce a novel MARL learning algorithm, CoPO, that is designed to facilitate coordination in SDP systems. In addition to independent value functions, CoPO learns a neighborhood value function according to a return based on the normalized sum of neighboring agent rewards.  Both independent and neighborhood returns are mixed together according to a popular measure of social value orientation. Using a centralized bi-level optimization process, the agent social value distribution is optimised at the same time as the optimal low-level continuous decentralized control policies. The authors evaluate CoPO on a testbed of 5 common road traffic coordination tasks and according to three novel collective metrics. CoPO is shown to outperform other state-of-the-art CTDE MARL algorithms.

main_review: The authors' choice of topic fills a gap in MARL literature. While prior work on social value orientation in autonomous driving [e.g. Schwarting 2019] seeks to estimate the participants' social values from observed trajectories, CoPO seems to be the first work that tries to learn optimal social value orientations end-to-end in a CTDE MARL setting. I believe that this represents an important change of perspective: While prior work has focused on social value orientations being inherent properties of human drivers that cannot be influenced by autonomous drivers but have to be anticipated and respected, CoPO is based on the idea that incorporating social value functions in fact yields an effective inductive bias that allows end-to-end MARL methods to effectively solve certain general-sum games (also known as non-zero-sum or mixed cooperative-competitive) for purely autonomous drivers as well. Unlike human drivers, autonomous drivers can tune their "social" value distributions. The fact that this results in such a significant impact on task completion across a range of common traffic coordination problems is, in my view, a surprising insight and validates the original approach of this work.

The submission is written clearly. The bi-level optimization method CoOP is clearly described and well-motivated. Both evaluation environments and metrics used are well described, and experimental results are presented clearly and indicate significant outperformance.

Overall, I recommend this work for acceptance.
limitations_and_societal_impact: It would be great if the authors could connect their findings concerning the suboptimal performance of centralized value functions over decentralized ones in the light of recent developments in the field [Lyu et al 2021].

I would be interested to see what happens if we put a multimodal prior on the Phi-proposal - do we observe optimal multi-modal distributions for agent social value orientations? 

Another interesting line of research would be to study mixed autonomous-human traffic. For this purpose, one might want to consider dynamically varying Phis depending on what social value orientation is detected for a given traffic neighborhood. 

The authors would do well to check grammar and spelling in isolated instances.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors aim to design controllers for traffic simulation agents in the mixed-motive reinforcement learning setting. They introduce a smooth interpolation between individual and mean global reward (social value orientation) which agents locally optimize while using a meta-optimization process to optimize the degree of interpolation. They show that in a suite of 5 traffic scenarios, their method generally produces agent populations that succeed at higher rates, are more efficient, and produce a smaller amount of critical failures.
main_review: Overall, I enjoyed reading this paper. However, I think it could improve its positioning relative to existing work as well as compare to more related baselines. For these reasons I would suggest rejecting the paper for this conference, but I would look forward to seeing an improved version in a future venue!

In the abstract the authors write "However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the agents vary to be cooperative or competitive even within one episode." — However, in the introduction and related work the do note some mixed-motive RL works which study exactly this problem. Furthermore, I think the authors missed some key and very related works in their literature review for example "Social Diversity and Social Preferences
in Mixed-Motive Reinforcement Learning" McKee et. al also experiments with social value orientation explicitly. Furthermore, recent works such as Baker et. al 2020, Hughes et. al 2018, Jaques et. al 2019 all work in mixed motive RL in temporally extended games. For these reasons, I would suggest the authors reframe their claims to be more focused on traffic simulations as opposed to general mixed-motive RL. I would also suggest they experiment with some of these prior methods as baselines since they are designed for the same setting.

If I understand correctly, equation 6 is the mean reward of all agents in the episode. If the meta-optimization process is optimizing this value, doesn't that contradict one of the goals of the method which is to not have individuals sacrifice themselves for the good of the whole? I ask because Phi is allowed to go up to 90 (altruistic) so it seems that this would optimize the global reward best. Are there suboptimal local minima the agents fall into when optimizing with phi=90? I think it would be interesting to see Figure 3 but also plotted with equation 6 (global mean reward) as the y-axis in order to understand this better.

Is it possible that there is a different local reward function that would optimize the global efficiency, success rate, and critical failure rate? Is this reward function intractable?

Are there any links to existing game theory literature? It may be a nice addition to this paper to add experiments to smaller more well understood domains to more easily see what types of equilibria it induces (e.g. iterated prisoners dilemma, the coins game, etc).

In the introduction you mention that that your work is unique because it operates in the continuous control space. However, there are many high fidelity robotics works that actually use a discretized action spaces (e.g. see the robotic hand work from OpenAI). Maybe you should consider discretizing the action space and running this baseline if you'd like to make this claim stronger? Or I'd suggest dropping the language about continuous action spaces since I don't think it's a very important part of your work here and maybe distracts from the main point about coordination and social value orientation.

I would be interested to see how results change with varying d_n.

Concatenation is 1 parameterization of a global centralized value function. There are many others (e.g. attention, mean, max, etc). I notice you say that your centralized critic performed worse (L209). Concatenation is not entity invariant, but attention/mean/max are. I would suggest trying these instead.
limitations_and_societal_impact: There was limited to no discussion of limitations and societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors propose a multi-agent reinforcement learning approach to finding controllers in self-driven particles systems. The problem of SDP is formulated as dec-POMDP where each agent has an individual reward function that is wishes to maximize using local observations only.

The algorithm consists of two components: local coordination relying on neighboring rewards and global coordination. 

The local coordination part modifies the objective to be a combination of individual rewards and neighbor reward using a Local Coordination Factor. The LCF is used to compute a coordinated objective. 

The global coordination component is responsible to find the best LCF, for tractability this objective is factorized into individual global objective.

The algorithm is evaluated on a set of traffic scenarios against multiple baselines: independent policies, mean field methods and curriculum learning. 

The reward model using LCF and the split between local and global coordination is sound and interesting. However, the derivation lack of clarity, and it would have been useful to study the algorithm on other domains, especially given that the claim is on general SDP systems. 

The authors should revise the claim formulation and provide more justification of the global coordination method.

main_review: The paper is clear and well written and the topic is interesting.

The experiments are focused on traffic scenarios which weakens a bit the claim of the paper. The challenges addressed by the author are valid, it is in practice very difficult to hand-engineer controllers for those systems.  The utility of the solution found by the algorithm needs to be better justified, can we really draw conclusions from the output of this algorithm? 

The authors propose a general claim in the introduction, however at the core of the algorithm lies the computation of a neighbor set for each agent. In the traffic domain, neighbors are computed based on distances, how would one compute neighbors in other domain, how is this heuristics affecting the results?

Figure 3: how is success rate defined? It is not clear if the agents are sharing the same LCF, in practice it seems like every individual would have different LCF.

It is understandable that the authors need to factorize the global objective for tractability. However, it is not very clear what the individual global objective represents (the term is contradicting itself), the authors should provide some intuition. 

In the proof there seem to be a hidden assumption, a priori, the expectation over trajectories depends on all the agents. Why would the term inside the sum be only depending on \theta_i? This step needs to be clarified.

The experiment show that the proposed algorithm outperforms the baselines on multiple objective on most of the traffic scenarios.

One of the main usage of this training procedure seems to generate a variety of behavior to perform analysis. The interpretation of the generated behavior is very subjective, the author should comment on how to quantitively assess the quality of those behaviors. 

In the evaluation procedure, it would be useful to evaluate the generalization of the policy in terms of interaction. What if part of the agent in the environment follows a different policy trained by another algorithm or following a heuristic?

Figure 3 from the appendix should make it in the main paper. 

limitations_and_societal_impact: The conclusions of this paper are limited to the traffic domain. It does have societal impact as such solution could help the development of intelligent transportation systems which would have wide impact on our society.
review_title: null

Meta-review: The authors propose a method that uses bi-level optimization to overcome social dilemmas amongst large groups of self-interested agents. Specifically, each agent considers a weighted sum of its own reward and the reward of its neighbours. The weighting term, phi, trading off between these two is optimized using meta-gradients of the global objective. 

Importantly, by assuming that the method can undertake reward-shaping for the individual agents, the actual social dilemma is no longer relevant and the problem can be considered fully cooperative. 


There is a clear divide between the reviewers: While 3 of the reviewers provided very positive reviews, one reviewer remained critical. While the authors addressed the concerns and provided additional (favorable) experimental comparison at short notice, this did not change the score.


Given the positive reviews, the discussion and the overall contributions of the paper, I recommend this paper to be accepted.

However, here are two requests: The problem is currently introduced as a Dec-POMDP. This is inaccurate since different agents have different reward functions. 

Furthermore, two necessary baselines need to be added: 1)  Comparison to a fully cooperative setting where the global reward (sum of all agents’ reward) is maximized. 2) Each agent maximizes the total local reward (i.e. neighbors’ and own reward). This is very close to phi=90 degrees but includes the agent’s own reward. 

Decision: Accept (Poster)