Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: 
This work proposes a MARL method called Socially-Attentive Policy Optimization (SAPO) for controlling autonomous vehicles to behave socially-compatible behaviors. 

Two key components are used: 

1. Self-attention module foot select most interactive participant for each AV
2. Social-aware reward shaping mechanism.

SAPO enables varying the social preferences of an agent within one episode. Evaluation results show that SAPO can improve training efficiency and also emerge socially-compliant behaviors.
strengths_and_weaknesses: 
### Strengths

* The method is concise and sound.
* The presentation is clear and smooth.

### Weaknesses

Generally, I can't find any major weakness that harms this work.

* Lack of baselines. (See Issues)
* Figure 2 can be simplified a little bit.


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: 
Generally, I like this work. The work addresses the social behaviors in traffic flow by first discovering the issues in current algorithms, namely the dynamic social preferences within one episode, and then proposing solutions accordingly.
The presentation is clear and the method is sound and reasonable. The evaluation on SMARTS simulator shows promising results.

I have two concerns. First is the lack of baselines (see Issues). The second is whether this work should be presented in CoRL since there are no real robot experiments nor discussion on the applicability of SAPO in real-world. Nevertheless, I admit that the social behaviors in multi-agent systems is indeed an important topic, since in future we may co-live with those robots. 

issues: 
In related work [Peng et al.], the authors use a driving simulator MetaDrive for investigating the detailed behaviors in traffic flow. This work use SMARTS benchmark where only limited number of vehicles coexists. Also, I would expect to see the comparison with the algorithm CoPO proposed in [Peng et. al.], since both use SVO measurement to fuse reward. For the completeness of this work, I would suggest to run CoPO in SMARTS or run SAPO in MetaDrive to fairly compare related methods.


[Peng et al.] Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization


---


I am a little worried about Figure 6. We see SP maintains fixed in ~75 degree and ~10 degree when AV-3. Are there exist hard limit of SP? How can the neural network predicted SP be so smooth within one episode?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: A method called Socially-Attentive Policy Optimization (SAPO) for individual, coordinated AV control in multi-agent driving environments, aiming to learn socially compliant behaviors between agents with different goals. Contributes two novel parts:
1. a self-attention module to select the most interactive participant (other AV) for each AV in the scene
2. a "social-aware" method to balance individual rewards (social coordination) as a balance between being purely egotistical (only ego agent reward matters) and purely altruistic (only other agent reward matters)
strengths_and_weaknesses: Strength:
 - Learning 5x faster in some cases (Fig 3b), and otherwise better performance in SMARTS simulation
 - Interpretability of "most interactive participant" for each vehicle. Also the output \psi having the semantic meaning to help interpret the level of egotistical-vs-altruistic for each vehicle.

Weaknesses:
 - Over-complicated solution that is self-limiting. Why select only one other vehicle, to model interaction with, why not all, like [30] was already doing. Lines 154-155 describe limiting something that could have been more general, into something that is now more complicated *and* more limited in terms of only considering *one* other vehicle.
 - Unclear how it would actually work on a real robotics system. Difficult to understand this without a real platform or real data (e.g. Waymo or Argo open datasets). While the SMARTS benchmark tests high level coordination, the robots are treated as "self driven particles" (line 27), which makes me wonder how appropriate this work is for a *robotics* conference.
 - Initially difficult to understand the method: much abstract language initially discussing "social coordination", defining these terms unambiguiously too late. I think for a technical paper like this: get to the point faster. There's many terms like this including "social preferences" and "socially-compatible behaviors". The language of "dynamic social preferences" can mean many things, but seems a rather grandiose way to describe what's really happening, which is a weighted sum to two rewards that are only a function of the current observations (ignoring history).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'd choose "borderline" if CoRL had such a recommendation, but without it: Weak Reject. See Weaknesses and Issues sections.

----

update: I thank the authors for their responses. Apologies I did not get to this sooner. But your responses have addressed most of my concerns. I have thus updated my score to accept
issues: ## Larger points:
 - section 2: what about other works like Multiple Futures Prediction and Trajectron? Multi-agent autoregressive method with categorical latent variables can encode different types of behaviors or preferences too, and are (in addition) functions of the full history of past behaviors, while still reacting to latest observation (compared to this work which only looks at latest observation).
 - equation 6c, 6d: should these be function of p?
 - line 171: "unreasonable behaviors such as sacrificing oneself to improve group reward", why is this bad? Isn't this just \phi=\pi/4 in equation 9? Especially when lines 245-246 say that summed group reward is the evaluation criteria anyway.
 - section 5.1: while SAPO trains 5x faster than IPPO, how do we know this isn't just poor hyperparameter choice for IPPO? It asymtotes the same. Also, what are the uncertainty regions on Figure 3, what do they represent?
- lines 183-184: mention "corresponding to the latest traffic conditions" but this seems to be overselling: it's *only* a function of the current observations. There is no history...
 - line 211: "(a) removing Interactive Attention": a more interesting ablation here would have been *random* Interactive Attention. This would isolate-away the selection of p, and test how good the socially-aware part of the method would be. Possible an indication of how "toy" these experiments are would be: is the performance just the same? I imagine it could be, but for any sufficiently complicated setup  (perhaps what is shown in figure 6, or some more complicated traffic simulator) the ablation would show just how necessary it is to selection p intelligently.
- Why does each vehicle only select the most interactive other vehicle? there's no reason it can't focus on all vehicles, similar to [30], which inspired this work?
- Clarity: I think reading the abstract and introduction, it was difficult to understand what was actually being proposed. There were mentions of abstract benefits like "social preferences", "social-aware integration mechanism to integrate objectives of interacting AVs", and even "dynamic social preferences", and "socially-compatible behaviors", but it'd be good to get to the point faster. At line 54 it's still unclear what "socially-compatible behaviors" means, also on line 73 and 76, would help clarify to define this concretely earlier.

## Small points:
 - line 23: "inreasing" --> "increasing"
 - line 63" "MARL is widely used to navigate AVs in MASD systems" perhaps clarify this means the research litreaeture, not real AVs? Hardly any real AVs use RL, let alone MARL.
 - line 72: "response" --> "responses"
 - line 95: why is the policy space \mathcal{U} and not \Pi?
 - equation 3: doesn't the advantage function usually (1) not take a reward as input and (2) is a function of the *expected* reward as Q-V
 - equation 6a: "p \in \mathcal{M}" --> "p \in \mathcal{M} \ \{m\}"? I.e. I assume "m" cannot itself be its own closest/interactive neighbor?
 - line 159: "features" --> "generalization" (Wlog?)
 - Figure 2 (yellow box): it is shown how r^SP: goes between egotistic (\phi=0) and alteristic (\phi=\pi/2). But equation 8 mentions the range can go to -pi/2 radians too. Is this intended? I.e. ego acting "maliciously", against the other vehicles interests? If so, should it be shown in Figure 2 as well, or should the range of \phi in equation 8 be limited to [0, \pi/2]?
 - lines 174-176: different how? perhaps \emph{} which specific part following "[20]" that is different?
- lines 197: "goals in orders": which order? does one *particular* car need to go first?
- figure 6: maybe use white background, and zoom in more.
- line 231: who were vehicles AV-0 and AV-3 each selecting, was it respectively AV-2 and AV-1?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents SAPO, a multi-agent RL framework for autonomous driving. SAPO includes an attention module that tries to identify the most important "other" car in an interactive driving scenario and then compactly represent the features of this interaction. It then attempts to integrate these features in such a way to dynamically update the reward structure for the agent, notionally capturing social preferences and adaptation. The paper presents an ablative study, attempting to isolate the effects of several design choices. It then presents an evaluation across several scenarios, compared to 3 popular baseline models. The paper is largely well-written, and the idea is conceptually clever and very important. I have described several concerns below and hope they are instructive and helpful. 
strengths_and_weaknesses: Strengths
- The notion that most RL agents and settings use a static reward function is important; the ability to consider dynamic, evolving rewards is a compelling one. 
- The combination of social attention in a MARL setting is quite interesting, and the paper develops these notions well

Weaknesses
- Many of the evaluations are presented in terms of total accumulated reward with episodes along the x-axis. This suffers the same weakness (in my opinion) as almost all of the RL literature in that (as the very RL literature itself suggests) the reward function does not always indicate whether the task itself has been solved in a satisfactory manner.
- The limitations sections is somewhat superficial
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper is conceptually very elegant, and the application area is very important (furthermore, this idea could be applied in many other settings). Although I voice my concerns below, I feel that the paper warrants publication and the community would benefit from seeing the notions of attention and MARL combined in this type of way. 
issues: Line 9, abstract: "or" instead of "of" in cooperative or competitive fashion...

Line 27: "self-driven particles"? What is a particle here?

Line 39-40: "most of the interaction events occur in the red zone when AVs are crossing the intersection". While I agree that intersections include a lot of interactions, it is unclear that "most" of the interaction events occur here.

The authors do well to review the literature on social preferences in section 2.2. However, there is another, broader literature on social preferences and context in navigation that could (should) be explored here, for example how human pedestrians navigate in crowded settings. For example, Sekhon and Fleming. (2021, May). SCAN: A Spatial Context Attentive Network for Joint Multi-Agent Intent Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence. and many others from Alexandre Alahi et al.  The dynamics of pedestrians (or ships, etc) are obviously quite different, but the attention mechanisms are likely quite similar and this body of literature could be helpful. 

Line 92 we see "particles" again. Is this a common term? Does it imply that these AVs do not have kinematics or something?

In lines 126 onward the authors intentionally omit the timestep in the notation for simplicity. In my opinion, this makes the text and equations more difficult to interpret. 

What is the justification for equation (9)? Why sine and cosine of the social preference? Why addition? Is this tailored to intersections? What about other types of interactions (e.g. merging or passing in multi-lane traffic)? Would these types of interactions require different integrative rewards?

In general, I like the ideas and notions presented in section 4. The ability to focus one's attention and adapt rewards accordingly is both clever and important. However, I am a bit concerned about the lack of justification for the particular choices, and then a lack of alternatives and/or comparisons to state of the art in the ensuing sections. These design choices either need a theoretical justification or empirical evaluation, or better yet both. There are ablative studies, which is at least a start in the right direction. 

The ablation studies (and other evaluations) are presented in terms of total rewards. This is a fine place to start, but it assumes that the individual rewards are themselves representative of the desired task. What about (just making some metrics up here) collision rates? Average time to nagivate across an intersection? Or a normalized score comparing this travel time to perfect performance when no other vehicles are present? Etc etc etc. The problem with using rewards only is that we do not really know whether the RL agents have learned anything useful without any other context. Again, this is ok as part of a larger set of evaluations, but I do not see this.

I appreciate the comparison with several popular baselines, across a variety of scenarios. SAPO does well in these comparisons. In particular, it does well with the cross-road scenario, but is this due to the overall concept? Or due to particular design choices (I am thinking of equation (9) for example)? It seems like a general notion of attention and integration should be better across all interaction scenarios, and merging and ramps are perfectly valid instances of interactive scenarios but may be not-quite represented by the authors' design choices. 

Line 253: "This proves...". This does not really prove anything. It demonstrates that SAPO did better than a few baselines, across a few scenarios, in terms of overall reward. It is an *indication* of *something*, which is fine and quite enough to just say this. 

There are several limitations (and thus future research directions) that are not mentioned in section 7. More exploration of other attention and integration mechanisms is perhaps the most important, IMHO. Furthermore, this limitations section appears to be quite superficial.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The article presents SAPO: a modified version of the IPPO algorithm to solve AV interaction stated as a Multi-Agent Reinforcement Learning problem. The first modification is using social attention (Leurent and Mercat, 2019) to select *one* "most interactive" agent. The second modification is estimating a dynamic "social preference" (aka Social Value Orientation, Schwarting, et al., 2019) which combines rewards from ego and interactive agents to form a new reward for ego.
The article shows a favorable comparison of SAPO performance against a set of baselines, using the total of individual rewards as the metric.
strengths_and_weaknesses: Strengths:
1. The presented way of combining recently developed techniques (social attention, SVO, IPPO) seems to be novel, and potentially promising.
1. The set of baselines is large and diverse (perhaps not comprehensive though, see weakness 2).
1. The article presents in a clear way the methods it uses (IPPO, social attention).
1. Fig. 6. is a very nice qualitative presentation of a particular result.

Weaknesses:
1. In my opinion, the main problem of this work is that the motivation does not match with the proposed method and evaluation. The motivation is that AVs are becoming more popular and are sharing the space with other traffic participants (lines 23-24), so safe and smooth interactions with other agents are required (line 34). The solution is a Multi-Agent Self-Driving system, where all agents are trained in the same training loop. Even if such a multi-agent system achieves a high total reward (the metric used by authors), it does not guarantee that the agents would be compatible with human drivers or with different AVs.
    * The work uses simulator SMARTS, which supports "diverse behavior models of road users". Proving a good performance of agents trained with SAPO in interaction with SMARTS provided agents could address the issue to some extent.
1. The method makes a surprising assumption, that the "AV should pay attention to *the most interactive* vehicle", and the way it uses interaction attention is by selecting only one agent for the downstream processing. I think it is easy to imagine a situation, where the ego should take into account interaction with more than one agent.
     * Perhaps the point above is mitigated by including the baseline with "social attention". The method is however described with very few details. It does mention that it is based on "centralized control of multiple AVs". Maybe the lower performance is caused not by using full social attention (rather than just the most relevant), but rather due to centralized control (which is not always better).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In its current form, the paper does not have a good match between the motivation and the method, as well as its evaluation. Moreover, I don't see how this method could be used in a practical setup of an actual autonomous vehicle (as it assumes that other traffic participants are trained together with ego). This limits the applicability to robotics, which is the scope of the conference. While the work presents a new way of combining recently developed techniques (social attention, SVO, IPPO), the overall novelty is limited.
Unfortunately, in its current form, I recommend rejecting the article.

----

The additional experiments added during the discussion period (in particular on Argoverse and Waymo Dataset) have convinced me to increase my score to weak accept.
issues: 1. Addressing the Weakness 1. Perhaps the motivation can be changed: e.g. simulating self-driven particles for improving traffic simulation as in Peng et al., 2021; this would however require changing of metric as well. Alternatively demonstrating that the SAPO agents will be able to deal with a diverse set of other agents. There might be other possibilities as well.
    * It would be very useful to underline relevance to robotics, e.g. if you could show the perspective of integrating this into a real-world autonomous driving system.

2. Explain in more detail, how the method SAPO without Interactive Attention works. Without the Interactive Attention, there is no AV p, so whose reward is used in computing social-aware reward (Eq. 9)?

3. Provide more details on the social attention baseline. If possible, create another baseline, where social attention is used in a decentralized setup.
review_title: null

Decision: Accept (Poster)