Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a MARL approach for driving where each agent acts based on a combination of 'behavioral' incentives, which govern preferences over driving style, and 'instant' incentives, which govern the local preferences over avoiding collisions. Agents learn to predict the incentives of other agents while they learn to act. They evaluate in multi-agent driving domains and demonstrate performance improvements.
review: # Strengths

## Originality

The paper proposes an interesting idea: where agents in a heterogenous MARL environment learn opponent models. Given that there is lots of work on driver modeling for autonomous driving, this is a promising approach for their domain. I would be curious to see how this handles human driving behavior.

## Significance
The underlying idea has promise and so I think there is some upside for a high-impact paper.

## Quality

Although there is some room for improvement in presentation, the experiments seem to be a reasonable test of the method. 

# Weaknesses

## Clarity

[major] I had trouble following the overall presentation of the inference training method. I think that a more clear example of the method and edits to Algorithm 1 would help the reader follow and/or replicate the method. 

[minor] the typesetting in lines 171-182 is a bit strange due to the inline math.

[major] I had trouble understanding the practical difference between the behavioral and instant incentive. Maybe a better description of the method would help, or a more clear caption. E.g., consider rephrasing the sentence "The instance incentive inference... [uses data] .. to infer other vehicles instance incentives." It would be helpful to make it extra clear what an example of the instant incentives is. 

[major] there's not enough description of the different environments in the main paper, I recommend restructuring the paper so you can include a paragraph describing the environments in more detail
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I think the idea has promise, but I worry that the presentation/evaluation is not quite where is should be. If the authors can address the presentation issues, I think the paper should be accepted. I think a more clear demonstration on a more complex domain could also improve the paper's impact. 

Update after rebuttal/discussion: I appreciate the author’s efforts during the discussion to address my concerns about presentation and to better scope the claimed contributions. As a result, I am happy to raise my score and confidence level in the result.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper focuses on path planning for autonomous vehicles, by training agents (individual vehicles) to learn the driving styles (termed “incentives”) of neighboring vehicles, to both make short-term path predictions about other agents and finally reason about their own path decisions. The work is demonstrated in simulation on two different traffic scenarios (each with two levels of difficulty), where it is compared to three standard algorithms (QMIX, MAPPO, and independent PPO (iPPO)) as well as with a few ablation versions of the proposed approach.
review: This work considers a relevant problem for the community, and focuses on an interesting approach to tackle it: driving style estimation (i.e., essentially agent modelling) to enable and rely on predictions of the other agents’ short-term actions. I found the paper overall pleasant to read, and in particular appreciated the rigor (yet readability) of Section 3. That being said, I have several concerns with the actual premises of the work and with the results, which I believe call for more time to refine the overall story and the evaluation.

In the introduction, authors state that existing MARL algorithms can be classified as either centralized (e.g., QMIX or MAPPO) or distributed (e.g., iPPO). However, this does not seem to be what is generally accepted in the field, as far as I know. “Centralized” approaches like QMIX/MAPPO only rely on information centralization during training (e.g., mixing network, or centralized critic) but remain entirely decentralized at runtime. Instead, one generally refers to QMIX/MAPPO as “centralized training, decentralized execution (CTDE),” a paradigm that could be well-suited to many AV scenarios since training is often done in simulation (where centralization is not an issue), while offering the desired purely decentralized execution capabilities and scalability at deployment time. Therefore, I was first confused by this statement and premise.

Authors then follow up by stating that distributed approaches like iPPO often rely on tricks like parameters sharing, which cannot apply in AV path planning (but without substantiating why). However, the whole approach then described in this work, which consists of three modules (behavioral incentive inference, instant incentive inference, and controller/policy), has two of these modules completely symmetric and agent-agnostic, thus naturally allowing (and likely even benefitting) from parameter sharing! That is, while each vehicle’s policy network (controller) may justifiably want to assume independent weights for heterogeneity, the inference modules could and likely should share weights, since these modules all ask the same question, e.g., “based on what your neighbors did recently, how would you classify their intention?” for the first module. More to this point, one could even argue that the policy network (controller) could also use parameter sharing without drastically restricting performance/generality…
There again, I am confused by the overall premise of the work, which drastically weakens its contribution. It seems to me that the paper relies on “perceived”/”stated” requirements of the proposed solution (no “centralization” during training, no parameters sharing) that I am not convinced have reasons to be. This distracts the reader from appreciating the actual contribution of the work, i.e., the proposed planning framework, which does not really need/builds upon any of these requirements.

Results then compare the proposed approach and its ablation variants to standard baselines (MAPPO, QMIX, and iPPO), but does so in random training scenarios instead of freezing the models and running them through a rigorous, fixed set of randomized experiments. As a result, I am not convinced by the results, where many of the curves also seem to overlap and where algorithms were tested on potentially widely different scenarios. This is not the way algorithms are/should be compared in our field, and I believe a more rigorous testing phase is necessary.

Finally, I also have major concerns with the choice of design for the incentive inference modules, as learning to reconstruct/predict the full raw state seems like a rather flawed process: since each agent’s state also contains information about neighboring agents, learning to simply L1-reconstruct/predict such state may not allow agents to really focus on the most important/key portions of it. As a result, agents may often learn to grossly reconstruct/predict states, which may involve “small” yet very impactful mistakes, leading to very poor incentive learning and downstream policies. This is a common issue in many (single-agent) model-based learning methods that rely on similar encoder-decoder structure, and I believe that the fact that this paper considers a multi-agent problem only makes this issue more important. I also believe that this aspect of the work may be the one that, once properly addressed, could lead to the largest performance improvement.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While this paper considers a relevant problem, which it approaches along an interesting and novel axis, I worry that the current writing is still built upon shaky premises and inaccurate statements that weaken its story. Some of the key design considerations are also debatable and may merit to be more thoroughly explained and/or investigated for further performance improvement. Results are remain questionable, as no actual testing phase was carried out and algorithms were only tested via their trend at the end of the training phase, thus on potentially widely different scenarios. Performance differences also remain rather marginal. All in all, while I believe this work has a lot of potential, I also think that it needs more time to be polished, and I sadly cannot recommend it for publication at CORL 2023 in its present form.

EDIT: Authors have made significant improvements to the paper, and after rebuttal I believe it is now ready for publication.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper considers autonomous driving in a decentralized multi-agent setting where agents vary in their instant and behavioral incentives. The ego agent is unaware of these incentives but can estimate them from observations. A two-stream architecture for incentive inference is proposed. Experiments in two non-cooperative simplified autonomous driving domains compare to several baselines and ablations.

review: ## Strengths

* The problem setting is very clearly stated. I like the framing of the problem as a HiP-POSG. Overall, the formalisms are precise and the notation, while extensive, is helpful in my opinion.
* The problem setting is also incredibly difficult. Getting something to work at all with multiple agents and partial observability is a feat.
* The related work is very thorough and the paper seems well positioned in the literature.
* Besides a few details mentioned below, the figures are very helpful.


## Weaknesses

* 3 random seeds is not sufficient to draw meaningful conclusions.
* Relatedly, the empirical results provide only tentative support for the main hypothesis that decoupling behavioral and instant incentive inference is better than a unified inference scheme. In particular, the performance of iPlan versus IPPO-GAT does not seem convincingly better.
* Although a graph attention network is used to model instant interactions during instant incentive inference, my understanding is that each agent’s observation set is otherwise fixed in terms of the opponents that it is tracking. A more realistic setting would have the observation set change, and the model would need to be updated accordingly -- for example, the behavior incentive inference model may also need to be a graph neural network. I don’t think the model needs to be changed for this paper, but I do think this assumption should be clarified. I also think that this criticism of other works should be applied to this work as well: “Other works… require the involvement of the same group of agents between episodes, which is not practical in the real world.”
* In Figure 1 and part of the text, the behavior incentive inference model is said to “reconstruct” historical states. I think the use of the word “reconstruct” here is quite confusing. As far as I understand (from L203 in particular), the model is actually _predicting_ future states from earlier states. Alternatively, we could say that it is reconstructing _trajectories_, rather than states, which made me think that individual states are being auto-encoded (which wouldn’t make sense). I was especially confused in Figure 1 because it appears that single time steps are being reconstructed, since only $t$ appears in the orange box.
* The GIFs in the supplementary material are helpful to understand the environments tested, but they are also a bit overwhelming without context, especially given that multiple animations are concatenated together and each individually is quite busy. It would be very helpful to have the video draw the viewer’s attention to the important parts.
* I like the separation between behavioral incentives and instant incentives, but there is a third component of the hidden state -- vehicle type / kinematics -- and it’s not clear where that component comes into play. I wonder if a third “stream” would make sense. For example, incentives drift over time (fast and slow), but kinematic constraints are almost always constant. I also want to better understand where this component manifests in the current architecture. Is it one or both of the incentive streams?
* (I wrote the following, but then I realized that my concern was addressed in the experiments with the inclusion of randomly-behaving agents. I’m including the concern anyway in case the authors want to head it off earlier in the paper.) Other decentralized approaches that use parameter sharing or global rewards are criticized in the introduction because they “introduce strong and unrealistic assumptions for applications like autonomous driving.” I am not completely convinced that these mechanisms require stronger assumptions than the ones required here. This work assumes that all of the agents are being trained using the same MARL approach. This posits that all cars on the road are autonomous and that there is one autonomous driving company that determined the software on those cars. As far as I understand, parameter sharing and global rewards would only require the same assumption. Perhaps the approach in this work could be extended to a setting where not all agents are using the same MARL algorithm, but we would need to see experimental evidence for that (nevermind!).

### Minor
* Equation 1 uses $\mathcal{F}_i$ but the subsequent text uses $f_i$. Similarly there is some switching between $\mathcal{R}_i$ and $r_i$.
* There are a few awkward word choices in the paragraph L207-216: “temporal relation”, “blank prior”, “authentic estimation”. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper is very well written and carefully done. If the results were a bit stronger, particularly in terms of including more trials (random seeds), I would consider recommending strong accept.

### After Rebuttal

The original submission was solid, and the paper was improved substantially during the rebuttal period. I appreciate the engagement of the authors and the other reviewers.
review_title: null

Meta-review: This paper presents a multi-agent RL approach in the context of path planning of autonomous vehicles to learn behavior of neighboring vehicles. 

The reviewers have found that the proposed approach is interesting and technically sound. The reviewer concenrs on readability and contribution has been sufficiently resolved during the rebuttal phase.
Decision: Accept (Oral)