Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes recursive reasoning to model the opponents in a multi-agent environment, especially when the opponents are capable to learn and reason. The proposed method MBOM models the environment and the joint opponents. Particularly, MBOM simulates the recursive reasoning process and fine-tune the opponent models on multiple levels. The multiple-level models are combined by a Bayesian mixing strategy. The experiments are performed extensively and show improved performance on several benchmarks.
strengths_and_weaknesses: Strengths:
1. The paper aims to solve an interesting and important problem that how to model the reasoning and learning opponents in multi-agent tasks. The study can inspire the community potentially;
2. The paper is well presented and the experiments are extensive;

Weakness:
1. The proof of MBOM is not convincing and may present some problematic deductions;
2. The experiments cannot fully prove the effectiveness of the proposed method, especially it is hard to claim the MBOM has learned a more accurate model of the opponents even though the improved performance. Some more specific ablation study needs to be performed. 

questions: 1. Line 288-290 “MBOM w/o IOPs obtains similar results to MBOM when facing fixed policy opponents because /phi_0 could accurately predict the opponent behaviors if the opponent is fixed.” and similar claims that the model can accurately predict sth. It will be more convincing if the ablation study is directly performed on the prediction accuracy of the opponent model, e.g. using a controlled opponent, since the final performance of RL agents can be affected by many factors other than the opponent model accuracy;

2. Regarding the Theorem 2, how do you define the true probability of the opponent since the opponent's policy is changing based on its own reasoning? If correct, the proof of Theorem 2 can be also generalized to the regular reasoning learners, since given sufficient data, the regular model can also approximate the true distribution;

3. Line 556-576, the ablation study regarding the weights /alpha learned during the training is a bit confusing. In the task of Triangle Game, the scores of the task facing fixed policy opponents are the highest compared to the other two tasks facing naive learner and reasoning learner, however, in the ablation study it claims that (line 563-564) the models of level-0 IOP or level-1 IOP are not accurate;

4. Is it possible that MBOM has better performance because it uses a combination of several models. It would be more convincing if the baselines involve a corresponding ensemble of models;

5. As mentioned in the Future work, “MBOM implicitly assumes that the relationship between opponents is fully cooperative”, which is very tight and significantly limits the application of MBOM;

limitations: It would be more convincing if the ablation study on the model accuracy is directly performed. In addition, personally, I would like to see a baseline with the same number of ensemble models.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces an approach model-based opponent modeling (MBOM) for deep multiagent reinforcement learning (MARL) that combines (1) recursive imagination to estimate the reasoning of other agents at different hierarchical levels of reasoning (common in Theory of Mind reasoning like planning-based I-POMDPs, but novel in RL) and (2) Bayesian mixing that estimates the collective behaviors of all other agents in the system as a mix of the lower level recursive reasoning.  Such an approach enables the agent to learn how to behave when its oppponents follow static or adaptive policies without requiring explicit models of the neighbor's reasoning (including learning algorithm or learning gradients).  Theoretical analysis establishes error bounds on the value estimates.  Empirical evaluation on zero-sum games, competitive, and cooperative tasks from MPE demonstrate the advantages of the approach over appropriate baselines.

strengths_and_weaknesses: Overall, the research will be of interest to the popular MARL community at NeurIPS.  The approach is novel and well evaluated.  The paper is relatively easy to follow and has a thorough literature review.  I appreciated the inclusion of both theoretical and empirical analysis to evaluate the approach and derive key properties.  The supplement provided important information for reproducibility.
questions: 1) I didn't quite follow how \alpha_m = p(m | a^o).  Shouldn't \alpha_m (the mixing proportion) simply be p(m)?  Otherwise, shouldn't Eq. 6 also have a sum over all possible actions (if the policy is stochastic)?
limitations: The limitations of the approach are adequately described.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a model-based opponent modeling method to handle the interaction with other agents in multiagent systems. The main contribution of the proposed method is that it can adapt to different kinds of opponent policy, e.g., fixed-type, adaptive-type, and reasoning-type. Concretely, the proposed method first lets the agent interact with diverse opponents and collects the interaction experience. Then it uses the collected experience to train an environment model. Using this environment model, the agent imagines the adaption of the opponent policy and uses the imagined best response of the opponent to refine the opponent model. The agent repeats this process to get a set of opponent models reflecting different recursive reasoning levels. The authors further use Baysian mixing to get the final opponent model based on the opponent model set. As the Bayesian mixing is non-parametric, the final opponent model can quickly adapt to the true dynamic opponent policy.
strengths_and_weaknesses: Strengths: this paper tackles a challenging opponent modeling problem and the proposed method is a good combination of well-known techniques. The existing works are discussed in detail and this paper situates itself in the literature well. The presentation of this paper is also clear. The experiment results are sound and support the authors' claim.

Weaknesses: some technical parts need more description and explanation. Some existing works may also be worthy to compare.
questions: 1. In the proposed MBOM, the authors pre-train the agent’s policy using PPO while interacting with $v$ different opponents with diverse policies. However, I am surprised that the PPO can train a good agent policy in this way. In my opinion, the RL training highly relies on the sampled experience. The training process is trying to find a (local) optimal policy that can maximize the expected return given the sampled experience. Therefore, when trained with a certain type of opponent A, it should be easy for PPO to learn a good policy. However, if a different type of opponent B joins the training process, it will affect the interaction experience, i.e., in some episodes, the agent interacts with opponent A while in other episodes the agent interacts with opponent B. In this case, the PPO will adjust the policy trained with opponent A because it needs to consider the experience corresponding to opponent B. As a consequence, the new policy will likely have worse performance when facing opponent A especially when opponent A is very different from opponent B. Therefore, I would like to know more details about how to avoid this kind of issue and learn a good policy when interacting with $v$ different opponents with diverse policies. For example, what is the input of agent policy in every step during the training? I think the agent policy pre-training is critical because, during the recursive imagination, the agent policy is fixed and plays an important role.

2. Lemma 2 is critical for the theoretical analysis. However, is it a practical assumption? In what kind of scenarios this assumption will hold?

3. The authors use LOLA-DiCE as one baseline because it considers the potential learning of opponents. However, if the assumptions required by LOLA-DiCE do not hold, the consideration of opponent learning may bring adverse effects instead of benefits. This is also observed in Figure 2 where LOLA-DiCE has almost the worst performance in every case. Therefore, it may be better if the authors can also compare with other opponent modeling works, e.g. [27], [30]. Although the authors think they only consider opponent policies that are independent of the agent, sometimes simpler settings can be better than complex settings like in LOLA-DiCE.

4. MBOM has a pre-train phase where it sees a set of opponents with diverse policies. In comparison, Meta-PG only uses trajectories from the current opponents. Do the pre-train phase brings more information to MBOM and thus, gives it some advantages when interacting with the current opponents?
limitations: 1. The assumption in Lemma 3 seems strong. Although the authors claim that "larger M improves the representation capability of IOPs and thus better satisfies the assumption in Lemma 3", larger M also brings higher approximation error (which hinders the representation capability). Therefore, there is no guarantee that the assumption in lemma 3 can hold. If it does not hold, the Bayesian mixing may not approximate the true probability of the opponent well. Moreover, the recursive imagination will incur a considerable amount of computational cost.

2. There are duplicate texts on page 6: "However, larger M also has advantages. To analyze, we first define the benefit using the mixed IOP as..."
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work tackles the problem of modelling agent behaviour in multiagent systems. It focuses explicitly on modelling agents that, in their decision-making process, either (i) adapt their policy to other agents' exhibited behaviour or (ii) reason about other agents' behaviour. This work proposes MBOM, a model-based recursive reasoning-based approach to model agents that reason about others' behaviour. This approach is combined with a method that mixes inferred agent policies from different levels of recursive reasoning to (a) improve the model accuracy for predicting agents' behaviour while also (b) helping model learning agents that adapt their policies to other agents' behaviour.

This work evaluates MBOM in the Triangle  Game, One-on-One, and Predator-Prey environment against two types of baseline agent modelling techniques. The first type of baselines consists of approaches designed for modelling adaptive agents that change their policies according to others' behaviour. The second baselines are ablations of MBOM designed to elucidate the importance of MBOM's main components. When a controlled agent uses MBOM and the baseline agent modelling techniques for decision making, the experiments show agents equipped with MBOM achieving higher returns than agents that use the baseline agent modelling techniques.
strengths_and_weaknesses: Strengths

To my knowledge, MBOM's deep model-based approach to learning an agent's optimal policy at each level of reasoning is novel. Furthermore, its approach of mixing agent policies inferred at various levels of reasoning to improve modelling accuracy is also novel. By contrast, other methods such as PR2 (Wen et al., 2019) or GR2 (Wen et al., 2019) only consider the policy resulting from the deepest level of reasoning for decision making. 

The proposed ablation study over MBOM's component is designed well. The baseline selection elucidates the importance of MBOM's (i) recursive reasoning process and (ii) its usage of model mixing. In future iterations of this work, this ablation study should remain a part of the paper.

Except for the lack of descriptions regarding MBOM's model architecture in Figure 1, the model description is clear. The author's effort to explain the role of each of MBOM's components makes it easier to understand the model.


Weaknesses

This work lacks citations to older works in recursive reasoning for opponent modelling. In particular, works based on the I-POMDP framework (Gmytrasiewicz et al., 2005) should also be relevant to this work. Furthermore, an approach that applies deep neural networks to solve I-POMDPs has been explored by Han et al. (2019). In the case of the I-POMDP-Net proposed by Han et al. (2019), their approach does not rely on the CTDE learning paradigm, which the authors used to characterise prior deep learning-based works on recursive reasoning.

Major weaknesses:
1. The uncertain role of policy mixing in modeling adaptive learning agents
Despite positioning MBOM as an approach for modelling (i) adaptive learning agents and (ii) agents that also learn models of the controlled agent for decision-making, it is not clear which components of MBOM contribute toward modelling adaptive learning agents. While the authors attributed MBOM's strong performance when dealing with (i) to its policy mixing method in various parts of the manuscript (lines 195 and 279), the intuition behind why policy mixing helps in modelling (i) is not clear. More specifically, the non-parametric design of the policy mixer does not seem to help in learning the changes in modelled agent's policy resulting from the learning agent's actions.

My scepticism regarding the policy mixer's role in modelling (i) is further illustrated in the ablation study, which results are displayed in Figures 3c and 3d. Notice that there are ablations of MBOM that deliver similar performance to MBOM even without policy mixing (MBOM-{\phi_{0}}, MBOM-{\phi_{1}}, and MBOM-{\phi_{2}}) as long as the recursive reasoning process is done to an appropriate level. Comparing this to results in Figures 3a and 3b where we have an ablation of MBOM without recursive reasoning, the most significant drop in performance when dealing with adaptive learning agents results from not applying any recursive reasoning. Thus, this indicates that recursive reasoning is why MBOM models (i) well. The manuscript lacks further explanations why recursive reasoning results in improved modelling performance against (i) despite not modelling the changes in agents' policies resulting from learning.

2. Baseline selection and implementation
The baseline selection and implementation are also highly questionable. In particular, the authors selected baselines designed for modelling adaptive learning agents. Yet against adaptive learning agents, these baselines performed significantly worse than PPO and the proposed approach, which is not equipped with anything to model adaptive learning agents. This raises the question of whether these baselines are correctly implemented in the first place. 

To demonstrate the need to model adaptive agents (agents that learn) as opposed to modeling just fixed agents, it would be helpful to include agent modeling baselines which were not specifically designed to model learning agents, such as LIAM:
Georgios Papoudakis, Filippos Christianos, Stefano V. Albrecht. Agent Modelling under Partial Observability for Deep Reinforcement Learning. NeurIPS 2021

Although previous works model (i) with recursive reasoning, there is a lack of recursive reasoning baselines. While the authors mentioned other recursive reasoning methods' reliance on the CTDE paradigm as justification for not comparing against them (line 89), I highly believe a comparison against these methods must be made. Since the learning agent decides its action without any centralised component during execution, we can see this as utilising privileged information that only exists during training. Even MBOM uses some privileged information that is not ordinarily accessible during execution (e.g. modelled agent's rewards) for training. In the worst case, recursive reasoning methods not based on CTDE training like I-POMDP-Net can be used as a baseline.

3. Robustness to diverse opponents
While one of the central claims of this work is that MBOM allows the learning agent to perform well against a diverse set of opponents, MBOM has not been evaluated against agents whose reward function is highly different to those encountered during training. This is particularly important since many applications require learning agents to deal with previously unseen decision-makers whose reward functions are unknown (e.g. humans in autonomous driving scenarios). In this case, an experiment to evaluate MBOM's capability in those scenarios is also necessary.

4. Lack of experiments against agents with deeper levels of recursive reasoning.
While the reasoning learner is an example of agents that model the controlled agent, there is a lack of evaluation against agents with deeper levels of recursive reasoning. A potential improvement is to evaluate MBOM against another MBOM agent with various depths of reasoning.

5. A lack of analysis on model accuracy
The current work's analysis only reports the returns resulting from applying MBOM and the baselines. Nevertheless, it is important to report the model accuracy in any work in opponent modelling. Future iterations of this work can measure the log-likelihood of modelled agents' predicted actions.

6. Scalability to scenarios with more opponents.
It would be interesting to see whether MBOM scales to scenarios with more modelled agents. In particular, the rollout procedure done at every level of recursive reasoning requires more rollouts as the joint action space increases. Yet, the experiments have been limited to scenarios with small number of agents.

Clarity
1. Imprecise statements.
Overall, there are a few statements that are imprecise in the manuscript. Since these may potentially confuse readers, I recommend fixing these highlighted sentences:

(Line 8) "All kinds of opponents" --> Be precise on the type of agents that are evaluated in this work's experiments. 
(Line 20-21) "interacting with diverse opponents makes the environment nonstationary from the agent's perspective." --> This is only true when other agents' policies are changing. When agents' policies are fixed, one can account for the effect of agents' actions in the transition function. Even when agents' policies are unknown, the problem can be seen as a POMDP as long as these policies remain fixed.

2. Lack of details in Figure 1
If done correctly, Figure 1 can help readers understand the proposed approach. However, the lack of captions in Figure 1 explaining the components of MBOM makes it challenging to understand the Figure. Also, consider adding labels associated to the dashed boxes to indicate which components of MBOM they represent.

3. Justification on baseline and opponent design
The experiment section can be improved by highlighting why specific baselines or opponents are designed the way they are. Focusing on the insights gained from comparisons against specific baselines or experiments using certain opponents can also help highlight the claims provided in this work.


Significance

While this work presents an interesting approach that has potentially major significance for people working in MARL and opponent modelling, further experiments and comparisons are required to fully demonstrate its use in modelling (i) adaptive learning agents and (ii) agents that also learn models of the learning agent. Its limited comparison against prior recursive reasoning methods and the lack of analysis in terms of model accuracy particularly stands out as why this work has limited significance as is.
questions: 1) Why are the baselines designed for modelling adaptive learning agents even worse than PPO when evaluated against learning agents?
2) What is the intuition why MBOM's policy mixing approach improves performance when dealing with adaptive learning agents?
3) How will MBOM perform against agents with unknown reward functions (i.e. those whose reward function highly differs from agents encountered during training)
4) How does MBOM's rollouts process cope with the exponential increase in joint action space size as the number of agents increases?
5) (Line 22) "Overall supremacy" --> Supremacy in terms of what? Maybe even be more specific on what needs to be optimised (e.g. model accuracy or returns).
6) How accurate is MBOM (and other baselines) when modelling the actions of the modelled agents?
7) How are the reasoning learner finetuned during interaction?
8) In Figure 4b), why are Meta-PG and Meta-MAPG represented by 1 line? 
limitations: As mentioned in the above points, the work has not provided sufficient analysis on the limitations of MBOM to (i) deal with agents with previously unseen reward functions and (ii) to environments with more number of agents. At the same time, the work is currently fairly constricted to generic MARL setting. Thus, I do not believe it requires any additional statements on its potential societal impact.
review_title: null

Meta-review: metareview: This paper tackles the problem of modeling agents that are simultaneously learning or are able to reason during interaction. The proposed approach employs an environment model to simulate the opponent's reasoning process. The initial reviews are split and main concern is that the results do not adequately demonstrate that the proposed approach actually models opponents better. I believe the added ablation study and baselines have adequately addressed the concerns. Two reviewers also support acceptance after discussion (the other two didn't respond). I believe the work tackles and important problem in MARL and would spur valuable discussion at the conference. Thus, I'm leaning towards acceptance.
confidence: Less certain
Decision: Accept