Model-Based Opponent Modeling
Authors: XiaoPeng Yu
Jiechuan Jiang
Wanpeng Zhang
Haobin Jiang
Zongqing Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-agent reinforcement learning
opponent modeling

Abstract: When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponents. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponents. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in a variety of tasks, respectively with different types of opponents, i.e., fixed policy, naive learner, and reasoning learner.

Model-Based Opponent Modeling
Xiaopeng Yu
Jiechuan Jiang
Wanpeng Zhang
Haobin Jiang Zongqing Lu
School of Computer Science, Peking University
Abstract
When one agent interacts with a multi-agent environment, challenging to deal with various opponents unseen before. Modeling the behaviors, goals, beliefs of opponents could help the agent adjust its policy adapt to different opponents. In addition, it is also important to consider opponents who are learning simultancously or capable reasoning. However, existing work usually tackles only one of the aforementioned types of opponents. In this paper, propose model-based opponent modeling (MBOM), which employs the environment model adapt all kinds of opponents. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in a variety of tasks, respectively with different types of opponents, i.e., fixed policy, naive learner, and reasoning learner.
Introduction
Reinforcement learning (RI. has made great progress in multi-agent competitive games, e.g., Al- phaGo [33]. OpenAl Five [26], and AlphaStar [39]. In multi-agent environments, an agent usually has to compete against or cooperate with diverse other agents (collectively termed as opponents whether collaborators or competitors) unseen before. Since the opponent policy influences the transi- tion dynamics experienced by the agent, interacting with diverse opponents makes the environment nonstationary from the agent's perspective. Due to the complexity and diversity in opponent policies, it is very challenging for the agent to retain overall supremacy.
Explicitly modeling the behaviors, goals, or beliefs of opponents [2]. rather than treating them as part of the environment, could help the agent adjust its policy to adapt to different opponents. Many studies rely on predicting the actions [11, 13, 10, 27] and goals [29, 30] of opponents during training. When facing diverse or unseen opponents, the agent policy conditions on such predictions or representations generated by corresponding modules. However, opponents may also have the same reasoning ability, e.g., an opponent who makes predictions about the agent's goal. In this scenario, higher-level reasoning and some other modeling techniques are required handle such sophisticated opponents [40, 43, 45). In addition, opponents may learn simultaneously. the modeling becomes unstable, and the fitted models with historical experiences lag behind. To enable the agent to continuously adapt to learning opponents, LOLA [8] takes into account the gradients of the opponent's learning for policy updates, Meta-PG PG [1] formulates continuous adaptation meta-lcarning problem, and Meta-MAPC [16] combines meta-lcarning with LOLA. However, LOLA requires knowing the learning algorithm of opponents, while Meta-PG and Meta-MAPG require all opponents use the same learning algorithm.
"Correspondence te zongqing luēpku. edu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unlike existing work, we do not make such assumptions and focus on enabling the agent to learn effectively by directly representing the policy improvement of opponents when interacting with them, even if they may be also capable of reasoning. Inspired by the intuition that humans could anticipate the future behaviors of opponents by simulating the interactions in the brain after knowing the rules and mechanics of the environment, in this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to predict and capture the policy improvement of opponents. By simulating the interactions in the environment model, we could obtain the best responses of opponents to the agent policy that is conditioned on the opponent model. Then, the opponent model can be finetuned using the simulated best responses to get higher-level opponent model. By recursively repeating the simulation and finetuning, MBOM imagines the learning and reasoning of opponents and generates set of opponent models with different levels, which could also be seen as recursive reasoning. However, since the learning and reasoning of opponents are unknown, a certain-level opponent model might erroneously estimate the opponent. To effectively and accurately represent the opponent policy, we further propose to mix the imagined opponent policies according to the similarity with the real behaviors of opponents updated by the Bayesian rule.
We empirically evaluate MBOM in a variety of competitive tasks, against three types of opponents, i.e., fixed policy, naïve learner, and reasoning learner. MBOM outperforms strong baselines, especially when against naive learner and reasoning learner. Ablation studies verify the effectiveness of recursive imagination and Bayesian mixing. We also show that MBOM can be applied in cooperative tasks.
2
Related Work
Opponent Modeling. In multi-agent reinforcement learning (MARL), it is a big challenge to form robust policies due to the unknown opponent policy. From the perspective of an agent, if opponents are considered as a part of the environment, the environment is unstable and complex for policy learning when the policies of opponents are also changing. If the information about opponents is included, e.g., behaviors, goals, and beliefs, the environment may become stable, and the agent could learn using single-agent RL methods. This line of research is opponent modeling.
One simple idea of opponent modeling is to build model each time new opponent group of opponents is encountered [48]. However, learning a model every time is not efficient. A more computationally tractable approach is represent an opponent's 's policy with embedding vector. Grover et al. [10] uses a neural network as an encoder, taking as input the trajectory of one agent. Imitation learning and contrastive learning used to train the encoder. Then, the learned encoder can be combined with RL by feeding the generated representation into policy or/and value networks. Learning of the model can also be performed simultaneously with RL, as an auxiliary task [14). Based on DQN [24], DRON [11] and DPIQN [13] use a secondary network that takes observations as input and predicts opponents' actions. The hidden layer of this network is used by the DQN module to condition on for a better policy. It is also feasible to model opponents using variational auto-encoders [27], which means the generated representations are no longer deterministic vectors, but high-ddmenniioal distributions. ToMnet [29] tries to make agents have the same Theory of Mind [28] as humans. ToMnet consists of three networks, reasoning about the agent's ' action and goal based on past and current information. SOM [30] implements Theory of Mind from a different perspective. SOM uses its own policy, opponent's 's observation, and opponent's 's action to work backward to learn opponent's ' goal by gradient ascent.
The methods aforementioned only consider opponent policies that independent of the agent. If opponents hold a belief about the agent, the agent can form a higher-level belief about opponents' beliefs. This process can perform recursively, termed recursive reasoning. In repeated games, R2-B2 [5] performs recursive reasoning by assuming that all agents select actions according to GP-UCB acquisition function [34], and shows that recursive reasoning on one level higher than the other agents leads to faster regret convergence. When it comes to more complicated stochastic games, recursive reasoning should be compatible with reinforcement learning algorithms. PR2 [40] and GR2 [41] use the agent's 's joint Q-function to obtain recursive reasoning. Yuan et al. [45] takes both level-0 and level-1 beliefs as input to the value function, where level-0 belief is updated according to the Bayesian rule and level-1 belief is updated using a learnable neural network. However, these methods [40, 41, 45] use centralized training with decentralized execution algorithms to train a set of fixed agents that cannot handle diverse opponents in execution.

---

If the opponents are also learning, the modeling mentioned above becomes unstable and the fitted models with historical experiences lag behind. So it beneficial take into consideration the learning process of opponents. 1.OA [8] introduces the impact of the agent's policy on the anticipated parameter update of the opponent. estimate the learning gradient of the opponent's policy, implying that the learning algorithm used by the opponent should be known, otherwise the estimated gradient will be inaccurate. Further, the opponents may still be learning continuously during execution. Meta-PG [1] is method based on meta policy gradient, using trajectories from the current opponents to do multiple meta-gradient steps and construct policy that is good for the updated opponents. Meta-MAPG [16] extends this method by including an additional term that accounts for the impact of the agent's current policy on the future policies of opponents, similar to LOLA. These meta-learning based methods require that the distribution of trajectories matches across training and test, which implicitly means all opponents use the same learning algorithm.
neural network used to model the opponent's s policy and
Unlike existing work, we go one step further and consider a more general setting, where the opponents could be fixed, randomly sampled from an unknowable policy set, or continuously learning using an unknowable and changeable RL algorithm, both in training and execution. We learn an environment model that allows the agent to perform recursive reasoning against opponents who may also have the same reasoning ability.
Model-Based RL and MARL. Model-based RL allows the agent to have access to the transition function. There are two typical branches of model-based RL approaches: background planning and decision-time planning. In background planning, the agent could use the learned model to generate additional experiences for assisting learning. For example, Dyna-style algorithms [35, 19, 23] perform policy optimization
simulated experiences, and model-angmented value expansion algorithms [7, 25, 3] use model-based rollouts to improve the update targets. decision-time planning [4], the agent could use the model to rollout the optimal action at given state by looking forward during execution, e.g. model predictive control. Recent studies have extended model-based methods to multi-agent settings for sample efficiency [46, 47], centralized training [42], and communication [17]. Unlike these studies, we exploit the environment model for opponent modeling.
3 Preliminaries
In general, we consider an n-agent stochastic game AS ,..., A", ",,,,,7 P. R", .R).)). where S is the state space, A'' is the action space of agent i E [l..... m². A I[ A' is the joint action space of agents, P : S x A x S - [o, 1] is transition function, R' S x X IR is the reward function of agent i, and y the discount factor. The policy of agent i is n¹, and the joint policy of other agents is (s) f x² (a'((s) [s), where
is the joint action except agent i. All agents interact with the environment simultaneously without communication. The historical trajectory is available, ie., for agent at timestep f, {su, ab. ap" ..... 8t-1, a/-1, az.1 is observable. The goal of the agent i is to maximize its expected cumulative discount rewards
E S SL41 (alsaaaaaa a'wws' a'>ar(()s)c.e ((1s))a, ((118))
Σ
af
For convenience, the learning agent treats all other agents as a joint opponent with the joint action a ~ (-1s) and reward ro. The action and reward of the learning agent are denoted as a ~ x (.1s)) and r, respectively.
4 Method
MBOM employs the environment model to predict and capture the learning of opponent policy. By simulating recursive reasoning via the environment model, MBOM imagines the learning and reasoning of the opponent and generates set of opponent models. To obtain a stronger representation ability and accurately capture the adaptation of the opponent, MBOM mixes the imagined opponent policies according to the similarity with the real behaviors of the opponent.
3

---

(1..2%
(()))))))))
2u C (K000))
Bayesian Mivie Feuf f.
redi
KC
Beyoka
(5 11.00
Recursive Imagination
Figure Architecture of MBOM
4.1 Recursive Imagination
the opponent is also learning during the interaction, the opponent model fitted with historical experiences always lags behind, making the agent hard to adapt to the opponent. Moreover, if the opponent could adjust its policy according to the actions, intentions, or goals of the agent, then recursive reasoning may occur between agent and opponent. However, based on the lagged opponent model, the agent would struggle to keep up with the learning opponent. To adapt to the learning and reasoning opponent, the agent should predict the current opponent policy and reason more deeply than the opponent.
MBOM explicitly simulates the recursive reasoning process utilizing the environment model, called recursive imagination, to generate a series of opponent models, called Imagined Opponent Policies (IOPs). First, we pretrain the agent's 's policy using PPO [32] while interacting with different opponents with diverse policies that can be learned for example by [37]. and collects a buffer D which contains the experience (s, a, a", s', r), where and fully cooperative game (r = r"), 7 can be casily obtained, while for general-sum game we make assumption that r" can be accessed during experience collection. Then, using the experience buffer DD we can train the environment model [(s', rls, a, a²; () by minimizing the mean square error
(r, r²). For zero-sum game (r + r 0)
1 s'1 + 2 r -11
Ils'
(2)
3,0,0 P-D
where = aP; (), and obtain the level-0 IOP #g(-|s: Φn) by maximum likellhood estima- tion,
log (a dn)-
To imagine the learning of the opponent. illustrated in Figure I, the rollout algorithm [38] to get the best response of the opponent to the agent policy æ. For each opponent action of at timestep t, uniformly sample the opponent action sequences in the next i timesteps, simulate the trajectories using the learned environment model [ç. and select the best response with the highest rollout value
Pri
argmax of
UniE
During the simulation, the agent acts according to the policy conditioned the modeled opponent ; 0), and the learned environment model provides the transition 814-1, Pt = T [Ss, az, af; {). With larger k. the rollout has a longer planning horizon, and thus could evaluate the action a* more accurately, assuming a perfect environmental model. However, the computation cost of rollout increases exponentially with the planning horizon to get an accurate estimate of a²* while in practice the compounding error of the environmental model also increases with the planning horizon [15]. Therefore. the choice of k is tradeoff between accuracy and cost. Specifically, for zero-sum game and fully cooperative game, we can approximately estimate the opponent state value (s) as (s) and V(s), respectively, and modify the rollout value like n-step return [36] to obtain a longer horizon (see Appendix C for the empirical effect of k).
policy, ag ~ (-(19t, af ~ # (8;; Φo);
0 argmax
max
Prit + k+1 + (s
(5)
Unit

---

Algorithm MBOM
Pretraining:
2: Initialize the recursive imagination layer M.
3: Initialize the weights with (1/M, 1/M,... 1/M).
4: Interact with learning opponents to train the agent policy 9 and collect the experience buffer D.
Train the level-0 IOP Op, the environment model I'ç using experience buffer D.
6: Interaction:
7: f max_epoch do
8: Interact with the opponent to observe the real opponent actions.
\//Recursive Imagination)
Finetune énn with the opponent actions
10:
for T = 1...., M do
11:
Rollout in r with I [-[ss, â ~ ( 1s; (m-))):) get a"" by (4) (5).
12:
Finetune 6m-1 with best responses to get the 9m.
13:
end for
\/BBayesian Mixing)
14:
Update a by (8).
15:
Mix the IOPs to get amix by (6).
16:
end for
By imagination, we can obtain the best response of the opponent to the agent policy and construct the simulated data {(s, a)")). Then, we use the data finetune the level-0 IOP (-(sss (s; Φo) by maximum-likelihove
estimation, and obtain the level-1 IOP (-|s; ②1). The level-1 IOP can be seen as the best response of the opponent to the response of the agent conditioned on level-0 IOP. In the imagination, it is the nested form as "the opponent believes [that the agent believes (that the opponent believes ...)]." The existing imagined opponent policy the innermost "(that the opponent believes)," while the outermost "the opponent believes" is a"* which is obtained by the rollout process. Recursively repeating the rollout and finetuning, where the agent policy is conditioned on the IOP #2 (-Is; pm- could derive the level-m IOP # (-|s; Φm).
4.2 Bayesian Mixing
By recursive imagination, we get M IOPs with different reasoning levels. However, since the learning and reasoning of the opponent are unknown, single IOP might erroneously estimate the opponent. To obtain stronger representation capability and accurately capture the learning of the opponent, we linearly combine the IOPs to get mixed IOP,
M Σ m==
Tiix (-1s(11s)=
am (m),
(6)
where am the weight of level-m IOP, which taken as p(#o, la²), also denoted as p(m]a"), the probability that the opponent action a is generated from the level-m IOP. Thus, p(m]a²) indicates the similarity between the level-m IOP T. and the opponent policy nO in the most recent stage. By Bayesian rule, we have
p(a4|m)p(m) p(mla") M- ((a))))p())
#o (a 8; 6m (m(m
Σ (a dx)p((i)
Updating p(m]a") during interaction can obtain a more accurate estimate of the improving opponent policy.
In practice, we use the moving average of p(m a)) as the prior p(m), take the decayed moving average of pmm (a) as M... and obtain a by the IOPs mixer,
a (ao, a
softer-softmax((yy (V.... MI)-
(8)
Softer-softmax [12] is variant of softmax, which uses higher temperature to control softy of the probability distribution. The IOPs mixer is non- pon-maramttric, which could be updated quickly and efficiently without parameter training and many interactions (see Appendix B for empirical analysis on o.). Therefore, the IOPs mixer could adapt to the fast-improving opponent.
For completeness, the full procedure of MBOM in given in Algorithm 1.
5

---

4.3 Theoretical Analysis
MBOM consists of several components, including a conditional policy, an environment model, and M IOPs. They all interact with each other through model rollouts, which however makes it extremely hard give an error analysis based on each component of MBOM. Therefore, we instead focus mainly recursive imagination and Bayesian mixing, and give profound understanding of MBOM. Proofs are available in Appendix A.
Without loss of generality, we define om as the discrepancy between level-m and m-1 IOPs in terms of estimated value function given the IOP, and Em as the error of level-m IOP compared the true value function given the opponent policy, i.e,, 6m Vn Vn 1/1 6m IVm. - V7. and dg - 60-
Then, we have the following two lemmas.
Lemma 1. For the mixed imagined opponent policy (IOP) fii (-1s) E - Om Rm (-18; Prn), the
total enror ttota! = " M-1
Om satisfies the following inequality:
M-1
M-1
Etotal
a5.
Lemma 2. Suppose the value function is Lipschitz continuous on the state space S, K is the Lipschitz a, aa; {) #g [a]] 15 (be) is the transition distribution given and a, then constant, (s, = a(,,a,aa
V - Vj
K E
1-9 s
According to (6) and (7), we can express am as the following
To (a (s; 6m) p(m)) = Σ d(s, a²)
O' E [p(m)a"))
-1 Li= #H (a de) p(i)
where d(ss, a²) is the stationary distribution of pairs of state and action of opponent. Then, based on Lemma 1 and 2, we can obtain the following theorem, where M_1 denotes P(-Is, a,a")) x (a")ss) the true transition dynamics agent.
Theorem 1. Define the error between the approximated value function of the Bayesian mixing IOP and the true value function Etrue the following inequality:
V vl. For the Bayesian mixing IOP, the true error satisfies
M-1
1. - Mm
M-1 Σ क
Σ m=U s.axx
Σ yKd(s,0) 0")
Mm-
Etrue S Etotal S
M-1
8,09
Σ #3 (a%)s (a)):;
Theorem tells us the error accumulates as level M increases. However, larger M also has advantages. To analyze, we first define the benefit using the mixed IOP as the error bound of Theorem 1, is easy to see that the eиror accumulates level M increases, but at the same time, higher level also has advantages. To analyze, we need to define the benefit of using the mixed IOP as the optimization. We use the negative distribution distance 7 = - f π|| to denote the benefit of using policy in different levels, where # is the estimated opponent policy, æ is the true opponent policy and form of distribution distance function. For mixing policy, can define the benefit as:
|| is a general
aM = (g
+ C.M - TI.
Then, we have the following lemma and theorem.
Lemma 3, Assume that the true opponent policy T" has a probability distribution over the given set of 1OPs ( - and probability Pr(œ" # Pm for each #2 then the maximum expectation of TM can be achieved if and only if am Pm, Vm E [0, M 1].

---

Theorem 2. Given the action trajectory of opponent {az. dig.ai.... ...... H; the posterior probability updated by Bayesian mixing approximates the true probability of opponent as the length of the trujectory grows, i.e,
P(mla,, .M ai, 08) (true(m)= = Pm - 00.
Then the maximum expectation of л\ can be achieved with Om E[P(maa) ;, di)]
According to Theorem 2, we know that the estimated probability distribution of the opponent con- verges the true distribution of the opponent. It is obvious that larger M improves the representation capability IOPs and thus better satisfies the assumption Lemma 3, but also increases the error bound in Theorem 1. Therefore, the selection of M is tradeoff between them (see Appendix C for empirically study on M).
Experiments
We first evaluate MBOM thoroughly in two-player zero-sum tasks. Then, investigate MBOM when against multiple opponents and in a cooperative task. In all the experiments, the baselines trained for five runs with
have the same neural network architectures MBOM. All the methods different random seeds, and results are presented using mean and 95% confidence intervals. More details about experimental settings and hyperparameters available in Appendix E.
5.1 Two-Player Zero-Sum Tasks
We evaluate MBOM in two competitive tasks: (1) Triangle Game is an asymmetric zero-sum game implemented Multi-Agent Particle Environments (MPE) [21]. When facing different policies of the opponent, the agent has adjust its policy to adapt to the opponent for higher reward. (2) One-on-One is two-player competitive game implemented goalkeeper controlled by the agent could only passively react to the strategies of the shooter controlled by the opponent and makes policy adaptation when the shooter strategy changes.
Google Research Football [18]. The
Baselines. In the experiments, we compare MBOM with the following methods:
LOLA-DiCE [9] is an expansion of the LOLA, which uses Differentiable Monte-Carle Estimator (DiCE) operation to consider how to shape the learning dynamics other agents.
Meta-PG [1] uses trajectories from the current opponents to do multiple meta veta-ggaadeet steps and construct policy that is good for the updated opponents.
Meta-MAPG [16] includes an additional term that accounts for the impact of the agent's current policy on the future policies of opponents, compared with Meta-PG.
PPO [32] is classical single-agent RL algorithm, without any other modules.
Opponents. We construct three types of opponents:
Fixed policy. The opponents are pre-trained but not updated during interaction.
Naïve learner. The opponents are pre-trained and updated using PPO during interaction.
Reasoning learner. The opponents pre-trained and model the behavior of the agent. The model is finetuned during interaction and their policy is conditioned on the predicted action of the model.
The experimental results against test opponents in Triangle Game and One-on-One are shown in Figure 2, and the mean performance with standard deviation over all test opponents is summarized in Table 1. Without explicitly considering opponent policy, PPO achieves poor performance. The learning of LOLA-DiCE depends the gradient of the opponent model. However, the gradient information cannot clearly reflect the distinctions between diverse opponents, which leads that LOLA-DiCE cannot adapt the unseen opponent quickly and effectively. Meta-PG and Meta-MAN MAPG show mediocre performance in Triangle Game. In One-on-Ona since Meta-PG and Meta-MAPG heavily rely on the reward signal, adaptation is difficult for the two methods in this sparse reward task. MBOM outperforms Meta-MAPG and Meta-PG, because the opponent model of MBOM improves the ability to adapt to different opponents. And the performance gain becomes significant when the opponent naive learner reasoning learner, which is attributed
Performance.

---

oppeatiii
(a) Triangle Garne, versus lixed policy
(b) Triangle Gare,, versus maive leurner
(c) Triangle Game, versus reasoning learner
(d) One-on-One versus lixed policy
(e) One-on- versus naive learner
(1) One-on-One versus reasoning learner
LOLA-DiCE
Meta-PG
Meta-MAPG
PPO
MBOM
Figure 2: Performance against different types of opponents, i.e., fixed policy, naive learner, and reasoning learner, where x-axis is opponent index. The results show that MBOM outperforms other baselines, especially against naïve learner and reasoning learner.
the recursive reasoning capability brought by recursive imagination in the environment model and Bayesian mixing that quickly captures the learning of the opponent.
Table
Performance on Triangle Game and One-on-One
Triangle Game (score
One-on-One (win rate %)
Methods
Fixed Policy
Naive Leamer Reasoning Leamer
Fixed Policy
Naive Learner Reasoning Learner
LOLA-DiCE
22.51 (5.22)
-0.48 (4.02)
21..5 (4.43)
33.0
21.2(1.2)
(0.9)
Meta-PG
-3.78 (0.13)
-6.72
(1.87)
(1.7
(2.5
(0.0)
Mcta-MAPG
-2.01 (0.06)
(0.29)
-6.14 (0.84)
(1.5
36.6(1.)
31.1
PPO
13.29
(8.42
20.51 (1.80)
(0.7
21.7
25.8 (1.0)
MBOM
(0.03)
-1.66
(0.89)
(9.5)
51.3((.0)
64.2
Ablation Studies. The opponent modeling module of MBOM is composed of both recursive imagination and Bayesian mixing. In the following, we respectively test the functionality of these two components.
In recursive imagination, MBOM generates a sequence of different levels of IOPs finetuned by the imagined best response of the opponent the environment model. For comparison, we only do as the opponent model (i.e., without recursive imagination). denoted as MBOM w/o IOPs, and use random actions finetune the IOPs rather than using the best response, denoted as MBOM-BM. Note that MBOM w/o IOPs is essentially the same as the simple opponent modeling methods that predict opponents' actions, like [11, 13, 10]. The experimental results of MBOM, MBOM w/o IOPs, and MBOM-BM are shown in Figure 3(a) and 3(b). MBOM w/o IOPs obtains similar results to MBOM when facing fixed policy opponents because Φu could accurately predict the opponent behaviors if the opponent is fixed, which corroborates the empirical results in [11, 13, 10]. However, the opponent is learning or reasoning, Φu cannot accurately represent the opponent, thus the performance of MBOM w/o IOPs drops. The methods with IOPs perform better than MBOM w/o IOPs, which means that variety of IOPs can capture the policy changes of learning opponents. When facing the reasoning learner, MBOM outperforms MBOM-BM, indicating that the IOPs finetuned by the imagined best response have a stronger ability to represent the reasoning learner.
In Bayesian mixing, MBOM mixes the generated IOPs to obtain policy that is close to the real opponent. As a comparison, we directly use the individual level-m IOP generated by recursive imagination without mixing, denoted as MBOM- and use uniformly mixing instead of Bayesian mixing, denoted as MBOM-unif, The experimental results are illustrated in Figure 3(c) and ((d). Benefited from recursive imagination, MBOM-0, 1 and MBOM-0y show stronger performance than MBOM- MBOM consistently outperforms the ablation baselines without mixing when fighting against reasoning learners. The middling performance of MBOM-unif is due to not exploiting the more representative IOPs, indicating that Bayesian mixing could obtain a more accurate estimate of reasoning opponents.

---

00.0%
200r
1300
4389
Hatumry sahe
(a) Triangle Game
(b) One Fine-on-Op One
Triangle Game
(d) One on: One
MDOM wio IOPs
MDOM UM
MBOM
MBOM-O0 do
MBOM d1
MBOM-09
MBOM unif
MBOO
Figure 3: Ablation study MBOM. MBOM compared with MBOM-BM and MBOM w/o IOPs on recursive imagination in (a) Triangle Game and (b) Oneeon MBOM is compared with MBOM-00 MBOM-61, MBOM-02, and MBOM-unif on Bayesian mixing in (c) Triangle Game and (d) One-on-One.
LOSAO
Number Truuch
Meta-PG
PPO
LOLA- DICE
16:00 p0
PO ng hartr
150 120
215
(a) Predator Prey
(b) Learning curves on Coin Game
Figure 4: (a) Performance against different types of opponents in Predator-Prey, Prey, where smaller touch number means better performance. (b) Learning curves in Coin Game, where the numbers in the legend are the mean score with standard deviation over 200 iterations.
We additionally provide analysis the weight a and ablation studies on hyperparameters including the rollout length & and the level of recursive imagination M, which are available Appendix B and C, respectively.
5.2 Multiple Opponents
When facing multiple opponents, MBOM takes them a joint opponent, which is consistent with the single- angle-opponeet case. We perform experiments Predator-Pres Prey [21]. We control the prey as the agent and take the three predators opponents. In this task, the agent tries not be touched by the three opponents. When the policies of opponents are fixed (e.g. enveloping the agent), for the agent get high reward. However, if the policies of opponents change during interaction, there may be changes for the agent to escape. The results are shown in Figure 4(a). When facing learning opponents, MBOM achieves performance gain, especially competing with the reasoning learner, which indicates that MBOM adapts to the learning and reasoning of the opponents and responds effectively. Meta-MAPG and Meta-PG do not effectively adapt to the changes of opponents' policies and are prone to be touched by the opponents many times in a small area, resulting in poor performance. LOLA-DICE Meta-PG and Meta-MAPG underperforms PPO when against multiple reasoning learners, indicating their adaptation may induce a negative affect on the agent performance. Detailed experimental results with different opponents are available Appendix D.
difficult
5.3 Cooperative Task
MBOM could also be applied cooperative tasks. We test MBOM on a cooperative scenario. Coin Game, which is high-dimension expansion of the iterated prisoner dilemma with multi-step actions [20, 8). Both agents simultaneously update their policies using MBOM or the baseline methods to maximize the sum of rewards. The experiment results are shown in Figure 4(b). Meta-PG and Meta-MAPG degenerate to Policy Gradients for this task as there no training set. Both learn greedy strategy that collects any color coin, which leads to a zero total score of two players. LOLLA-DCCE learns too slow and does not learn to cooperate within 200 iterations, indicating the inefficiency of estimating the opponent gradients. PPO learns cooperate quickly and successfully,

---

which corroborates the good performance of independent PPO in cooperative multi-agent tasks as pointed out in [6, 44]. MBOM slightly outperforms PPO, which indicates that MBOM can also be applied to cooperative tasks without negative effects. Note that we do not compare other cooperative MARL methods like QMIX [31], as they require centralized training.
5.4 Opponent Model Accuracy Analysis
MBOM intends to achieve higher return by improving model accuracy via adapting to the changing opponent. In previous sections, have shown that MBOM achieves superior performance. In this section, we investigate the correlation between the performance and the prediction accuracy of the opponent model.
First, we tested the prediction error of the opponent model, using KL divergence to measure the distance between the predicted action distribution and the real action distribution of the opponent policy. The results compared with MBOM w/o IOPs (without recursive imagination) are shown in Table 2 (the column Error). Overall, MBOM's opponent model achieves more accurate prediction than MBOM w/o IOPs.
Then, we tested the performance of the same policy k[-[s, ū²;0) with different opponent models. The opponent model of MBOM w/o IOPs continuously updated by supervised learning using the observed opponent's action, which has lower prediction accuracy than MBOM's opponent model. MBOM-pro, which uses the true actions of the opponent as the input of the policy during the interaction phase instead of the actions predicted by IOPs. That is, it can be considered that the prediction accuracy of MBOM-pro's s opponent model 100%. With the increasing prediction accuracy of the opponent model, the performance also increases correspondingly. from MBOM w/o IOPs + MBOM > MBOM-pr Moro,,,,aaaa as shown in Table 2 (the column Performance). The results show that accurate opponent model could improve performance, and that MBOM indeed obtains more accurate opponent model when the opponent is learning.
Table 2: Prediction accuracy opponent model and performance in Triangle Game and One-on-One.
Fixed policy
Nuive learner
Reusoning leurner
Error 1
Performance
Error L
Perfornance
Error |
Performance
MBOM w/o IOPs MBOM
15.72 15.47 (0.04)
-1.14 (0.03)
15.17
-3.23 (0.07)
(0.09)
(2.00)
Triangle Game
-1.19 (0.03)
13.47 (0.43)
-1.66 (0:29)
7,81 (0.38)
-2.75
MBOM-pro
0.96 (0.03)
-0.11 (0.18)
0.82 (0.35)
MBOM w/o IOPs
5.62 (0.06)
(0.7
1.04
(8-8
7.36 (0.34)
55.7
Onc-on-One
MBOM
(0.8()
(0.59
(0.06)
61.((10)
(0.12)
(1.2
MBOM-pm
(9.6
52.0(1-7)
(7.4
6 Conclusion and Future Work
We have proposed MBOM, which employs recursive imagination and Bayesian mixing to predict and capture the learning and improvement of opponents. Empirically, evaluated MBOM in a variety of competitive tasks and demonstrated MBOM adapts to learning and reasoning opponents much better than the baselines. These make MBOM simple and effective RL method whether opponents be fixed, continuously learning, or reasoning in competitive environments. Moreover, MBOM can also be applied in cooperative tasks.
In MBOM, the learning agent treats all opponents as a joint opponent. the size of the joint opponent is large, the agent will need a lot of rollouts to get accurate best response. The cost increases dramatically with the size of the joint opponent. How to reduce the computation overhead in such scenarios will be considered in future work. Moreover, MBOM implicitly assumes that the relationship between opponents is fully cooperative. How to deal with the case where their relationship is non-cooperative is also left future work.
Acknowledgments and Disclosure of Funding
This work was supported part by NSF China under grant 61872009. The authors would like to thank the anonymous reviewers for their valuable comments.
10

---

References
[1] Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous Adaptation Via Meta-Learning in Nonstationary and Competitive Environ- ments. In International Conference on Learning Representations (ICLR), 2018.
[2] Stefano V Albrecht and Peter Stone. Autonomous Agents Modelling Other Agents: A Compre- hensive Survey and Open Problems. Artificial Intelligence, 258:66-95, 2018.
[3] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[4] Kurtland Chua, Roherto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in A Handful of Trials Using Probabilistic Dynamics Models. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[5] Zhongxiang Dai, Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Teck-Hua Ho. R2-b2: Recursive International Conference on Machine Learning (ICML), 2020.
bayesian optimization for no-regret learning in games, In
[6] Christian Schröder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. Torr, Mingfei Sun, and Shimon Whiteson. Is Independent Learning All You Need in the Starcraft Multi-Agent Challenge? arXiv preprint arXiv:201 1.09533, 2020.
[7] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-Basee Value Estimation for Efficient Model-Free Reinforcement Learning. arXiv preprint arXiv: 0888000.000.. 2018
[8] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Picter Abbeel, and Igor Mordatch. Learning with Opponent-Leamming Awareness. In International Conference Autonomous Agents and MultiAgent Systems (AAMAS). 2018.
[9] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DICE: The Infinitely Differentiable MOnte CArlo Estimator. In International Conference Machine Learning (ICML), 2018.
[10] Aditya Grover, Maruan Al-Shedivat, Jayesh K. Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning (ICML), 2018.
[11] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent Modeling in Deep Reinforcement Learning. In International Conference Machine Learning (ICMI), 2016.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling The Knowledge A Neural Network. arXiv preprint arXiv:1503.02553 2015.
[13] Zhang-Wer Wei Hong. Shih- Yang Su, Tzu-Yu Yun Shann, Yi-Hsiang Chang, and Chun-Y Yi Lee. A Deep Policy Inference Q-Network for Multi-Agent Systems. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
[14] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, and K. Kavukeuoglu. Reinforcement Learning with Unsupervised Auxiliary Tasks. In International Conference Learning Repre- sentations (ICLR), 2016.
[15] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model-Based Policy Optimization. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[16] Dong-Ki Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. A Policy Gradient Algorithm for Learning To Learn in Multiagent Reinforcement Learning. International Conference Machine learning proceedings (ICML), 2021.

---

[17] Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International Conference on Learning Representations (ICLR), 2021.
[18] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google Research Football: A Novel Reinforcement Learning Environment. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
[19] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model- Eneembe Trust-Region Policy Optimization. In International Conference Learning Representations (ICLR), 2018.
[20] Adam Lerer and Alexander Peysakhovich. Maintaining Cooperation in Complex Social Dilem- mas Using Deep Reinforcement Learning. arXiv preprint arXiv: 1770001700.01068, 2018.
[21] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-Agent Actor-Critic for Mixed Cooperativ-Cooppttiive Environments. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[22] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mic Framework for Moddll-Based Deep Reinforcement Learning with Theoretical Guarantees. In International Conference on Learning Representations (ICLR), 2019.
[23] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mic Framework for Model-Based Deep Reinforcement Learning with Theoretical Guarantees. In International Conference on Learning Representations (ICLR), 2019.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533) 2015.
[25] Junhyuk Oh, Satinder Singh, and Honglak Lee. Value Prediction Network. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[26] OpenAl. Openai five. https://blogg
wttp://aaooooopaaiiiiiiiaa 2018.
[27] Georgios Papoudakis and Stefano V Albrecht. Variational Autoencoders for Opponent Modeling in Multi-Agent Systems. arXiv preprint arXiv:2001. 10829, 2020.
[28] David Premack and Guy Woodruff. Does The Chimpanzee Have A Theory of Mind? Behavioral and brain sciences, 1(4):515-526, 1978.
[29] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine Theory of Mind. In International Conference on Machine Learning (ICML), 2018.
[30] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
[31] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi- agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint /t0..//777.6333 2017.
[33] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering The Game of Go with Deep Neural Networks and Tree Search, Nature, 529(7787): 484-489, 2016.
12

---

[34] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:09/2.3995, 2009.
[35] Richard S Sutton. Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming. In International Conference on Machine learning proceedings (ICML), 1990.
[36] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[37] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference on Learning Representations (ICLR), 2021.
[38] G. Tesauro and G. R. Galperin. On-Line Policy Improvement Using Monte-Carlo Search. In Advances Neural Information Processing Systems (NeurIPS), 1996.
[39] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi. Richard Powell, Timo Ewalds, Petko Georgiev, al. Grand- master Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575(7782): 350-354, 2019.
[40] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic Recursive Rea- soning for Multi-Agent Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2019.
[41] Ying Wen, Yaodong Yang, Rui Luo, and Jun Wang. Modelling bounded rationality in multi- agent interactions by generalized recursive reasoning. In International Joint Conference on Artificial Intelligence (ICCAI), 2020.
[42] Daniël Willemsen, Mario Coppola, and Guido CHE de Croon. Mambpo: Sample-efficient multi-robot reinforcement learning using learned world models. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
[43] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Chongjie Zhang, Yan Zheng, and Ze Zheng. To- wards efficient detection and optimal response against sophisticated opponents. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[44] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games. arXiv preprint arXiv:2103.01955, 2021.
[45] Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Zhu. Emergence of Pragmatics From Referential Game Between Theory Mind Agents. arXiv preprint arXiv:200..07752. 2020.
[46] Kaiqing Zhang, Sham M. Kakade, Tamer Basar, and Lin F. Yang. Model-based multi-agent RL in zero-sum markov games with near-optimal sample complexity. arXiv preprint arXiv:2007.07461 2020.
[47] Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou. Model-based multi-agent policy optimization with adaptive opponent-wise rollouts. arXiv preprint arXiv:2105.0366, 2021.
[48] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents. In Advances in Neural Information Processing Systems (NeurIPS), 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec
(c) Did you discuss any potential negative societal impacts of your work? [No] As we just propose reinforcement learning algorithm, we do not immediately see any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix E.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In Appendix E.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "u0i05YAqQEX", "reviewer": "Reviewer_yT2T", "summary": "This paper introduces an approach model-based opponent modeling (MBOM) for deep multiagent reinforcement learning (MARL) that combines (1) recursive imagination to estimate the reasoning of other agents at different hierarchical levels of reasoning (common in Theory of Mind reasoning like planning-based I-POMDPs, but novel in RL) and (2) Bayesian mixing that estimates the collective behaviors of all other agents in the system as a mix of the lower level recursive reasoning.  Such an approach enables the agent to learn how to behave when its oppponents follow static or adaptive policies without requiring explicit models of the neighbor's reasoning (including learning algorithm or learning gradients).  Theoretical analysis establishes error bounds on the value estimates.  Empirical evaluation on zero-sum games, competitive, and cooperative tasks from MPE demonstrate the advantages of the approach over appropriate baselines.\n", "questions": "1) I didn't quite follow how \\alpha_m = p(m | a^o).  Shouldn't \\alpha_m (the mixing proportion) simply be p(m)?  Otherwise, shouldn't Eq. 6 also have a sum over all possible actions (if the policy is stochastic)?", "limitations": "The limitations of the approach are adequately described.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the research will be of interest to the popular MARL community at NeurIPS.  The approach is novel and well evaluated.  The paper is relatively easy to follow and has a thorough literature review.  I appreciated the inclusion of both theoretical and empirical analysis to evaluate the approach and derive key properties.  The supplement provided important information for reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nSdF8sVO-nz", "reviewer": "Reviewer_PoB7", "summary": "This paper proposes recursive reasoning to model the opponents in a multi-agent environment, especially when the opponents are capable to learn and reason. The proposed method MBOM models the environment and the joint opponents. Particularly, MBOM simulates the recursive reasoning process and fine-tune the opponent models on multiple levels. The multiple-level models are combined by a Bayesian mixing strategy. The experiments are performed extensively and show improved performance on several benchmarks.", "questions": "1. Line 288-290 “MBOM w/o IOPs obtains similar results to MBOM when facing fixed policy opponents because /phi_0 could accurately predict the opponent behaviors if the opponent is fixed.” and similar claims that the model can accurately predict sth. It will be more convincing if the ablation study is directly performed on the prediction accuracy of the opponent model, e.g. using a controlled opponent, since the final performance of RL agents can be affected by many factors other than the opponent model accuracy;\n\n2. Regarding the Theorem 2, how do you define the true probability of the opponent since the opponent's policy is changing based on its own reasoning? If correct, the proof of Theorem 2 can be also generalized to the regular reasoning learners, since given sufficient data, the regular model can also approximate the true distribution;\n\n3. Line 556-576, the ablation study regarding the weights /alpha learned during the training is a bit confusing. In the task of Triangle Game, the scores of the task facing fixed policy opponents are the highest compared to the other two tasks facing naive learner and reasoning learner, however, in the ablation study it claims that (line 563-564) the models of level-0 IOP or level-1 IOP are not accurate;\n\n4. Is it possible that MBOM has better performance because it uses a combination of several models. It would be more convincing if the baselines involve a corresponding ensemble of models;\n\n5. As mentioned in the Future work, “MBOM implicitly assumes that the relationship between opponents is fully cooperative”, which is very tight and significantly limits the application of MBOM;\n", "limitations": "It would be more convincing if the ablation study on the model accuracy is directly performed. In addition, personally, I would like to see a baseline with the same number of ensemble models.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper aims to solve an interesting and important problem that how to model the reasoning and learning opponents in multi-agent tasks. The study can inspire the community potentially;\n2. The paper is well presented and the experiments are extensive;\n\nWeakness:\n1. The proof of MBOM is not convincing and may present some problematic deductions;\n2. The experiments cannot fully prove the effectiveness of the proposed method, especially it is hard to claim the MBOM has learned a more accurate model of the opponents even though the improved performance. Some more specific ablation study needs to be performed. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jv2ceA1SUXU", "reviewer": "Reviewer_Le9a", "summary": "This work tackles the problem of modelling agent behaviour in multiagent systems. It focuses explicitly on modelling agents that, in their decision-making process, either (i) adapt their policy to other agents' exhibited behaviour or (ii) reason about other agents' behaviour. This work proposes MBOM, a model-based recursive reasoning-based approach to model agents that reason about others' behaviour. This approach is combined with a method that mixes inferred agent policies from different levels of recursive reasoning to (a) improve the model accuracy for predicting agents' behaviour while also (b) helping model learning agents that adapt their policies to other agents' behaviour.\n\nThis work evaluates MBOM in the Triangle  Game, One-on-One, and Predator-Prey environment against two types of baseline agent modelling techniques. The first type of baselines consists of approaches designed for modelling adaptive agents that change their policies according to others' behaviour. The second baselines are ablations of MBOM designed to elucidate the importance of MBOM's main components. When a controlled agent uses MBOM and the baseline agent modelling techniques for decision making, the experiments show agents equipped with MBOM achieving higher returns than agents that use the baseline agent modelling techniques.", "questions": "1) Why are the baselines designed for modelling adaptive learning agents even worse than PPO when evaluated against learning agents?\n2) What is the intuition why MBOM's policy mixing approach improves performance when dealing with adaptive learning agents?\n3) How will MBOM perform against agents with unknown reward functions (i.e. those whose reward function highly differs from agents encountered during training)\n4) How does MBOM's rollouts process cope with the exponential increase in joint action space size as the number of agents increases?\n5) (Line 22) \"Overall supremacy\" --> Supremacy in terms of what? Maybe even be more specific on what needs to be optimised (e.g. model accuracy or returns).\n6) How accurate is MBOM (and other baselines) when modelling the actions of the modelled agents?\n7) How are the reasoning learner finetuned during interaction?\n8) In Figure 4b), why are Meta-PG and Meta-MAPG represented by 1 line? ", "limitations": "As mentioned in the above points, the work has not provided sufficient analysis on the limitations of MBOM to (i) deal with agents with previously unseen reward functions and (ii) to environments with more number of agents. At the same time, the work is currently fairly constricted to generic MARL setting. Thus, I do not believe it requires any additional statements on its potential societal impact.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nTo my knowledge, MBOM's deep model-based approach to learning an agent's optimal policy at each level of reasoning is novel. Furthermore, its approach of mixing agent policies inferred at various levels of reasoning to improve modelling accuracy is also novel. By contrast, other methods such as PR2 (Wen et al., 2019) or GR2 (Wen et al., 2019) only consider the policy resulting from the deepest level of reasoning for decision making. \n\nThe proposed ablation study over MBOM's component is designed well. The baseline selection elucidates the importance of MBOM's (i) recursive reasoning process and (ii) its usage of model mixing. In future iterations of this work, this ablation study should remain a part of the paper.\n\nExcept for the lack of descriptions regarding MBOM's model architecture in Figure 1, the model description is clear. The author's effort to explain the role of each of MBOM's components makes it easier to understand the model.\n\n\nWeaknesses\n\nThis work lacks citations to older works in recursive reasoning for opponent modelling. In particular, works based on the I-POMDP framework (Gmytrasiewicz et al., 2005) should also be relevant to this work. Furthermore, an approach that applies deep neural networks to solve I-POMDPs has been explored by Han et al. (2019). In the case of the I-POMDP-Net proposed by Han et al. (2019), their approach does not rely on the CTDE learning paradigm, which the authors used to characterise prior deep learning-based works on recursive reasoning.\n\nMajor weaknesses:\n1. The uncertain role of policy mixing in modeling adaptive learning agents\nDespite positioning MBOM as an approach for modelling (i) adaptive learning agents and (ii) agents that also learn models of the controlled agent for decision-making, it is not clear which components of MBOM contribute toward modelling adaptive learning agents. While the authors attributed MBOM's strong performance when dealing with (i) to its policy mixing method in various parts of the manuscript (lines 195 and 279), the intuition behind why policy mixing helps in modelling (i) is not clear. More specifically, the non-parametric design of the policy mixer does not seem to help in learning the changes in modelled agent's policy resulting from the learning agent's actions.\n\nMy scepticism regarding the policy mixer's role in modelling (i) is further illustrated in the ablation study, which results are displayed in Figures 3c and 3d. Notice that there are ablations of MBOM that deliver similar performance to MBOM even without policy mixing (MBOM-{\\phi_{0}}, MBOM-{\\phi_{1}}, and MBOM-{\\phi_{2}}) as long as the recursive reasoning process is done to an appropriate level. Comparing this to results in Figures 3a and 3b where we have an ablation of MBOM without recursive reasoning, the most significant drop in performance when dealing with adaptive learning agents results from not applying any recursive reasoning. Thus, this indicates that recursive reasoning is why MBOM models (i) well. The manuscript lacks further explanations why recursive reasoning results in improved modelling performance against (i) despite not modelling the changes in agents' policies resulting from learning.\n\n2. Baseline selection and implementation\nThe baseline selection and implementation are also highly questionable. In particular, the authors selected baselines designed for modelling adaptive learning agents. Yet against adaptive learning agents, these baselines performed significantly worse than PPO and the proposed approach, which is not equipped with anything to model adaptive learning agents. This raises the question of whether these baselines are correctly implemented in the first place. \n\nTo demonstrate the need to model adaptive agents (agents that learn) as opposed to modeling just fixed agents, it would be helpful to include agent modeling baselines which were not specifically designed to model learning agents, such as LIAM:\nGeorgios Papoudakis, Filippos Christianos, Stefano V. Albrecht. Agent Modelling under Partial Observability for Deep Reinforcement Learning. NeurIPS 2021\n\nAlthough previous works model (i) with recursive reasoning, there is a lack of recursive reasoning baselines. While the authors mentioned other recursive reasoning methods' reliance on the CTDE paradigm as justification for not comparing against them (line 89), I highly believe a comparison against these methods must be made. Since the learning agent decides its action without any centralised component during execution, we can see this as utilising privileged information that only exists during training. Even MBOM uses some privileged information that is not ordinarily accessible during execution (e.g. modelled agent's rewards) for training. In the worst case, recursive reasoning methods not based on CTDE training like I-POMDP-Net can be used as a baseline.\n\n3. Robustness to diverse opponents\nWhile one of the central claims of this work is that MBOM allows the learning agent to perform well against a diverse set of opponents, MBOM has not been evaluated against agents whose reward function is highly different to those encountered during training. This is particularly important since many applications require learning agents to deal with previously unseen decision-makers whose reward functions are unknown (e.g. humans in autonomous driving scenarios). In this case, an experiment to evaluate MBOM's capability in those scenarios is also necessary.\n\n4. Lack of experiments against agents with deeper levels of recursive reasoning.\nWhile the reasoning learner is an example of agents that model the controlled agent, there is a lack of evaluation against agents with deeper levels of recursive reasoning. A potential improvement is to evaluate MBOM against another MBOM agent with various depths of reasoning.\n\n5. A lack of analysis on model accuracy\nThe current work's analysis only reports the returns resulting from applying MBOM and the baselines. Nevertheless, it is important to report the model accuracy in any work in opponent modelling. Future iterations of this work can measure the log-likelihood of modelled agents' predicted actions.\n\n6. Scalability to scenarios with more opponents.\nIt would be interesting to see whether MBOM scales to scenarios with more modelled agents. In particular, the rollout procedure done at every level of recursive reasoning requires more rollouts as the joint action space increases. Yet, the experiments have been limited to scenarios with small number of agents.\n\nClarity\n1. Imprecise statements.\nOverall, there are a few statements that are imprecise in the manuscript. Since these may potentially confuse readers, I recommend fixing these highlighted sentences:\n\n(Line 8) \"All kinds of opponents\" --> Be precise on the type of agents that are evaluated in this work's experiments. \n(Line 20-21) \"interacting with diverse opponents makes the environment nonstationary from the agent's perspective.\" --> This is only true when other agents' policies are changing. When agents' policies are fixed, one can account for the effect of agents' actions in the transition function. Even when agents' policies are unknown, the problem can be seen as a POMDP as long as these policies remain fixed.\n\n2. Lack of details in Figure 1\nIf done correctly, Figure 1 can help readers understand the proposed approach. However, the lack of captions in Figure 1 explaining the components of MBOM makes it challenging to understand the Figure. Also, consider adding labels associated to the dashed boxes to indicate which components of MBOM they represent.\n\n3. Justification on baseline and opponent design\nThe experiment section can be improved by highlighting why specific baselines or opponents are designed the way they are. Focusing on the insights gained from comparisons against specific baselines or experiments using certain opponents can also help highlight the claims provided in this work.\n\n\nSignificance\n\nWhile this work presents an interesting approach that has potentially major significance for people working in MARL and opponent modelling, further experiments and comparisons are required to fully demonstrate its use in modelling (i) adaptive learning agents and (ii) agents that also learn models of the learning agent. Its limited comparison against prior recursive reasoning methods and the lack of analysis in terms of model accuracy particularly stands out as why this work has limited significance as is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1kjuE56_l2Q", "reviewer": "Reviewer_tVUV", "summary": "This paper proposes a model-based opponent modeling method to handle the interaction with other agents in multiagent systems. The main contribution of the proposed method is that it can adapt to different kinds of opponent policy, e.g., fixed-type, adaptive-type, and reasoning-type. Concretely, the proposed method first lets the agent interact with diverse opponents and collects the interaction experience. Then it uses the collected experience to train an environment model. Using this environment model, the agent imagines the adaption of the opponent policy and uses the imagined best response of the opponent to refine the opponent model. The agent repeats this process to get a set of opponent models reflecting different recursive reasoning levels. The authors further use Baysian mixing to get the final opponent model based on the opponent model set. As the Bayesian mixing is non-parametric, the final opponent model can quickly adapt to the true dynamic opponent policy.", "questions": "1. In the proposed MBOM, the authors pre-train the agent’s policy using PPO while interacting with $v$ different opponents with diverse policies. However, I am surprised that the PPO can train a good agent policy in this way. In my opinion, the RL training highly relies on the sampled experience. The training process is trying to find a (local) optimal policy that can maximize the expected return given the sampled experience. Therefore, when trained with a certain type of opponent A, it should be easy for PPO to learn a good policy. However, if a different type of opponent B joins the training process, it will affect the interaction experience, i.e., in some episodes, the agent interacts with opponent A while in other episodes the agent interacts with opponent B. In this case, the PPO will adjust the policy trained with opponent A because it needs to consider the experience corresponding to opponent B. As a consequence, the new policy will likely have worse performance when facing opponent A especially when opponent A is very different from opponent B. Therefore, I would like to know more details about how to avoid this kind of issue and learn a good policy when interacting with $v$ different opponents with diverse policies. For example, what is the input of agent policy in every step during the training? I think the agent policy pre-training is critical because, during the recursive imagination, the agent policy is fixed and plays an important role.\n\n2. Lemma 2 is critical for the theoretical analysis. However, is it a practical assumption? In what kind of scenarios this assumption will hold?\n\n3. The authors use LOLA-DiCE as one baseline because it considers the potential learning of opponents. However, if the assumptions required by LOLA-DiCE do not hold, the consideration of opponent learning may bring adverse effects instead of benefits. This is also observed in Figure 2 where LOLA-DiCE has almost the worst performance in every case. Therefore, it may be better if the authors can also compare with other opponent modeling works, e.g. [27], [30]. Although the authors think they only consider opponent policies that are independent of the agent, sometimes simpler settings can be better than complex settings like in LOLA-DiCE.\n\n4. MBOM has a pre-train phase where it sees a set of opponents with diverse policies. In comparison, Meta-PG only uses trajectories from the current opponents. Do the pre-train phase brings more information to MBOM and thus, gives it some advantages when interacting with the current opponents?", "limitations": "1. The assumption in Lemma 3 seems strong. Although the authors claim that \"larger M improves the representation capability of IOPs and thus better satisfies the assumption in Lemma 3\", larger M also brings higher approximation error (which hinders the representation capability). Therefore, there is no guarantee that the assumption in lemma 3 can hold. If it does not hold, the Bayesian mixing may not approximate the true probability of the opponent well. Moreover, the recursive imagination will incur a considerable amount of computational cost.\n\n2. There are duplicate texts on page 6: \"However, larger M also has advantages. To analyze, we first define the benefit using the mixed IOP as...\"", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: this paper tackles a challenging opponent modeling problem and the proposed method is a good combination of well-known techniques. The existing works are discussed in detail and this paper situates itself in the literature well. The presentation of this paper is also clear. The experiment results are sound and support the authors' claim.\n\nWeaknesses: some technical parts need more description and explanation. Some existing works may also be worthy to compare.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept