Model-Based Opponent Modeling
Authors: XiaoPeng Yu
Jiechuan Jiang
Haobin Jiang
Zongqing Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning
opponent modeling

Abstract: When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponent. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponent. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in competitive and cooperative environments, respectively with different types of opponent, i.e., fixed policy, naive learner, and reasoning learner.

Under review as a conference paper at ICLR 2022
MODEL-BASED OPPONENT MODELING
Anonymous authors
Paper under double-blind review
ABSTRACT
When one agent interacts with multi-agent environment, it is challenging deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simulta- neously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponent. In this paper, we propose model-bass --bsde opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponent. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To ef- fectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors opponents. Empirically,
show that MBOM achieves than existing methods in competitive and cooperative environments, respectively with different types of opponents, Le., fixed policy, naïve learner, and reasoning learner.
effective adaptation
INTRODUCTION
Reinforcement learning (RL) has made great progress in multi-agent competitive games, e.g., Al- phaGo (Silver et al., 2016). OpenAl Five (OpenAI, 2018), and AlphaStar (Vinyals et al., 2019). In multi-agent environments, an agent usually has to compete against or cooperate with diverse other agents (collectively termed as opponents whether collaborators or competitors) unseen before. Since the opponent policy influences the transition dynamics experienced by the agent, interacting with di- verse opponents makes the environment seem nonstationary from the agent's perspective. Due to the complexity and diversity in opponent policies, it very challenging for the agent to retain overall supremacy.
Explicitly modeling the behaviors, goals, or beliefs of opponents (Albrecht & Stone, 2018), rather than treating them as part of the environment, could help the agent adjust its policy adapt to different opponents. Many studies rely on predicting the actions (He et al., 2016; Hong et al., 2018; Grover et al., 2018: Papoudakis & Albrecht, 2020) and goals (Rabinowitz et al., 2018; Raileanu et al., 2018) of opponents during training. When facing diverse or unseen opponents, the agent policy conditions on the prediction representation generated by corresponding modules. However, opponents may also have the same reasoning ability, e.g., an opponent who makes prediction about the agent's goal. In this scenario, higher-level reasoning and some other modeling techniques are required to handle such sophisticated opponents (Wen et al., 2019; Yang et al., 2019; Yuan et al., 2020). In addition, the opponents may learn simultaneously, and the modeling becomes unstable and the fitted models with historical experiences lag behind. To enable the agent to continuously adapt to learning opponents, LOLA (Foerster et al., 2018a) takes into account the gradients of the opponent's 's learning for policy update, Meta-PG (Al-Shedivat et al., 2018) formulates continuous adaptation as a meta-learning problem, and Meta-MAPG (Kim et al., 2021a) combines meta-learning with LOLA. However, LOLA requires knowing the learning algorithm of opponents, while Meta-PG and Meta-MAPG require all opponents use a same learning algorithm.
Unlike existing work, we do not make such assumptions and focus on enabling the agent to learn ef- fectively by directly representing the policy improvement of opponents when interacting with them, even if they may be also capable of reasoning. Inspired from the intuition that human could antici- pate the future behaviors of opponents by simulating the interactions in the brain after knowing the rules and mechanics of the environment, in this paper, we propose model-based opponent modeling

---

Under review as a conference paper at ICLR 2022
(MBOM), which employs the environment model to predict and capture the policy improvement of opponents. By simulating the interactions in the environment model, we could obtain the best responses of opponents under the agent policy conditioned on the opponent model. Then, the op- ponent model can be fine-tuned using the simulated best responses to get the higher-level opponent model. The higher-level opponent model reasons more deeply and thus more competitive. By re- cursively repeating the simulation and fine-tuning, MBOM imagines the learning and reasoning of opponents and generates a set of opponent models with different levels, which could also be seen as recursive reasoning. However, since the learning and reasoning of opponents are unknown, certain-level opponent model might overestimate or underestimate the opponent. To effectively and accurately represent the opponent policy, we further propose to mix the imagined opponent policies according to the similarity with the real behaviors of opponents updated by Bayesian rule.
We evaluate MBOM in the two competitive tasks, Triangle Game and One-on-One, against three types of opponent, egg., fixed policy, naïve learner, and reasoning learner. MBOM outperforms strong baselines, especially when against naive learner and reasoning learner, which demonstrates the effectiveness of recursive imagination and Bayesian mixing. We also show the adaptation ability of MBOM in the cooperative task Coin Game
2 RELATED WORK
2.1 OPPONENT MODELING
In multi-agent reinforcement learning (MARL), is big challenge to form robust policy due to the unknown opponent policy. From the perspective of an agent, opponents are considered as a part of the environment, the environment is unstable and complex for policy learning when the policies of opponents are also changing. If the information about opponents is included, e.g., behaviors, goals and belicfs, the environment may become stable, and the agent could learn using single-agent RL methods. This line of research is opponent modeling.
One simple idea of opponent modeling is build model each time new opponent group of opponents is encountered (Zheng et al., 2018). However, learning a model every time is not efficient. A more computationally tractable approach is to represent an opponent's policy with an embedding vector. Grover et al. (2018) uses a neural network as encoder, taking as input the trajectory of one agent. Imitation learning and contrastive learning are used to train the encoder. Then, the learned encoder can be combined with RL by feeding the generated representation into policy or/and value network. Learning of the model can also be performed simultaneously with RL, as an auxiliary task (Jaderberg et al., 2016). Based on DQN (Mnih et al., 2015), DRON (He et al., 2016) and DPIQN (Hong et al., 2018) use a secondary network which takes observations as input and predicts opponents' actions. The hidden layer of this network is used by the DQN module to condition on for a better policy. It is also feasible to model opponents using variational auto- encoders (Papoudakis & Albrecht, 2020), which means the generated representations deterministic vectors, but nigh-dimensional distributions. ToMnet (Rabinowitz et al., 2018) tries to make agents have the same Theory of Mind (Premack & Woodruff, 1978) humans. ToMnet consists of three networks, reasoning about the agent's is action and goal based on past and current information. SOM (Raileanu et al., 2018) implements Theory of Mind from a different perspective. SOM uses own policy, opponent's observation and opponent's 's action to work backwards to learning opponent's 's goal by gradient ascent.
no longer
The methods aforementioned only consider opponent policies which are independent of the agent. If opponents hold belief about the agent, the agent can form higher level belief about opponents' belief. This process can perform recursively, termed recursive reasoning. PR2 (Wen et al., 2019) uses SAC (Haarnoja et al., 2018) to approximate opponents' policies conditioning on the agent's action. Since PR2 uses the agent's Q-function to estimate opponents' policies, it can only be applied in cooperative environments. Yuan et al. (2020) takes both level-O and level-1 beliefs as input to the value function. Level-0 belief is updated according to Bayesian rule, and level-1 belief is updated using a learnable neural network. However, these methods use centralized training decentralized execution algorithms to train set of fixed agents that cannot handle diverse opponents in execution. Bayes-ToMoP (Yang et al., 2019) is developed to detect and handle different reasoning opponents, but the implementation is complex and not scalable.
2

---

Under review as a conference paper at ICLR 2022
If the opponents are also learning, the modeling mentioned above becomes unstable and the fitted models with historical experiences lag behind. So it is beneficial to take into consideration the learning process of opponents. LOLA (Foerster et al., 2018a) introduces the impact of the agent's policy on the anticipated parameter update of the opponent. A neural network is used to model the opponent's policy and estimate learning gradient of the opponent's policy, implying that the learning algorithm used by the opponent should be known, otherwise the estimated gradient will be inaccurate. Further, the opponents may still be learning continuously during execution. Meta-PG (Al-Shedivat et al., 2018) 88) is a method based on meta policy gradient, using trajectories come from the current opponents do multiple meta- cetaa-grdieet steps and construct policy that is good for the updated opponents. Meta-MAPG (Kim et al., 2021a) extends this method by including an additional term that accounts for the impact of the agent's current policy on the future policies of opponents, similar to LOLA. These meta-learning based methods require that the distribution of trajectories match across training and test, which implicitly means all opponent use a same learning algorithm.
2.2 MADEL-BASED RL AND MARL
Model-based RL allows the agent to have access to the transition function. There are two typical branches of model-based RI. approaches: background planning and decision-time planning. In background planning, the agent could the learned model to generate additional experiences for assisting learning. For example, Dyna-style algorithms (Sutton, 1990; Kurutach et al., 2018; Luo al., 2019) perform policy optimization on simulated experiences, and model-augmented value expansion algorithms (Feinberg et al., 2018; Oh et al., 2017; Buckman et al., 2018) use model-based rollouts to improve the update targets. In decision-time planning (Chua al., 2018), the agent could use the model to rollout the optimal action at a given state by looking forward during execution, e.g., model predictive control.
The model-based MARL methods could be divided into two classes: centralized model and decen- tralized model. AMLAPN (Park al., 2019) builds a centralized auxiliary prediction network to model the environment dynamics and the opponent actions to alleviate the non-stationary dynam- ics. A centralized multi-step generative model (Krupnik et al., 2020) with a disentangled variational auto-encoder has been proposed for performing trajectory planning. For decentralized model, IS (Kim et al., 2021b) uses the model to generate imagined trajectories for multi-agent communication to share intention. HPP (Wang et al., 2020) uses the models to propose and evaluate navigation subgoals for the rendezvous task.
Our method relaxes the limitations opponent modeling. We make assumptions about the variation of opponents' policies. They could be fixed, randomly sampled from an unknowable policy set, continuously learning using unknowable and changeable RL algorithm, both in training and execution. We lear an environment model that allows the agent to perform recursive reasoning against opponents who may also have the same reasoning ability.
PRELIMINARIES
We consider an n-agent stochastic game (S.A .S... space, AA' the action space of agent E [1...., n.). A S -> [0. 1] is transition function, R$ S AxAxS
Ath P. R R², where S is the state
= II A is the joint action space of
the reward function of agent i, and is the discount factor. The policy of agent i is n², and the joint policy of other agents is π <a<aa-i = 11, n² (a/ls), where ai is the joint action except agent i. All agents interact with the environment simultaneously without communication. The historical trajectory is available, i.e., for agent at timestep t, {so, ab. ap ..... St-1, af a is observable. The goal of the agent i is to maximize its expected cumulative discount rewards
agents, P:
y 'R' ai ar' s0ii)
E 21 P111~P(1s a ( (-Is:)a, for '(-1ec)
LE=
For convenience, the learning agent treats all other agents a joint opponent with the joint action a ~ 7 (-Is) and reward r.. The action and reward of the learning agent are denoted as a ~ (-|ss) and r, respectively.
3

---

Under review as a conference paper at ICLR 2022
Recursive Imagination
for m=0 M-2
Rollout
Agent
Recursive Imagination
Environment model
aa 1s,a";
Opponeet
BayesSan Mixing
Bayesian Mixing
+ to
1P los vhu)
4/
nofur-softmax
Catel (s)
* di)
(lb.0":0)
*- los sp
Fiace (+)
Figure 1: Architectures of MBOM
4 MODEL-BASED OPPONENT MODELING
MBOM employs the environment model to predict and capture the learning of opponent policy. By simulating recursive reasoning via the environment model, MBOM imagines the learning and reasoning of the opponent and generates set of opponent models. To obtain stronger representation ability and accurately capture the adaptation of the opponent, MBOM mixes the imagined opponent policies according to the similarity with the real behaviors opponent.
4.1 RECURSIVE IMAGINATION
If the opponent is also learning during interaction, the opponent model fitted with historical experi- ences always lag behind, making the agent hard adapt the opponent. Moreover, if the opponent could adjust its policy according to the actions. intentions, or goals of the agent, then recursive rea- soning may occur between agent and opponent. However, based on the lagged opponent model, the agent would struggle to keep with the learning of opponent. To adapt to the learning and reasoning opponent, the agent should predict the current opponent policy and reason more deeply than the opponent.
MBOM explicitly simulates the recursive reasoning process utilizing the environment model, called recursive imagination, to generate series of opponent models, called Imagined Opponent Poli- cies (IOPs). Initially, the agent interacts with 77 different opponents which are learning with PPO (Schulman et al., 2017). and collect buffer D which contains the experience (s, a, a", 8 r). where r r= (r, r"). ) For zero-sum game (r + r" = 0) and fully cooperative game (r r"), r can be easily obtained, while for general-sum game
make mild assumption that r can be accessed dur- ing experience collection. Using the experience buffer D, we can train the the environment model I'(s,, r|s, a, a"; () by minimizing the mean square error
E 6,0,a PND
2 I1s' - s'11 + 2||rr 2
(2)
where 8 T(s, a, a²; (P;C), (), obtain the level-0 IOP #0 (-1s; Φo) by maximun-likelihood estimation
log # (a -a Φo),
33)
and learn the policy of the agent T(-|s, aa: 0) using PPO. By running PPO, we could also get the value function (s) of the agent. To imagine the learning of the opponent, as shown in Figure 1, we use the rollout algorithm (Tesauro & Galperin, 1996) get the best response of the opponent under the agent policy æ. For each opponent action af at timestep t, we uniformly sample the opponent action sequences in the next à timesteps, simulate the trajectories using the leamedd environment

---

Under review as a conference paper at ICLR 2022
Algorithm MBOM
Pretraining:
2: Initialize the recursive imagination layer M.
3: Initialize the weights a with (1,0,.... 0).
4: Interact with 1) learning opponents and collect the experience buffer D.
5: Train the level-0 IOP du, the environment model c and the agent policy using the experience buffer D.
6: Interaction:
7: for = L..... _ae__eecchhce do
8: Interact with the opponent to observe the real opponent actions.
\//Recursive Imagination)
9:
Fine-tune 6 with the real opponent actions
10:
for = 1, . .M- do
11:
Rollout in I'c with π T-Iss, a" ~ (-Is; Φi- 1); 0) to get the best responses of the opponent
axx by equation 4 or equation 5. Fine-tune 0;-1 with the best responses to get the Ø;.
12:
end for \/Baaessann Mixing)
Update by equation
15:
Mix the IOPs get ahix by equation 6.
16:
end for
model c. and select the best response with the highest rollout value
0 argmax oy
max
Unil
==0
During the simulation, the agent acts according to the policy conditioned on the modeled opponent policy, a; ~ ot (-118t, ag ~ (-[8;; Φo); 0), and learned environment model provides the transition 8141,7E Pt = T(st, az, o:: (). With larger k, the rollout has longer planning horizon, and thus could evaluate the action a" more accurately, assuming a perfect environmental model. However, the computation cost of rollout increases exponentially with the planning horizon to get an accurate estimate a while in practice the compounding error the environmental model also increases with the planning horizon. Therefore, the choice of k is trade-off between accuracy and cost. Specifically, for zero-sum game and fully cooperative game, we can approximately estimate the opponent state value (s) (s) and Vss), respectively, and modify the rollout value like n-step return (Sutton & Barto, 2018) to obtain longer horizon
D 01
argmnax of
max
At 245 + V (st-R41).
(5)
Unit
==
By imagination, we can obtain the best response of the opponent under the agent policy π and level- IOP 7 (=[sss; Φo) and construct the simulated data {(s, d_")). Then, we use the data to fine-tune the level-0 IOP (- [=]s; Φo) by maximum -ikimummlikellhoodd estimation, and obtain the level-1 1 IOP #- ).. The level-1 IOP can be see as the best response of agent to the response of opponent to level-0 IOP. In the imagination, it is the nested form as "the opponent believes [that the agent believes (that the opponent believes ...)]." The existing imagined opponent policy is the innermost "(that the opponent believes)," " while the outermost "the opponent believes" is a which is obtained by rollout process. Recursively repeating the rollout and fine-tuning, where the agent policy is conditioned on the IOP (-(-ss; dm-1), we could derive the level-m IOP # (-|s; pm). The higher-level IOP reasons more deeply, and thus more competitive.
4.2 BAYESIAN MIXING
By recursive imagination, we get M IOPs with different reasoning levels. However, since the learn- ing and reasoning of the opponent unknown, single IOP might overestimate underestimate the opponent. To obtain stronger representation ability and accurately capture the learning of the opponent, illustrated in Figure we linearly combine the IOPs get mixed policy, similar idea
5

---

Under review as a conference paper at ICLR 2022
P1
P2
L3
(a) Triangle Game
(b) One-on-One
Figure 2: Illustrations of the scenarios.
was also present in (Wen al., 2020)
M
π mii (-18) = Σ o; 1=0
(-);;
(6)
where is the weight of level-i IOP, which is produced by the IOPs mixer,
(ao. ................................ (MM.:)))MM.))) softer-softmax Wo MM-1).
Softer-softmax (Hinton et al., 2015) is variant of softmax, which uses higher temperature control a softy of the probability distribution over classes. Y is the decayed moving average of p(m) a"), which is the probability of using the level-m IOP given the action of the opponent a". By Bayesian rule, we have
p(m)a p(a(m))((m)
(a om pp(m)
(8)
En p(a)li)p(i) E [ (()(()
where p(m) is the probability of using the level-m IOP, and we estimate as the moving average of p(mla"). V... indicates the similarity between the level-m IOP and the opponent in the most recent stage. Given the opponent actions, the higher V means that the actions are more likely be generated from the level-m IOP, and thus the level-m IOP is more similar the opponent. Adjusting the weights according to the similarity could obtain more accurate estimate of the improving opponent policy. Moreover, the IOPs mixer is non non-parametrie, which could be updated quickly and efficiently without parameter training and too many interactions. Therefore, the IOPs mixer could adapt to the fast-improving opponent.
For completeness, the whole procedure of MBOM in given in Algorithm 1.
5
EXPERIMENTS
5.1
SETTING UP
We evaluate MBOM in competitive environments.
Triangle Game is an asymmetric zero-sum game on two-dimension with multi-step actions, imple- mented on Multi-Agent Particle Environment (Mordatch & Abbeel, 2017; Lowe al., 2017) (MIT license). As shown in Figure 2(a), there are two moving players, Pl and P2, and three fixed land- marks, L1-L3, triangle with the side length 0.6. When the distance between a player and a landmark is less than 0.15, the agent touches the landmark and has the state T. T1 indicates that the player touches the landmark LI, and matrix of the two players shown in Table 2(a). P2 has inherent disadvantages since the optimal solution of P2 always strictly depends the state of P1. When facing different policies of P1, P2 has to adjust its policy to adapt to P1 for higher reward. We control P2 as the agent and take P1 as the opponent.
square field. The landmarks are located at the three vertexes of an equilateral
If the player does not touch any landmark, the player state is F. The payoff
One-on-Ona two-player competitive game implemented on Google Research Football Envi- ronment (Kurach et al., 2020) (Apache-2.0 License), physics-based 3D simulator. As shown in

---

Under review as a conference paper at ICLR 2022
Table I: Payoff matrix of Triangle Game
Player 2
F
T1
T2
T3
F
0/0
-0.5/ + 0.5
-0.5/ + 0.5
-0.5/ + 0.5
T1
+0.5/
- 0.5
+1/
+1/
-1/ +1 +1
Playerl
T2
+0.5/
0.5
-1/+1
+1/ +1/-1
+1/-1
T3
+0.5/ 0.5
+1/-1
-1/+1
+1/-1
Figure 2(b). there are two players, the shooter, which could dribble and shoot the ball, and the goal- keeper. The shooter controls the ball in the initial state. The episode ends after 30 timesteps or when the shooter loses possession of the ball. At the end of an episode, if the shooter shoots the ball into the goal, the shooter will get a reward +1, and the goalkeeper will get a reward -1. Otherwise, the shooter will get reward of 1,, and the goalkeeper will get reward of +1. The goalkeeper could only passively react to the strategies of the shooter and makes policy adaptation when the shooter strategy changes. We control the goalkeeper the agent and take the shooter as the opponent.
Baselines. In the experiments, we compare MBOM with the following methods:
LOLA-DiCE (Foerster et al., 2018b) is an expansion of the LOLA, which uses Differ- entiable Monte-Carlo Estimator (DiCE) operation to considers how shape the learning dynamics other agents.
Meta-PG (Al-Shedivat et al., 2018) uses trajectories come from the current opponents to do multiple meta-gradient steps and construct a policy that is good for the updated opponents.
Meta-MAPG (Kim et al., 2021 a) includes an additional term that accounts for the impact of the agent's current policy on the future policies of opponents, compared with Meta-PG. PPO (Schulman et al., 2017) is classical single agent RL algorithm, without any other modules.
MBOM w/o IOPs always uses do as the opponent model, without recursive imagination and Bayesian mixing.
The baselines have the same neural network architectures MBOM. All the models are trained for five runs with different random seeds. All the curves are plotted using mean and standard deviation. More details about experimental settings and hyperparameters available in Appendix A.
Opponents. For the opponents, we run independent PPO (Schulman et al., 2017) algorithms for 10 times. During each running, store 20 opponent policies in the training set, 3 opponent policies in the validation set, and opponent policies in the test set. So the sizes the training set, validation set, and test set 200, 30, and 30. We pre-train the agent with the opponents in the training set and make the agent interact with the opponents in the test set to evaluate the ability of generalization and adaptation. The validation set is only required by Meta-PG and Meta-MAPG. We construct three types of opponents:
Fixed policy. The opponents in the test set will not update during the interaction. Naïve learner. The opponents in the test set will be updated with PPO during the interac- tion.
Reasoning learner. The opponents could model the behavior of the agent and will be fine-tuned during the interaction.
To increase the diversity of the opponents, we adopt the reward shaping technique and add invisible barriers in the environment during the training.
5.2 PERFORMANCE
The experimental results against test opponents are shown in Figure 3, and the mean performance with standard deviation over all test opponents is summarized in Table and 3. In Triangle Game, the opponent could take different strategies, e.g., hovering around landmark. commuting between two landmarks, or rotating among three landmarks, whereas the agent has to anticipate and adapt to the opponent strategies. The learning I LOLA-DiCE depends the gradient of the opponent

---

Under review as a conference paper at ICLR 2022
entroward
apponent
opponentindes
Triangle Game, fixed policy
(b) One-on-One, fixed policy
opponort
opponant indox
(c) Triangle Game, versus naive learner
(d) One-on-One versus naive learner
ancewond)
(c) Triangle Game, reasoning learner
(f) One-on-One versus reasoning leamne
Meta-MAPG
Meta-PG
LOLA-DiCE
PPO
MBOM w/o IOPs
MBOM
Figure 3: Performance of adaptation against different types of opponents, i.e., fixed policy, naive leamner, and reasoning learner. The results are plotted using mean and standard deviation with five different random seeds (x-axis is opponent index). The results show that MBOM can outperform other baselines, especially against naïve learner and reasoning learner.
model. However, the gradient information cannot clearly reflect the distinctions between diverse opponents, which leads to that LOLA-DiCE cannot adapt to the unseen opponent quickly and effec- tively. Meta-PG and MAPa show stronger adaptation abilities than LOLA-DiCE when facing different opponents, but fail some opponents (index 0, 1, and 2 in Figure 3(a). 3(c), and 3(e)). By visualization, we find that those opponents are hovering around one landmark, which are out of the training set. Since Meta-PG and Meta-MAPG assume that the test set follows the distribu- tion of the training set, these two methods are unable to adapt to the out-of-distribution opponents. MBOM w/o IOPs obtains similar results MBOM when facing fixed policy opponents because du could accurately predict the opponent behaviors the opponent is fixed. However, if the opponent learning and reasoning, dn cannot accurately represent the opponent, and thus the performance of MBOM w/o IOPs drops. By explicitly simulating the recursive reasoning process, MBOM shows more obvious performance gain against naive learner and reasoning learner, as Table 2. In One- on-One, the shooter has absolute dominance and more flexible strategy choices, including shooting location, shooting angle, and shooting time. Due the limitation of gradient information, LOLA- DiCE still shows poor udaptability. Meta-PG and Meta-MAPG heavily rely on the reward signal, and thus adaptation is difficult for the two methods this sparse reward task. MBOM w/o IOPs and MBOM achieve similar performance with fixed policy, but the performance of MBOM significantly improves the opponent improving as shown Table 3, which contributed the recursive reasoning capability given by recursive imagination in the environment model and Bayesian mixing that quickly captures the learning opponent.
5.3 COOPERATIVE TASK
MBOM could also be applied to cooperative tasks. We test the performance MBOM on coop- erative scenario, Coin Game, which is high-dimension expansion of the iterated prisoner dilemma
8

---

Under review as a conference paper at ICLR 2022
Table 2: Performance Triangle Game
Fixed Policy
Naïve Learner
Reasoning Learner
LOLA-DiCE
-22.513 (18.208)
-204477 (18.914)
-21.554 (18.708)
Meta-Pe PG
-3.777(6.034)
-6.718 (5.245)
-8.350 (11.779)
Meta-MAPG
-2.007 (5.639)
-5.764 (5.180)
-6.136 (9.911)
PPO
-13.292 (8.223)
-18.416 (8.084)
-20,508 (12.335)
MBOM w/o IOPs
-1.142 (3.895)
-3.230 (2.583)
-7.101 (16.985)
MBOM
-1.188 (3.871)
-1.659 (2.817)
-2.746 (14.243)
Table 3: Performance on One-on-One
Fixed Policy
Naive Learner
Reasoning Learner
LOLA-DiCE
-0.339 (0.414)
-0.496 (0.365)
-0.632 (0.433)
Meta-PG
-0.165 (0.417)
-0.291(0.371)
-0.356 (0.439)
Meta-MAPG
-0.112 (0.419)
-0.268 (0.365)
-0.378 (0.433)
PPO
-0.188 (0.274)
-0.569 (0.260)
-0,483 (0.309)
MBOM w/o IOPs
0.188 (0.372)
-0.023 (0.382)
0.113 (0.465)
MBOM
0.190 (0.378)
0.025 (0.377)
0.284 (0.403)
The numbers in table refer and standard deviation of return 30 opponents with different random seeds.
2
O
-1/+1
0/0
20-
O
O
+1/0
-2/+1
LOLNOICE
PL McGU
-
NDON AN ICPs MDON
+1/-2
0/+1
200
125
150
172
202
Iaraaoon
(a) Coin Game
(b) Learning curves Coin Game
Figure 4: (a) Illustration of Coin Game; (b) Learning curves in Coin Game, which are plotted using mean and standard deviation with five runs with different random seeds.
with multi-step actions (Lerer & Peysakhovich, 2018: Foerster al., 2018a). There are two players, red and blue, moving on a 3 x grid field, and two types of coins, red and blue, randomly generated on the grid field. If the player moves to the position of the coin, the player collects the coin and receives a reward +1. However, the color the collected coin is different from the player's color, the other player receives reward of 2,, illustrated in Figure 4(a). The length of the game is 150 timesteps. Both agents simultancously update their policies using the same MBOM or other baselines maximize the sum of rewards.
The experiment results are shown in Figure 4(b). Meta-PG and Meta-MAPG degenerate to Policy Gradient for this task as there is no training set. Both learn greedy strategy that collecting any color coin, which leads to total score of two players is zero. LOLA-DiCE learns too slow and does not learn to cooperate within 300 iterations, indicating the inefficiency of estimating the opponent gradients. MBOM w/o IOPs and MBOM learn to cooperate quickly and successfully. and MBOM shows smaller standard deviation than MBOM w/o IOPs, indicating that MBOM can also be applied to cooperative tasks without negative effect.
CONCLUSION AND FUTURE WORK
We have proposed model-based opponent modeling. MBOM employs recursive imagination and Bayesian mixing predict and capture the learning and improvement of opponents. Empirically,

---

Under review as a conference paper at ICLR 2022
evaluated MBOM in two competitive environments, and demonstrated MBOM adapts to learning and reasoning opponents much better than the baselines. These make MBOM a simple and effective RL method whether opponents be fixed, continuously learning, reasoning in competitive environ- ments. Moreover, we also verified the adaptation ability of MBOM in cooperative environments.
In MBOM, the learning agent treats all opponents a joint opponent. If the size of the joint opponent large, the agent will need a lot rollouts to get an accurate best response. The cost increases dramatically with the size of the joint opponent. How reduce the computation overhead in such scenarios will be considered in future work. Moreover, MBOM implicitly assumes that the relationship between opponents is fully cooperative. Dealing with the case where their relationship is non-cooperative is also left as future work.
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda. Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In In- ternational Conference on Learning Representations (ICLR), 2018.
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-92. 2018
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems (NeurIPS), 2018.
Vladimir Feinberg, Alvin Wan, lon Stoica, Michael I Jordan, Joseph E Gonzalez. and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv: 1803. 00101, 2018.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In International Conference on Au- tonomous Agents and MultiAgent Systems (AAMAS), 2018a.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In International Conference on Machine Learning (ICML), 2018b.
Aditya Grover, Maruan Al-Shedivat, Jayesh K. Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning (ICML), 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning (ICML), 2018
He He. Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference on Machine Learning (ICML), 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503 150.00233,, 2015.
Zhang-Wa Wei Hong, Shih- Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee, A deep policy inference nettwwrk for multi-agent systems. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, and K. Kavukcuoglu. Reinforcement learn ing with unsupervised auxiliary tasks. In International Conference on Learning Representations (ICLR), 2016.
10

---

Under review as a conference paper at ICLR 2022
Dong-Ki Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning, 2021a.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforeement learning: Intention sharing. In International Conference on Learning Representations (ICLR), 2021b.
Orr Krupnik, Igor Mordatch, and Aviv Tamar. Multi-agent reinforcement learning with multi-step generative models. In Conference on Robot Learning (CoRL), 2020.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zając, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research foot- ball: A novel reinforcement learning environment. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations (ICLR), 2018.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning, 2018.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments. Advances Neural Information Process- ing Systems (NeurIPS), 2017.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Interna- tional Conference on Learning Representations (ICLR), 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint
2017.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
OpenAl. Openai five. https: http:///ll..............o cotpenai.commppeaai--iiii
2018.
Georgios Papoudakis and Stefano Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829, 2020.
Young Joon Park, Yoon Sang Cho, and Seoung Bum Kim. Multi-agent reinforcement learning with approximate model learning for competitive games. PloS one, 14(9):e0222215, 20199.
David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):1155526 1978.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference Machine Leurning (ICML), 2018.
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.

---

Under review as a conference paper at ICLR 2022
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):844 489, 2016.
Richard Sutton. Integrated architectures for learning, planning, and reacting based approx- imating dynamic programming. In International Conference on Machine learning proceedings (ICML). 1990.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
G. Tesauro and G. R. Galperin. On-line policy improvement using monte-carlo search. In Advances in Neural Information Processing Systems (NeurlPS), 1996.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):550-354 2019.
Rose E Wang, Chase Kew, Dennis Lee, Edward Lee, Brian Andrew Ichter, Tingnan Zhang, Jie Tan, and Aleksandra Faust. Model-based reinforcement learning for decentralized multiagent rendezvous. Conference on Robot Learning (CoRL., 2020.
Ying Wen, Yaodong Yang. Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
Ying Wen, Yaodong Yang, Rui Luo, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. In International Joint Conference on Artificial Intelligence (IJCAI), 2020.
Tianpei Yang, Jianye Hao, Zhaopeng Meng, Chongjie Zhang, Yan Zheng, and Ze Zheng. Towards efficient detection and optimal response against sophisticated opponents. In International Joint Conference Artificial Intelligence (ICCAI), 2019.
Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Zhu. Emer- gence of pragmatics from referential game between theory of mind agents. arXiv preprint arXiv:2001.07752, 2020.
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang. Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non-stationary agents. In Advances in Neural Information Processing Systems (NeurIPS), 2018
12

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS
The hyperparameters of MBOM are summarized Table
Table 4: Hyper-parameters
Triangle Game
One-on-One
Coin Game
policy hidden units value hidden units
MLP[64,32]
LSTM[64,32]
MLP[64,32]
MLP[64.32]
MLP[64,32]
MLP[64.32]
activation function optimezer
ReLU
ReLU
ReLU
Adam
Adam
Adam
PPO
learning rate num. of updates
0.001
0.001
0.001
10
10
10
value discount factor
0.99
0.99
GAE parameter
0.99
0.99
0
clip parameter
0.115
0.115
0.115
hidden units
MLP[64.32]
MLP[64,32]
MLP[64.32]
learning rate
0.001
0.001
0.001
Opponent model
batch size num. of updates
64
64
64
10
10
num. of levels
3
2
learning rate
0.005
0.005
0.005
update times
3
3
3
IOPs
rollout horizon
2
5
1
decayed factor Yp
0.9
0.9
0.9
horizon of y s-softmax parameter
10
10
10
1
1.1/e
1
13

---

Under review as a conference paper at ICLR 2022
B ALPHA WEIGHTS ANALYSIS
Figure visualizes a of IOPs when the agent against different types of opponents, i.e., fixed policy, naive learner, and reasoning learner Triangle Game and One-on-One. With the three opponent types, (the weight of level-2 IOP) is remarkably higher than others, which indicates that recursive imagination of MBOM does learn pervasive policy of the opponent by the environmental model.
SulSoM cude
apponent
opponent
(a) Triangle Game, versus fixed policy
(b) One-on- One, versus fixed policy
suitow cude
opponent
opponent
(c) Triangle Game, versus learner
(d) One-on One, versus naive learner
sidtow cucke"
opponent
opponent
Triangle Game, versus reasoning learner
(f) One-on-One One, versus reasoning learner
Og
Q22
Figure 5: Visualization of cg 01 02 of MBOM, where a; is the weight of level-i IOP. The results are plotted using mean and standard deviation with five different random seeds (r-axis is opponent index).
14


Reviews:
[{"review_id": "rSG_JigEExu", "reviewer": "Reviewer_KD9g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments:\n\n1. When estimating the next level opponent, at time step t, the opponent's action is selected from the best action using the rollout with highest value, estimated by expanding k steps with the dynamics model. Below Algorithm 1, \"With larger k, the rollout has longer planning horizon, and thus could evaluate the action a^{o*} more accurately\" is problematic by noting that the rollout is generated by interacting with a learned dynamics model, which suffers uncertain dynamics error. Moreover, for zero-sum or cooperative setting, the opponent's value is estimated by directly using the negative self agent's value or the self agent's value. This is also questionable since for multi-agent settings, the problem is not an MDP again from the perspective of one agent. So, the value estimated from the self agent's observation conditions on only partial states without knowing the opponent's information. Based on this, the value is not exchangeable. Overall, I think the recursive reasoning part might rise many uncertainty issues and suffer approximation errors at least from the above two ways, and these uncertainty and error can be accumulated for higher level opponent reasoning. Unless the dynamics model error and value estimation error can be analyzed theoretically to guarantee the higher level opponent's value is also higher from the opponent's view, I am wondering whether the proposed recursive reasoning method can consistently generate stronger opponents or not. \n\n2. What is the case when the problem is not zero-sum? Also, when there are more than one opponents, would the recursive reasoning becomes more complicated, because when use a model to generate rollouts, the best response rollout should be selected by considering other opponents' actions?\n\n3. In the experiments, two-player games are considered. I think the original LOLA and a plain RL baseline without any reasoning should also be compared as references. Moreover, as the number of levels of IOPs and the length of rollouts k are important factors in the proposed method, these hyper-parameters should be studied with sufficient empirical results, which are absent from the current contexts. Most importantly, I think experimental results that can visualize the difference and improvements along the opponent models from lower level to higher level should be crucial to support the effectiveness of the proposed recursive reasoning. However, these results are not considered in the current experiment section.\n\nOverall, this paper proposes an interesting recursive way for modeling the opponent's policy, while there exists many issues to be solved in revisions.\n  \n\n", "summary_of_the_paper": "This paper proposes a model-based opponent recursive reasoning method for MARL problem. The recursive reasoning takes multiple levels of estimation to generate a number of opponents. The first level opponent is obtained by supervised learning from the true opponent actions. Then, with a learned dynamics model, the level-0 opponent interacts with self agent and the dynamics model to generate trajectories, from which by selecting one best response rollout, a better action at the current time step for the opponent is kept and used to estimate the next level opponent, by maximum log-likelihood again. After obtaining a number of levels of opponents, a mixture opponent is learned by softer-softmax weights. Experiments are conducted on multi-agent particle environment and google's football environment, and LOLA-DiCE and Meta-PG are compared.", "summary_of_the_review": "An interesting model-based opponent modeling method in MARL while approximation errors in recursive reasoning and more explanatory experimental results should be considered.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_uSST710u16", "reviewer": "Reviewer_SaHd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review**\nThis paper offers an interesting idea of effectively simultaenously modelling the same opponent under differing assumptions of their learning's complexity. It then used model-based predictive control with whichever opponent model is most similar to the opponents recent behavior (measured by comparing realization plans). This is a good, straight-forward, solution to the uncovered problem. Despite this, I am concerned both by the lacking of methodological details required to evaluate their empirical results, and the amount unsupported claims scattered throughout the paper. \n\n**Major Comments**\n- The set of baseline methods used are incredibly sophisticated making cross-comparisons very challenging. Simpler baselines should be included to help understand the benefit of the additional sophistication. Some easy examples are: variations on direct BR. Train a BR directly against a single opponent, mixture of the opponents, etc.. Moreover, a BR can be trained against the evaluation opponents to set an approximate upper bound on the performance these methods. Naive learner and regular LOLA can also be a baseline method, similar with other recrusive reasoning approahes discussed.\n- The proposed major contribution of this work is a learning-aware learning algorithm that is independent of the other-agent's learnin algorithm. This includes many claims about various levels of required recursive reasoning being necessary to achieve desired results. I would have liked to see quantitive and qualitative experiments that analyze the inter-relation of the levels of recursive reasoning of the agent wrt the levels of recursive reasoning performed by the other-agent. This would help the reader understand how necessary and under what settings MBOM is useful. Moreover, analysis into the distribution over the IOPs can provide further insights.\n- The authors liter throughout the paper unsupported claims that are often easily disproven. These distract the reader and undermine the credibility of the authors and the results herewithin. I've pointed out some of these within the Minor comments. Please ensure that claims are supported by either theoretical/empirical evidence or approriate citations.\n- I am having trouble following the construction and evaluation methodology wrt to the opponent policies. It would be helpful if included, perhaps in the appendix, was more details regarding the construction of the set of opponent policies. Especially focusing on evidence that supports that these policies are both diverse and represent a good coverage of the strategic space of the game. Additionally, how each evaluated algorithm was tuned and the opponent's algorithms. Without these details it's hard to appreciate the results provided and support claims such as \"out of distribution opponents\" -- your evaluation opponents may in-fact be within the same distribution and there is another issue.\n- If the coin game is a \"high-dimension expansion\" of IPD, it's confusing to me why there is an absolute failure to learn in LOLA-DiCE in Sec 5.3. IPD is the evaluation domain that is used in their work. Are you using their original implementation, or how have you evaluated that this method is working correctly? Has equal/sufficient tuning been applied to this method?\n- The \"Bayesian mixing\" methods proposed share many similarities with previous work on purification of mixed-strategy opponents and realization plans. Appropriate citations and dicussions of these relationships is necessary to contextualize the contributions here.\n\n**Minor Comments (not directly impacting score)**\n- Abstract (and throughout), \"opponents who are learning simultaneously or cpaable of reasoning\" please be careful about mentioning reasoning vs recursive reasoning, and define these terms early. I.e., reasoning doesn't say anything, should only use recursive reasoning.\n- Sec 1, Par 1, \"interacting with diverse opponents make the enviroment nonstationary\" this is not correct, stationarity of the environment is independent of the opponent. Upon abstracting the environment and other-agents into a black-box meta-environment, this meta-environment is only nonstationary if one of the components, environment or other-agent, is non-stationary (in the case of the other-agent being stochastic as mentioned here is insufficient).\n- Sec 1, Par 3, \"Inspired from the ... mechanics of the environment.\" I don't find this intuitive, humans  do not understand the reuls and mechanics of our environment. Suggestion, weaken staement, provide citations to recursive reasoning literature; esp. levels of recursive reasoning humans typically do\n- Sec 1, Par 3, \"Inspired from the intuition that human could anticipate ...\" Awkward\n- Sec 1, Par 3, \"The higher-level oppoennt model reasons more deeply and thus more competitive\" Awkward, and not strictly true, please justify such a claim.\n- Sec 1, Par 3, \"a certain-level opponnt model\", undefined terminology\n- Sec 2.1, Par 1, \"it is a big challeng to form robust policy\" Awkward\n- Sec 2.1, Par 2, \"One simple idea ... is encountered\", worth mentioning that this requires either the agent being notified of the opponent changing or defining a function that signals opponent change -- big assumptions, the later being error prone.\n- Sec 2.1, Par 2, \"A better appraoch is to represent an opponent's policy with an embedding vector\",  why is this a better approach? Please justify. It seems full models would be more accurate, but embeddings offer a more computationally tractable solution.\n- Sec 2.1, Par 3, \"Since PR2 uses the agent's.... only be applied in cooperative environments.\" This isn't true. Similar to this work, if assumptions are made about the game, such as zero-sum, we can infer opponent Q-values. \n- Sec 2.1, Par 3, \"However, these methods use ... cannot handle diverse opponents in execution.\" This is not generally true. The effectivness of a method's generalization to held-out opponent policies depends on how well the fixed set of agents covers the strategic landscape of the game. A point that the proposed algorithm is also vulnerable too. \n- Sec 2.1, Par 4, \"These meta-learning based methods require .... same learning algorithm\" why does the trajectory distribution imply similarity in learning algorithm?\n- Sec 2.2, Par 1, \"Model-based RL allows the agent to have access to the transition function\" Careful with phrasing, this isn't true. They depend on a transition function that may be learned or exact.\n- Sec 4.1, Par 1, \"To adapt to the learning and reasoning opponent... more deeply than the opponent\" why must this be true?\n- Sec 4.1, Par 2, \"obtain the level-0 IOP\", Awkward\n- Sec 4.2, Par 1, \"recursive imagination\" lots of interchanging of terminiolgy between simulation/imagination/reasoning, suggest adopting and defining one nomenclature and staying consistent. \n- Sec 4.2, Par 1, \"a softy\" ?\n- Sec 4.2, Par 1, How are you calculating p(i)?\n- Sec 4.2, Par 1, distinquishing where the prior work softer-softmax ends and modificaitons to work in this application begins can help the reader better understand contributions. \n- Sec 5.2, Par 1, \"The learning of LOLA-DiCE ... opponent quickly and effectively.\" This presumes you're training LOLA-DiCe against a set of opponent policies, which is not the same problem defn the solution was designed for; however, with batches of experiences carefully constructed across the opponent policies the gradient information could be effective. Especially when the gradients agree.\n- Sec 5.2, Par 1, please avoid describing results as significant without corresponding statistical tests\n- Sec A, Triangle mispelled in table.\n\n", "summary_of_the_paper": "Agents learning in systems with other-agents that may be simultaneously learning presents the original agent with a dynamic learning problem. Critically, the agent does not know if the other-agents are non-learning, learning, or learning using recursive-reasoning. One such solution to this problem is to design a learning algorithm that accounts for how the other-agents may update their policy from the shared experiences, effectively eliminating the aforementioned problem. This work attempts to design such an algorithm without relying on any assumptions on the other-agent's learning algorithm. Their proposed algorithm, Model-Based Opponent Modeling (MBOM), learns parallel opponent models that are trained using various depths of recursive reasoning. Then, when played against an opponent, the agent can select the most likely opponent model (level-0: non-learning opponent, level-1: learning-opponent, level-k: k-1 level recursivly learning opponent). This algorithm assumes a centralized training and decentralized execution setting for two-player fully-observable general-sum games. ", "summary_of_the_review": "I recommend rejecting the paper in its current state. While the authors identify a problem and provide a nice and simple solution to the problem, these are undermined by experiments that are underspecified and a lack of analytical experiments into the interworkings of the algorithm. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O6Ochnle3FV", "reviewer": "Reviewer_8g3Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper address a significant problem of multi-agent reinforcement learning in the setting when agents are 1) heterogeneous 2) learn/reason about each other and adapt their policies based on observations of other agents' behavior. In the general discussion, the paper covers relevant related work, and discusses challenges and limitations. Empirical evaluations conclude both quantitative evaluation of performance and insights on the possible mechanisms and causes of observed performance patterns, and justifications of the proposed algorithm. This research can potentially become a significant and influential contribution to reinforcement learning community.\n\nThat said, I have several reservations about the paper and would like to see if they can be addressed in a revision. \n\n1) The empirical evaluation is not well aligned with related work. It compares MOBM with other baselines on three domains, one of them is collaborative and two are competitive. It seems that none of these domains appears in the same form in the cited work to which the new algorithm compared. Since this paper claims that SOTA, I would anticipate that the comparison is performed on domains on which the baselines claim SOTA, in addition to the new domains introduced or adapted in the paper.\n\n2) The main idea of the paper is in representation learning for recursive epistemic reasoning. However, convergence of such reasoning, and soundness of Bayesian mixing of such methods is not theoretically analyzed. This is not obvious at all that recursive epistemic reasoning (theory of mind) works in competitive domains, and what are the restrictions/conditions under which it is sound. \n  a) First, recursive epistemic reasoning is applied to competitive tasks. It is easy to show that it does not always converge. Consider Bob and Alice and two bars settings. Bob and Alice slightly prefer bar A, but both what to avoid each other. This is a competitive setting. Let us reason with Alice. On level 0, Alice chooses bar A. On level 1, Alice realizes that Bob chooses bar A too, so she chooses bar B. On level 2, Alice thinks about Bob choosing bar B on level 1 so she chooses bar A. On level 3, she would switch to bar A again, and so on. No convergence. By making Bob's and Alice's policies probabilitistic the analysis can be made more interesting, but still not converging under certain conditions to any equilibrium. This is not an edge case but a general property of recursive reasoning, which does not seem to be addressed in the paper.\n b) Second, recursive reasoning is built upon the best imagined action of the opponent. This is a problematic setting, related to Newcomb's paradox. Consider, again Bob and Alice, in a setting in which Alice has equal preferences regarding the bars and attributes reward 1 to avoiding Bob and 0 to meeting him. Bob chooses the first bar in 55% of cases; Alice employs a single level of epistemic\nreasoning. It is easy to see that the optimal policy for Alice is to always go to the second bar for the expected reward or 0.55.\nHowever, the policy inferred using conditioning on the best anticipated move of the opponent is  is for Alice to choose the second bar with\nprobability of 0.55, for the expected reward of 1! This is because Alice's decision is (erroneously) conditioned on Bob's\nanticipated choice of a bar, and hence Alice pretends that she can always choose the other bar (which she cannot). This issue also seems to be omitted in the paper. The agent behavior may seen reasonable in the complicated settings of the proposed domains, but in general I suspect that the algorithm is not sound.\n\nBased on the above, I would appreciate a formal theoretical treatment and an empirical evaluation to be improved in a revised version of the paper.\n\n", "summary_of_the_paper": "The paper proposes MBOM, a scheme for multi-agent reinforcement learning, which flexibly adapts to opponents with different or unknown learning abilities by relying on recursive reasoning.  The paper evaluates the algorithm on two competitive environment, empirically comparing to other baseline and state-of-the-art schemes.", "summary_of_the_review": "An interesting line of research but theoretical analysis and empirical evaluation should be improved in a revised version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7ruDEi65VJM", "reviewer": "Reviewer_3i5i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method proposed is elegant and seems promising, but I have a number of questions that require clarifications:\n1. the method is designed to work with multiple opponents that are treated as a joint opponent.  In both and the Triangle Game and the One-on-One game e there are only two agents.  This raises the issue of what happens in situation when there is more than one opponent.  It would have been useful to have at least one application with multiple agents to understand how well the proposed algorithm addresses the multi-agent opponent modeling as claimed.\n.2. The two games used for the experimental work seem to have sequential and not simultaneous moves, and the same applies to the cooperative game. It would have been useful to address the issue in the description.  Again, does the algorithm expect a sequential move game?\n3. Not much is said on the opponent types that are assumed, specifically the difference between the naive learner and the reasoning learner.  This is important because the readers need to know how to decide if an opponent is \"learnable\".   Also, it is not clear if the agent knows the type of opponent (fixed policy, etc) or not.  \n4. The computational costs are mentioned as being high, but there is no indication, for instance, of the computing time taken by the examples presented in the paper. The problem of reducing the computational complexity is listed as future work.\n5. There are graphs that show performance results for the games, but it is not clear how performance is measured.  Some explanations would help. The performance tables (Table 2 and Table 3) show numerical performance values but do not say what the values represent.\n6. The placement of tables and figures is not ideal. They are referenced very far form where they show up in the paper, making the reading harder.\n7. The paper does not include any theoretical results, and, given the comments made above, it is hard to understand when the method will work and how well it will work.  ", "summary_of_the_paper": "The paper addresses the problem of learning an opponent model in a multi-agent environment. What makes learning hard is that the opponent can be learning at the same time. The paper proposes an algorithm, called MBOM, that operates by simulating the reasoning process of the agent in the environment and considering different types of opponents. The process is recursive and uses the environment model to predict the policy improvement of the opponents. The process generates improved models of the opponent, that are then used with Bayesian mixing to produce an estimate of the opponents policy. The MBOM algorithm is tested on two games, the Triangle Game and the One-on-one two player game. Different algorithms are used and their performance is compared with MBOM.", "summary_of_the_review": "The paper presents a promising method for an agent to learn opponent models in a multi-agent environment. The method is compared empirically with similar algorithms which are outperformed.  Unfortunately, the paper does not provide theoretical guarantees, lacks details in some of the explanations, as listed above, and shows only examples with two agents, an agent and an opponent.  In conclusion, the method seem promising and well thought out,  but it is hard to be understand what are the requirements to make the method applicable in practice. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the challenging problem of learning against an opponent that may or may not be simultaneously learning as well. The key contribution of this paper is a learning algorithm that accounts for how the opponents may update their policies from past interactions. The proposed algorithm, MBOM, relies on the environment model to model a hierarchy of opponents using different depths of recursive reasoning (from non-learning agents to deep recursive agents). It is agreed that this papers studies an important problem and shows promise. However, the current results aren't convincing enough. In particular, since there is no theoretical analysis, more empirical validation of the method is expected. The current experiments only considers a single opponent, and it is unclear how well the method works given accumulated errors through the recursion. Future submissions would benefit from additional empirical analysis (e.g., ablations) to help understand when and why MBOM works.
Decision: Reject