Year: 2021

Abstract:
When one agent interacts with a multi-agent environment, it is challenging to deal with various opponents unseen before. Modeling the behaviors, goals, or beliefs of opponents could help the agent adjust its policy to adapt to different opponents. In addition, it is also important to consider opponents who are learning simultaneously or capable of reasoning. However, existing work usually tackles only one of the aforementioned types of opponent. In this paper, we propose model-based opponent modeling (MBOM), which employs the environment model to adapt to all kinds of opponent. MBOM simulates the recursive reasoning process in the environment model and imagines a set of improving opponent policies. To effectively and accurately represent the opponent policy, MBOM further mixes the imagined opponent policies according to the similarity with the real behaviors of opponents. Empirically, we show that MBOM achieves more effective adaptation than existing methods in competitive and cooperative environments, respectively with different types of opponents, i.e., fixed policy, naïve learner, and reasoning learner.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) has made great progress in multi-agent competitive games, e.g., AlphaGo (Silver et al., 2016), OpenAI Five (OpenAI, 2018), and AlphaStar (Vinyals et al., 2019). In multi-agent environments, an agent usually has to compete against or cooperate with diverse other agents (collectively termed as opponents whether collaborators or competitors) unseen before. Since the opponent policy influences the transition dynamics experienced by the agent, interacting with diverse opponents makes the environment seem nonstationary from the agent’s perspective. Due to the complexity and diversity in opponent policies, it is very challenging for the agent to retain overall supremacy.
Explicitly modeling the behaviors, goals, or beliefs of opponents (Albrecht & Stone, 2018), rather than treating them as a part of the environment, could help the agent adjust its policy to adapt to different opponents. Many studies rely on predicting the actions (He et al., 2016; Hong et al., 2018; Grover et al., 2018; Papoudakis & Albrecht, 2020) and goals (Rabinowitz et al., 2018; Raileanu et al., 2018) of opponents during training. When facing diverse or unseen opponents, the agent policy conditions on the prediction or representation generated by corresponding modules. However, opponents may also have the same reasoning ability, e.g., an opponent who makes prediction about the agent’s goal. In this scenario, higher-level reasoning and some other modeling techniques are required to handle such sophisticated opponents (Wen et al., 2019; Yang et al., 2019; Yuan et al., 2020). In addition, the opponents may learn simultaneously, and the modeling becomes unstable and the fitted models with historical experiences lag behind. To enable the agent to continuously adapt to learning opponents, LOLA (Foerster et al., 2018a) takes into account the gradients of the opponent’s learning for policy update, Meta-PG (Al-Shedivat et al., 2018) formulates continuous adaptation as a meta-learning problem, and Meta-MAPG (Kim et al., 2021a) combines meta-learning with LOLA. However, LOLA requires knowing the learning algorithm of opponents, while Meta-PG and Meta-MAPG require all opponents use a same learning algorithm.
Unlike existing work, we do not make such assumptions and focus on enabling the agent to learn effectively by directly representing the policy improvement of opponents when interacting with them, even if they may be also capable of reasoning. Inspired from the intuition that human could anticipate the future behaviors of opponents by simulating the interactions in the brain after knowing the rules and mechanics of the environment, in this paper, we propose model-based opponent modeling
(MBOM), which employs the environment model to predict and capture the policy improvement of opponents. By simulating the interactions in the environment model, we could obtain the best responses of opponents under the agent policy conditioned on the opponent model. Then, the opponent model can be fine-tuned using the simulated best responses to get the higher-level opponent model. The higher-level opponent model reasons more deeply and thus more competitive. By recursively repeating the simulation and fine-tuning, MBOM imagines the learning and reasoning of opponents and generates a set of opponent models with different levels, which could also be seen as recursive reasoning. However, since the learning and reasoning of opponents are unknown, a certain-level opponent model might overestimate or underestimate the opponent. To effectively and accurately represent the opponent policy, we further propose to mix the imagined opponent policies according to the similarity with the real behaviors of opponents updated by Bayesian rule.
We evaluate MBOM in the two competitive tasks, Triangle Game and One-on-One, against three types of opponent, e.g., fixed policy, naïve learner, and reasoning learner. MBOM outperforms strong baselines, especially when against naïve learner and reasoning learner, which demonstrates the effectiveness of recursive imagination and Bayesian mixing. We also show the adaptation ability of MBOM in the cooperative task Coin Game.

heading: 2 RELATED WORK

heading: 2.1 OPPONENT MODELING
text: In multi-agent reinforcement learning (MARL), it is a big challenge to form robust policy due to the unknown opponent policy. From the perspective of an agent, if opponents are considered as a part of the environment, the environment is unstable and complex for policy learning when the policies of opponents are also changing. If the information about opponents is included, e.g., behaviors, goals and beliefs, the environment may become stable, and the agent could learn using single-agent RL methods. This line of research is opponent modeling.
One simple idea of opponent modeling is to build a model each time a new opponent or group of opponents is encountered (Zheng et al., 2018). However, learning a model every time is not efficient. A more computationally tractable approach is to represent an opponent’s policy with an embedding vector. Grover et al. (2018) uses a neural network as encoder, taking as input the trajectory of one agent. Imitation learning and contrastive learning are used to train the encoder. Then, the learned encoder can be combined with RL by feeding the generated representation into policy or/and value network. Learning of the model can also be performed simultaneously with RL, as an auxiliary task (Jaderberg et al., 2016). Based on DQN (Mnih et al., 2015), DRON (He et al., 2016) and DPIQN (Hong et al., 2018) use a secondary network which takes observations as input and predicts opponents’ actions. The hidden layer of this network is used by the DQN module to condition on for a better policy. It is also feasible to model opponents using variational autoencoders (Papoudakis & Albrecht, 2020), which means the generated representations are no longer deterministic vectors, but high-dimensional distributions. ToMnet (Rabinowitz et al., 2018) tries to make agents have the same Theory of Mind (Premack & Woodruff, 1978) as humans. ToMnet consists of three networks, reasoning about the agent’s action and goal based on past and current information. SOM (Raileanu et al., 2018) implements Theory of Mind from a different perspective. SOM uses own policy, opponent’s observation and opponent’s action to work backwards to learning opponent’s goal by gradient ascent.
The methods aforementioned only consider opponent policies which are independent of the agent. If opponents hold belief about the agent, the agent can form higher level belief about opponents’ belief. This process can perform recursively, termed recursive reasoning. PR2 (Wen et al., 2019) uses SAC (Haarnoja et al., 2018) to approximate opponents’ policies conditioning on the agent’s action. Since PR2 uses the agent’s Q-function to estimate opponents’ policies, it can only be applied in cooperative environments. Yuan et al. (2020) takes both level-0 and level-1 beliefs as input to the value function. Level-0 belief is updated according to Bayesian rule, and level-1 belief is updated using a learnable neural network. However, these methods use centralized training decentralized execution algorithms to train a set of fixed agents that cannot handle diverse opponents in execution. Bayes-ToMoP (Yang et al., 2019) is developed to detect and handle different reasoning opponents, but the implementation is complex and not scalable.
If the opponents are also learning, the modeling mentioned above becomes unstable and the fitted models with historical experiences lag behind. So it is beneficial to take into consideration the learning process of opponents. LOLA (Foerster et al., 2018a) introduces the impact of the agent’s policy on the anticipated parameter update of the opponent. A neural network is used to model the opponent’s policy and estimate learning gradient of the opponent’s policy, implying that the learning algorithm used by the opponent should be known, otherwise the estimated gradient will be inaccurate. Further, the opponents may still be learning continuously during execution. Meta-PG (Al-Shedivat et al., 2018) is a method based on meta policy gradient, using trajectories come from the current opponents to do multiple meta-gradient steps and construct a policy that is good for the updated opponents. Meta-MAPG (Kim et al., 2021a) extends this method by including an additional term that accounts for the impact of the agent’s current policy on the future policies of opponents, similar to LOLA. These meta-learning based methods require that the distribution of trajectories match across training and test, which implicitly means all opponent use a same learning algorithm.

heading: 2.2 MODEL-BASED RL AND MARL
text: Model-based RL allows the agent to have access to the transition function. There are two typical branches of model-based RL approaches: background planning and decision-time planning. In background planning, the agent could use the learned model to generate additional experiences for assisting learning. For example, Dyna-style algorithms (Sutton, 1990; Kurutach et al., 2018; Luo et al., 2019) perform policy optimization on simulated experiences, and model-augmented value expansion algorithms (Feinberg et al., 2018; Oh et al., 2017; Buckman et al., 2018) use model-based rollouts to improve the update targets. In decision-time planning (Chua et al., 2018), the agent could use the model to rollout the optimal action at a given state by looking forward during execution, e.g., model predictive control.
The model-based MARL methods could be divided into two classes: centralized model and decentralized model. AMLAPN (Park et al., 2019) builds a centralized auxiliary prediction network to model the environment dynamics and the opponent actions to alleviate the non-stationary dynamics. A centralized multi-step generative model (Krupnik et al., 2020) with a disentangled variational auto-encoder has been proposed for performing trajectory planning. For decentralized model, IS (Kim et al., 2021b) uses the model to generate imagined trajectories for multi-agent communication to share intention. HPP (Wang et al., 2020) uses the models to propose and evaluate navigation subgoals for the rendezvous task.
Our method relaxes the limitations on opponent modeling. We make no assumptions about the variation of opponents’ policies. They could be fixed, randomly sampled from an unknowable policy set, or continuously learning using an unknowable and changeable RL algorithm, both in training and execution. We learn an environment model that allows the agent to perform recursive reasoning against opponents who may also have the same reasoning ability.

heading: 3 PRELIMINARIES
text: We consider an n-agent stochastic game (S,A1, . . . ,An,P,R1, . . . ,Rn, γ), where S is the state space, Ai is the action space of agent i ∈ [1, . . . , n], A = ∏n i=1Ai is the joint action space of agents, P : S ×A×S → [0, 1] is a transition function,Ri : S ×A×S → R is the reward function of agent i , and γ is the discount factor. The policy of agent i is πi, and the joint policy of other agents is π−i(a−i|s) = ∏ j 6=i π
j(aj |s), where a−i is the joint action except agent i. All agents interact with the environment simultaneously without communication. The historical trajectory is available, i.e., for agent i at timestep t, {s0, ai0, a−i0 , . . . , st−1, ait−1, a −i t−1} is observable. The goal of the agent i is to maximize its expected cumulative discount rewards
E st+1∼P(·|st,ait,a −i t ),
ai∼πi(·|st),a−it ∼π −i(·|st)
[ ∞∑ t=0 γtRi(st, ait, a−it , st+1) ] . (1)
For convenience, the learning agent treats all other agents as a joint opponent with the joint action ao ∼ πo(·|s) and reward ro. The action and reward of the learning agent are denoted as a ∼ π(·|s) and r, respectively.
𝜙𝑚
𝜑0 𝜑1 𝜑𝑛
Mixer
……
𝜑𝑚𝑖𝑥
𝑎𝑖
𝑎−𝑖
𝑎𝑖
𝜙0 𝜙1 𝜙𝑀
Ψ𝑖
𝑎−𝑖
𝜓0
𝜙𝑚𝑖𝑥
… …
𝑀𝐿𝑃
𝜓𝑖
… …
(3)
(5)
… …
𝜶
(4)
for m=0 to M-1
𝜙𝑚+1
𝜙𝑚
Rollout K steps
𝜋𝑖 ⋅ 𝑠, 𝜙𝑚; 𝜃
𝐸𝑛𝑣𝑖𝑟𝑜𝑛𝑚𝑒𝑛𝑡 𝑚𝑜𝑑𝑒𝑙
𝑎−𝑖
Recursive Imagination
𝜙𝑚s
𝑎−𝑖∗
Mixer
𝜙0 𝜙1 𝜙𝑀
Ψ𝑖
𝑎−𝑖
𝜙𝑚𝑖𝑥
… …
𝑀𝐿𝑃
𝜓𝑖
… …
(3)
(5)
… …
𝜶
(4)
𝜋 𝑎−𝑖~𝜌(∙|𝑠;𝜙𝑚)
(∙ |𝑠, 𝑎−𝑖; 𝜃)
𝜋 𝑎−𝑖~𝜌(∙|𝑠;𝜙𝑚𝑖𝑥)
(∙ |𝑠, 𝑎−𝑖; 𝜃)
𝜙𝑚
𝑀𝐿𝑃
Ψ ψ
𝜓0 𝜓1
𝜓𝑀

heading: 4 MODEL-BASED OPPONENT MODELING
text: MBOM employs the environment model to predict and capture the learning of opponent policy. By simulating recursive reasoning via the environment model, MBOM imagines the learning and reasoning of the opponent and generates a set of opponent models. To obtain stronger representation ability and accurately capture the adaptation of the opponent, MBOM mixes the imagined opponent policies according to the similarity with the real behaviors of opponent.

heading: 4.1 RECURSIVE IMAGINATION
text: If the opponent is also learning during interaction, the opponent model fitted with historical experiences always lag behind, making the agent hard to adapt to the opponent. Moreover, if the opponent could adjust its policy according to the actions, intentions, or goals of the agent, then recursive reasoning may occur between agent and opponent. However, based on the lagged opponent model, the agent would struggle to keep up with the learning of opponent. To adapt to the learning and reasoning opponent, the agent should predict the current opponent policy and reason more deeply than the opponent.
MBOM explicitly simulates the recursive reasoning process utilizing the environment model, called recursive imagination, to generate a series of opponent models, called Imagined Opponent Policies (IOPs). Initially, the agent interacts with η different opponents which are learning with PPO (Schulman et al., 2017), and collect a buffer D which contains the experience 〈s, a, ao, s′, r〉, where r = 〈r, ro〉. For zero-sum game (r + ro = 0) and fully cooperative game (r = ro), ro can be easily obtained, while for general-sum game we make a mild assumption that ro can be accessed during experience collection. Using the experience buffer D, we can train the the environment model Γ(s′, r|s, a, ao; ζ) by minimizing the mean square error
E s,a,ao,s′,r∼D
1 2 ‖s′ − ŝ′‖2 + 1 2 ‖r − r̂‖2, (2)
where ŝ′, r̂ = Γ(s, a, ao; ζ), obtain the level-0 IOP π̃o(·|s;φ0) by maximum-likelihood estimation
E s,ao∼D
log π̃o(ao|s;φ0), (3)
and learn the policy of the agent π(·|s, ao; θ) using PPO. By running PPO, we could also get the value function V (s) of the agent. To imagine the learning of the opponent, as shown in Figure 1, we use the rollout algorithm (Tesauro & Galperin, 1996) to get the best response of the opponent under the agent policy π. For each opponent action aot at timestep t, we uniformly sample the opponent action sequences in the next k timesteps, simulate the trajectories using the learned environment
Algorithm 1 MBOM 1: Pretraining: 2: Initialize the recursive imagination layer M . 3: Initialize the weights α with (1, 0, . . . , 0). 4: Interact with η learning opponents and collect the experience buffer D. 5: Train the level-0 IOP φ0, the environment model Γζ , and the agent policy θ using the experience
buffer D. 6: Interaction: 7: for t = 1, . . . , max_epoch do 8: Interact with the opponent to observe the real opponent actions. {//Recursive Imagination} 9: Fine-tune φo with the real opponent actions
10: for i = 1, . . . ,M − 1 do 11: Rollout in Γζ with π(·|s, ao ∼ π̃o(·|s;φi−1); θ) to get the best responses of the opponent ao∗ by equation 4 or equation 5. 12: Fine-tune φi−1 with the best responses to get the φi. 13: end for {//Bayesian Mixing} 14: Update α by equation 7. 15: Mix the IOPs to get π̃omix by equation 6. 16: end for
model Γζ , and select the best response with the highest rollout value
ao∗t = argmax aot max aot+1,··· ,a o t+k∼Unif k∑ j=0 γt+jrot+j . (4)
During the simulation, the agent acts according to the policy conditioned on the modeled opponent policy, at ∼ π(·|st, aot ∼ π̃o(·|st;φ0); θ), and the learned environment model provides the transition st+1, rt = Γ(st, at, a o t ; ζ). With larger k, the rollout has longer planning horizon, and thus could evaluate the action ao∗ more accurately, assuming a perfect environmental model. However, the computation cost of rollout increases exponentially with the planning horizon to get an accurate estimate of ao∗, while in practice the compounding error of the environmental model also increases with the planning horizon. Therefore, the choice of k is a trade-off between accuracy and cost. Specifically, for zero-sum game and fully cooperative game, we can approximately estimate the opponent state value V o(s) as −V (s) and V (s), respectively, and modify the rollout value like n-step return (Sutton & Barto, 2018) to obtain a longer horizon
ao∗t = argmax aot max aot+1,··· ,a o t+k∼Unif k∑ j=0 γt+jrot+j + γ t+k+1V o(st+k+1). (5)
By imagination, we can obtain the best response of the opponent under the agent policy π and level0 IOP π̃o(·|s;φ0) and construct the simulated data {〈s, ao∗〉}. Then, we use the data to fine-tune the level-0 IOP π̃o(·|s;φ0) by maximum-likelihood estimation, and obtain the level-1 IOP π̃o(·|s;φ1). The level-1 IOP can be see as the best response of agent to the response of opponent to level-0 IOP. In the imagination, it is the nested form as “the opponent believes [that the agent believes (that the opponent believes ...)].” The existing imagined opponent policy is the innermost “(that the opponent believes),” while the outermost “the opponent believes” is ao∗ which is obtained by rollout process. Recursively repeating the rollout and fine-tuning, where the agent policy is conditioned on the IOP π̃o(·|s;φm−1), we could derive the level-m IOP π̃o(·|s;φm). The higher-level IOP reasons more deeply, and thus more competitive.

heading: 4.2 BAYESIAN MIXING
text: By recursive imagination, we get M IOPs with different reasoning levels. However, since the learning and reasoning of the opponent are unknown, a single IOP might overestimate or underestimate the opponent. To obtain stronger representation ability and accurately capture the learning of the opponent, as illustrated in Figure 1, we linearly combine the IOPs to get a mixed policy, similar idea
was also present in (Wen et al., 2020)
π̃omix(·|s) = M−1∑ i=0 αiπ̃ o(·|s;φi), (6)
where αi is the weight of level-i IOP, which is produced by the IOPs mixer,
α = (α0, . . . , αM−1) = softer-softmax(Ψ0, . . . ,ΨM−1). (7)
Softer-softmax (Hinton et al., 2015) is a variant of softmax, which uses higher temperature to control a softy of the probability distribution over classes. Ψm is the decayed moving average of p(m|ao), which is the probability of using the level-m IOP given the action of the opponent ao. By Bayesian rule, we have
p(m|ao) = p(a o|m)p(m)∑M−1
i=0 p(a o|i)p(i) = π̃o(ao|s;φm)p(m)∑M−1 i=0 [π̃ o(ao|s;φi)p(i)] , (8)
where p(m) is the probability of using the level-m IOP, and we estimate it as the moving average of p(m|ao). Ψm indicates the similarity between the level-m IOP and the opponent in the most recent stage. Given the opponent actions, the higher Ψm means that the actions are more likely to be generated from the level-m IOP, and thus the level-m IOP is more similar to the opponent. Adjusting the weights α according to the similarity could obtain a more accurate estimate of the improving opponent policy. Moreover, the IOPs mixer is non-parametric, which could be updated quickly and efficiently without parameter training and too many interactions. Therefore, the IOPs mixer could adapt to the fast-improving opponent.
For completeness, the whole procedure of MBOM in given in Algorithm 1.

heading: 5 EXPERIMENTS

heading: 5.1 SETTING UP
text: We evaluate MBOM in two competitive environments.
Triangle Game is an asymmetric zero-sum game on two-dimension with multi-step actions, implemented on Multi-Agent Particle Environment (Mordatch & Abbeel, 2017; Lowe et al., 2017) (MIT license). As shown in Figure 2(a), there are two moving players, P1 and P2, and three fixed landmarks, L1-L3, in a square field. The landmarks are located at the three vertexes of an equilateral triangle with the side length 0.6. When the distance between a player and a landmark is less than 0.15, the agent touches the landmark and has the state T. T1 indicates that the player touches the landmark L1, and so on. If the player does not touch any landmark, the player state is F. The payoff matrix of the two players is shown in Table 2(a). P2 has inherent disadvantages since the optimal solution of P2 always strictly depends on the state of P1. When facing different policies of P1, P2 has to adjust its policy to adapt to P1 for higher reward. We control P2 as the agent and take P1 as the opponent.
One-on-One is a two-player competitive game implemented on Google Research Football Environment (Kurach et al., 2020) (Apache-2.0 License), a physics-based 3D simulator. As shown in
Figure 2(b), there are two players, the shooter, which could dribble and shoot the ball, and the goalkeeper. The shooter controls the ball in the initial state. The episode ends after 30 timesteps or when the shooter loses possession of the ball. At the end of an episode, if the shooter shoots the ball into the goal, the shooter will get a reward +1, and the goalkeeper will get a reward −1. Otherwise, the shooter will get a reward of −1, and the goalkeeper will get a reward of +1. The goalkeeper could only passively react to the strategies of the shooter and makes policy adaptation when the shooter strategy changes. We control the goalkeeper as the agent and take the shooter as the opponent.
Baselines. In the experiments, we compare MBOM with the following methods: • LOLA-DiCE (Foerster et al., 2018b) is an expansion of the LOLA, which uses Differ-
entiable Monte-Carlo Estimator (DiCE) operation to considers how to shape the learning dynamics of other agents.
• Meta-PG (Al-Shedivat et al., 2018) uses trajectories come from the current opponents to do multiple meta-gradient steps and construct a policy that is good for the updated opponents.
• Meta-MAPG (Kim et al., 2021a) includes an additional term that accounts for the impact of the agent’s current policy on the future policies of opponents, compared with Meta-PG.
• PPO (Schulman et al., 2017) is a classical single agent RL algorithm, without any other modules.
• MBOM w/o IOPs always uses φ0 as the opponent model, without recursive imagination and Bayesian mixing.
The baselines have the same neural network architectures as MBOM. All the models are trained for five runs with different random seeds. All the curves are plotted using mean and standard deviation. More details about experimental settings and hyperparameters are available in Appendix A.
Opponents. For the opponents, we run independent PPO (Schulman et al., 2017) algorithms for 10 times. During each running, we store 20 opponent policies in the training set, 3 opponent policies in the validation set, and 3 opponent policies in the test set. So the sizes of the training set, validation set, and test set are 200, 30, and 30. We pre-train the agent with the opponents in the training set and make the agent interact with the opponents in the test set to evaluate the ability of generalization and adaptation. The validation set is only required by Meta-PG and Meta-MAPG. We construct three types of opponents:
• Fixed policy. The opponents in the test set will not update during the interaction. • Naïve learner. The opponents in the test set will be updated with PPO during the interac-
tion. • Reasoning learner. The opponents could model the behavior of the agent and will be
fine-tuned during the interaction.
To increase the diversity of the opponents, we adopt the reward shaping technique and add invisible barriers in the environment during the training.

heading: 5.2 PERFORMANCE
text: The experimental results against test opponents are shown in Figure 3, and the mean performance with standard deviation over all test opponents is summarized in Table 2 and 3. In Triangle Game, the opponent could take different strategies, e.g., hovering around a landmark, commuting between two landmarks, or rotating among three landmarks, whereas the agent has to anticipate and adapt to the opponent strategies. The learning of LOLA-DiCE depends on the gradient of the opponent
model. However, the gradient information cannot clearly reflect the distinctions between diverse opponents, which leads to that LOLA-DiCE cannot adapt to the unseen opponent quickly and effectively. Meta-PG and Meta-MAPG show stronger adaptation abilities than LOLA-DiCE when facing different opponents, but fail on some opponents (index 0, 1, and 2 in Figure 3(a), 3(c), and 3(e)). By visualization, we find that those opponents are hovering around one landmark, which are out of the training set. Since Meta-PG and Meta-MAPG assume that the test set follows the distribution of the training set, these two methods are unable to adapt to the out-of-distribution opponents. MBOM w/o IOPs obtains similar results to MBOM when facing fixed policy opponents because φ0 could accurately predict the opponent behaviors if the opponent is fixed. However, if the opponent is learning and reasoning, φ0 cannot accurately represent the opponent, and thus the performance of MBOM w/o IOPs drops. By explicitly simulating the recursive reasoning process, MBOM shows more obvious performance gain against naïve learner and reasoning learner, as in Table 2. In Oneon-One, the shooter has absolute dominance and more flexible strategy choices, including shooting location, shooting angle, and shooting time. Due to the limitation of gradient information, LOLADiCE still shows poor adaptability. Meta-PG and Meta-MAPG heavily rely on the reward signal, and thus adaptation is difficult for the two methods in this sparse reward task. MBOM w/o IOPs and MBOM achieve similar performance with fixed policy, but the performance of MBOM significantly improves if the opponent is improving as shown in Table 3, which is contributed to the recursive reasoning capability given by recursive imagination in the environment model and Bayesian mixing that quickly captures the learning of opponent.

heading: 5.3 COOPERATIVE TASK
text: MBOM could also be applied to cooperative tasks. We test the performance of MBOM on a cooperative scenario, Coin Game, which is a high-dimension expansion of the iterated prisoner dilemma
with multi-step actions (Lerer & Peysakhovich, 2018; Foerster et al., 2018a). There are two players, red and blue, moving on a 3× 3 grid field, and two types of coins, red and blue, randomly generated on the grid field. If the player moves to the position of the coin, the player collects the coin and receives a reward of +1. However, if the color of the collected coin is different from the player’s color, the other player receives a reward of −2, as illustrated in Figure 4(a). The length of the game is 150 timesteps. Both agents simultaneously update their policies using the same MBOM or other baselines to maximize the sum of rewards.
The experiment results are shown in Figure 4(b). Meta-PG and Meta-MAPG degenerate to Policy Gradient for this task as there is no training set. Both learn a greedy strategy that collecting any color coin, which leads to total score of two players is zero. LOLA-DiCE learns too slow and does not learn to cooperate within 300 iterations, indicating the inefficiency of estimating the opponent gradients. MBOM w/o IOPs and MBOM learn to cooperate quickly and successfully, and MBOM shows smaller standard deviation than MBOM w/o IOPs, indicating that MBOM can also be applied to cooperative tasks without negative effect.

heading: 6 CONCLUSION AND FUTURE WORK
text: We have proposed model-based opponent modeling. MBOM employs recursive imagination and Bayesian mixing to predict and capture the learning and improvement of opponents. Empirically,
we evaluated MBOM in two competitive environments, and demonstrated MBOM adapts to learning and reasoning opponents much better than the baselines. These make MBOM a simple and effective RL method whether opponents be fixed, continuously learning, or reasoning in competitive environments. Moreover, we also verified the adaptation ability of MBOM in cooperative environments.
In MBOM, the learning agent treats all opponents as a joint opponent. If the size of the joint opponent is large, the agent will need a lot of rollouts to get an accurate best response. The cost increases dramatically with the size of the joint opponent. How to reduce the computation overhead in such scenarios will be considered in future work. Moreover, MBOM implicitly assumes that the relationship between opponents is fully cooperative. Dealing with the case where their relationship is non-cooperative is also left as future work.

heading: A HYPERPARAMETERS
text: The hyperparameters of MBOM are summarized in Table 4.

heading: B ALPHA WEIGHTS ANALYSIS
text: Figure 5 visualizes α of IOPs when the agent against different types of opponents, i.e., fixed policy, naïve learner, and reasoning learner in Triangle Game and One-on-One. With the three opponent types, α2 (the weight of level-2 IOP) is remarkably higher than others, which indicates that recursive imagination of MBOM does learn a pervasive policy of the opponent by the environmental model.

Review summaries:
The paper addresses the problem of learning an opponent model in a multi-agent environment. What makes learning hard is that the opponent can be learning at the same time. The paper proposes an algorithm, called MBOM, that operates by simulating the reasoning process of the agent in the environment and considering different types of opponents. The process is recursive and uses the environment model to predict the policy improvement of the opponents. The process generates improved models of the opponent, that are then used with Bayesian mixing to produce an estimate of the opponents policy. The MBOM algorithm is tested on two games, the Triangle Game and the One-on-one two player game. Different algorithms are used and their performance is compared with MBOM.

This paper proposes a model-based opponent recursive reasoning method for MARL problem. The recursive reasoning takes multiple levels of estimation to generate a number of opponents. The first level opponent is obtained by supervised learning from the true opponent actions. Then, with a learned dynamics model, the level-0 opponent interacts with self agent and the dynamics model to generate trajectories, from which by selecting one best response rollout, a better action at the current time step for the opponent is kept and used to estimate the next level opponent, by maximum log-likelihood again. After obtaining a number of levels of opponents, a mixture opponent is learned by softer-softmax weights. Experiments are conducted on multi-agent particle environment and google's football environment, and LOLA-DiCE and Meta-PG are compared.

Agents learning in systems with other-agents that may be simultaneously learning presents the original agent with a dynamic learning problem. Critically, the agent does not know if the other-agents are non-learning, learning, or learning using recursive-reasoning. One such solution to this problem is to design a learning algorithm that accounts for how the other-agents may update their policy from the shared experiences, effectively eliminating the aforementioned problem. This work attempts to design such an algorithm without relying on any assumptions on the other-agent's learning algorithm. Their proposed algorithm, Model-Based Opponent Modeling (MBOM), learns parallel opponent models that are trained using various depths of recursive reasoning. Then, when played against an opponent, the agent can select the most likely opponent model (level-0: non-learning opponent, level-1: learning-opponent, level-k: k-1 level recursivly learning opponent). This algorithm assumes a centralized training and decentralized execution setting for two-player fully-observable general-sum games.