Model-Based Opponent Modeling

1 INTRODUCTION . Reinforcement learning ( RL ) has made great progress in multi-agent competitive games , e.g. , AlphaGo ( Silver et al. , 2016 ) , OpenAI Five ( OpenAI , 2018 ) , and AlphaStar ( Vinyals et al. , 2019 ) . In multi-agent environments , an agent usually has to compete against or cooperate with diverse other agents ( collectively termed as opponents whether collaborators or competitors ) unseen before . Since the opponent policy influences the transition dynamics experienced by the agent , interacting with diverse opponents makes the environment seem nonstationary from the agent ’ s perspective . Due to the complexity and diversity in opponent policies , it is very challenging for the agent to retain overall supremacy . Explicitly modeling the behaviors , goals , or beliefs of opponents ( Albrecht & Stone , 2018 ) , rather than treating them as a part of the environment , could help the agent adjust its policy to adapt to different opponents . Many studies rely on predicting the actions ( He et al. , 2016 ; Hong et al. , 2018 ; Grover et al. , 2018 ; Papoudakis & Albrecht , 2020 ) and goals ( Rabinowitz et al. , 2018 ; Raileanu et al. , 2018 ) of opponents during training . When facing diverse or unseen opponents , the agent policy conditions on the prediction or representation generated by corresponding modules . However , opponents may also have the same reasoning ability , e.g. , an opponent who makes prediction about the agent ’ s goal . In this scenario , higher-level reasoning and some other modeling techniques are required to handle such sophisticated opponents ( Wen et al. , 2019 ; Yang et al. , 2019 ; Yuan et al. , 2020 ) . In addition , the opponents may learn simultaneously , and the modeling becomes unstable and the fitted models with historical experiences lag behind . To enable the agent to continuously adapt to learning opponents , LOLA ( Foerster et al. , 2018a ) takes into account the gradients of the opponent ’ s learning for policy update , Meta-PG ( Al-Shedivat et al. , 2018 ) formulates continuous adaptation as a meta-learning problem , and Meta-MAPG ( Kim et al. , 2021a ) combines meta-learning with LOLA . However , LOLA requires knowing the learning algorithm of opponents , while Meta-PG and Meta-MAPG require all opponents use a same learning algorithm . Unlike existing work , we do not make such assumptions and focus on enabling the agent to learn effectively by directly representing the policy improvement of opponents when interacting with them , even if they may be also capable of reasoning . Inspired from the intuition that human could anticipate the future behaviors of opponents by simulating the interactions in the brain after knowing the rules and mechanics of the environment , in this paper , we propose model-based opponent modeling ( MBOM ) , which employs the environment model to predict and capture the policy improvement of opponents . By simulating the interactions in the environment model , we could obtain the best responses of opponents under the agent policy conditioned on the opponent model . Then , the opponent model can be fine-tuned using the simulated best responses to get the higher-level opponent model . The higher-level opponent model reasons more deeply and thus more competitive . By recursively repeating the simulation and fine-tuning , MBOM imagines the learning and reasoning of opponents and generates a set of opponent models with different levels , which could also be seen as recursive reasoning . However , since the learning and reasoning of opponents are unknown , a certain-level opponent model might overestimate or underestimate the opponent . To effectively and accurately represent the opponent policy , we further propose to mix the imagined opponent policies according to the similarity with the real behaviors of opponents updated by Bayesian rule . We evaluate MBOM in the two competitive tasks , Triangle Game and One-on-One , against three types of opponent , e.g. , fixed policy , naïve learner , and reasoning learner . MBOM outperforms strong baselines , especially when against naïve learner and reasoning learner , which demonstrates the effectiveness of recursive imagination and Bayesian mixing . We also show the adaptation ability of MBOM in the cooperative task Coin Game . 2 RELATED WORK . 2.1 OPPONENT MODELING . In multi-agent reinforcement learning ( MARL ) , it is a big challenge to form robust policy due to the unknown opponent policy . From the perspective of an agent , if opponents are considered as a part of the environment , the environment is unstable and complex for policy learning when the policies of opponents are also changing . If the information about opponents is included , e.g. , behaviors , goals and beliefs , the environment may become stable , and the agent could learn using single-agent RL methods . This line of research is opponent modeling . One simple idea of opponent modeling is to build a model each time a new opponent or group of opponents is encountered ( Zheng et al. , 2018 ) . However , learning a model every time is not efficient . A more computationally tractable approach is to represent an opponent ’ s policy with an embedding vector . Grover et al . ( 2018 ) uses a neural network as encoder , taking as input the trajectory of one agent . Imitation learning and contrastive learning are used to train the encoder . Then , the learned encoder can be combined with RL by feeding the generated representation into policy or/and value network . Learning of the model can also be performed simultaneously with RL , as an auxiliary task ( Jaderberg et al. , 2016 ) . Based on DQN ( Mnih et al. , 2015 ) , DRON ( He et al. , 2016 ) and DPIQN ( Hong et al. , 2018 ) use a secondary network which takes observations as input and predicts opponents ’ actions . The hidden layer of this network is used by the DQN module to condition on for a better policy . It is also feasible to model opponents using variational autoencoders ( Papoudakis & Albrecht , 2020 ) , which means the generated representations are no longer deterministic vectors , but high-dimensional distributions . ToMnet ( Rabinowitz et al. , 2018 ) tries to make agents have the same Theory of Mind ( Premack & Woodruff , 1978 ) as humans . ToMnet consists of three networks , reasoning about the agent ’ s action and goal based on past and current information . SOM ( Raileanu et al. , 2018 ) implements Theory of Mind from a different perspective . SOM uses own policy , opponent ’ s observation and opponent ’ s action to work backwards to learning opponent ’ s goal by gradient ascent . The methods aforementioned only consider opponent policies which are independent of the agent . If opponents hold belief about the agent , the agent can form higher level belief about opponents ’ belief . This process can perform recursively , termed recursive reasoning . PR2 ( Wen et al. , 2019 ) uses SAC ( Haarnoja et al. , 2018 ) to approximate opponents ’ policies conditioning on the agent ’ s action . Since PR2 uses the agent ’ s Q-function to estimate opponents ’ policies , it can only be applied in cooperative environments . Yuan et al . ( 2020 ) takes both level-0 and level-1 beliefs as input to the value function . Level-0 belief is updated according to Bayesian rule , and level-1 belief is updated using a learnable neural network . However , these methods use centralized training decentralized execution algorithms to train a set of fixed agents that can not handle diverse opponents in execution . Bayes-ToMoP ( Yang et al. , 2019 ) is developed to detect and handle different reasoning opponents , but the implementation is complex and not scalable . If the opponents are also learning , the modeling mentioned above becomes unstable and the fitted models with historical experiences lag behind . So it is beneficial to take into consideration the learning process of opponents . LOLA ( Foerster et al. , 2018a ) introduces the impact of the agent ’ s policy on the anticipated parameter update of the opponent . A neural network is used to model the opponent ’ s policy and estimate learning gradient of the opponent ’ s policy , implying that the learning algorithm used by the opponent should be known , otherwise the estimated gradient will be inaccurate . Further , the opponents may still be learning continuously during execution . Meta-PG ( Al-Shedivat et al. , 2018 ) is a method based on meta policy gradient , using trajectories come from the current opponents to do multiple meta-gradient steps and construct a policy that is good for the updated opponents . Meta-MAPG ( Kim et al. , 2021a ) extends this method by including an additional term that accounts for the impact of the agent ’ s current policy on the future policies of opponents , similar to LOLA . These meta-learning based methods require that the distribution of trajectories match across training and test , which implicitly means all opponent use a same learning algorithm . 2.2 MODEL-BASED RL AND MARL . Model-based RL allows the agent to have access to the transition function . There are two typical branches of model-based RL approaches : background planning and decision-time planning . In background planning , the agent could use the learned model to generate additional experiences for assisting learning . For example , Dyna-style algorithms ( Sutton , 1990 ; Kurutach et al. , 2018 ; Luo et al. , 2019 ) perform policy optimization on simulated experiences , and model-augmented value expansion algorithms ( Feinberg et al. , 2018 ; Oh et al. , 2017 ; Buckman et al. , 2018 ) use model-based rollouts to improve the update targets . In decision-time planning ( Chua et al. , 2018 ) , the agent could use the model to rollout the optimal action at a given state by looking forward during execution , e.g. , model predictive control . The model-based MARL methods could be divided into two classes : centralized model and decentralized model . AMLAPN ( Park et al. , 2019 ) builds a centralized auxiliary prediction network to model the environment dynamics and the opponent actions to alleviate the non-stationary dynamics . A centralized multi-step generative model ( Krupnik et al. , 2020 ) with a disentangled variational auto-encoder has been proposed for performing trajectory planning . For decentralized model , IS ( Kim et al. , 2021b ) uses the model to generate imagined trajectories for multi-agent communication to share intention . HPP ( Wang et al. , 2020 ) uses the models to propose and evaluate navigation subgoals for the rendezvous task . Our method relaxes the limitations on opponent modeling . We make no assumptions about the variation of opponents ’ policies . They could be fixed , randomly sampled from an unknowable policy set , or continuously learning using an unknowable and changeable RL algorithm , both in training and execution . We learn an environment model that allows the agent to perform recursive reasoning against opponents who may also have the same reasoning ability . 3 PRELIMINARIES . We consider an n-agent stochastic game ( S , A1 , . . . , An , P , R1 , . . . , Rn , γ ) , where S is the state space , Ai is the action space of agent i ∈ [ 1 , . . . , n ] , A = ∏n i=1Ai is the joint action space of agents , P : S ×A×S → [ 0 , 1 ] is a transition function , Ri : S ×A×S → R is the reward function of agent i , and γ is the discount factor . The policy of agent i is πi , and the joint policy of other agents is π−i ( a−i|s ) = ∏ j 6=i π j ( aj |s ) , where a−i is the joint action except agent i . All agents interact with the environment simultaneously without communication . The historical trajectory is available , i.e. , for agent i at timestep t , { s0 , ai0 , a−i0 , . . . , st−1 , ait−1 , a −i t−1 } is observable . The goal of the agent i is to maximize its expected cumulative discount rewards E st+1∼P ( ·|st , ait , a −i t ) , ai∼πi ( ·|st ) , a−it ∼π −i ( ·|st ) [ ∞∑ t=0 γtRi ( st , ait , a−it , st+1 ) ] . ( 1 ) For convenience , the learning agent treats all other agents as a joint opponent with the joint action ao ∼ πo ( ·|s ) and reward ro . The action and reward of the learning agent are denoted as a ∼ π ( ·|s ) and r , respectively . 𝜙𝑚 𝜑0 𝜑1 𝜑𝑛 Mixer …… 𝜑𝑚𝑖𝑥 𝑎𝑖 𝑎−𝑖 𝑎𝑖 𝜙0 𝜙1 𝜙𝑀 Ψ𝑖 𝑎−𝑖 𝜓0 𝜙𝑚𝑖𝑥 … … 𝑀𝐿𝑃 𝜓𝑖 … … ( 3 ) ( 5 ) … … 𝜶 ( 4 ) for m=0 to M-1 𝜙𝑚+1 𝜙𝑚 Rollout K steps 𝜋𝑖 ⋅ 𝑠 , 𝜙𝑚 ; 𝜃 𝐸𝑛𝑣𝑖𝑟𝑜𝑛𝑚𝑒𝑛𝑡 𝑚𝑜𝑑𝑒𝑙 𝑎−𝑖 Recursive Imagination 𝜙𝑚s 𝑎−𝑖∗ Mixer 𝜙0 𝜙1 𝜙𝑀 Ψ𝑖 𝑎−𝑖 𝜙𝑚𝑖𝑥 … … 𝑀𝐿𝑃 𝜓𝑖 … … ( 3 ) ( 5 ) … … 𝜶 ( 4 ) 𝜋 𝑎−𝑖~𝜌 ( ∙|𝑠 ; 𝜙𝑚 ) ( ∙ |𝑠 , 𝑎−𝑖 ; 𝜃 ) 𝜋 𝑎−𝑖~𝜌 ( ∙|𝑠 ; 𝜙𝑚𝑖𝑥 ) ( ∙ |𝑠 , 𝑎−𝑖 ; 𝜃 ) 𝜙𝑚 𝑀𝐿𝑃 Ψ ψ 𝜓0 𝜓1 𝜓𝑀

Summary: This paper proposes a model-based opponent recursive reasoning method for MARL problem. The recursive reasoning takes multiple levels of estimation to generate a number of opponents. The first level opponent is obtained by supervised learning from the true opponent actions. Then, with a learned dynamics model, the level-0 opponent interacts with self agent and the dynamics model to generate trajectories, from which by selecting one best response rollout, a better action at the current time step for the opponent is kept and used to estimate the next level opponent, by maximum log-likelihood again. After obtaining a number of levels of opponents, a mixture opponent is learned by softer-softmax weights. Experiments are conducted on multi-agent particle environment and google's football environment, and LOLA-DiCE and Meta-PG are compared.