Model-Based Opponent Modeling

1 INTRODUCTION . Reinforcement learning ( RL ) has made great progress in multi-agent competitive games , e.g. , AlphaGo ( Silver et al. , 2016 ) , OpenAI Five ( OpenAI , 2018 ) , and AlphaStar ( Vinyals et al. , 2019 ) . In multi-agent environments , an agent usually has to compete against or cooperate with diverse other agents ( collectively termed as opponents whether collaborators or competitors ) unseen before . Since the opponent policy influences the transition dynamics experienced by the agent , interacting with diverse opponents makes the environment seem nonstationary from the agent ’ s perspective . Due to the complexity and diversity in opponent policies , it is very challenging for the agent to retain overall supremacy . Explicitly modeling the behaviors , goals , or beliefs of opponents ( Albrecht & Stone , 2018 ) , rather than treating them as a part of the environment , could help the agent adjust its policy to adapt to different opponents . Many studies rely on predicting the actions ( He et al. , 2016 ; Hong et al. , 2018 ; Grover et al. , 2018 ; Papoudakis & Albrecht , 2020 ) and goals ( Rabinowitz et al. , 2018 ; Raileanu et al. , 2018 ) of opponents during training . When facing diverse or unseen opponents , the agent policy conditions on the prediction or representation generated by corresponding modules . However , opponents may also have the same reasoning ability , e.g. , an opponent who makes prediction about the agent ’ s goal . In this scenario , higher-level reasoning and some other modeling techniques are required to handle such sophisticated opponents ( Wen et al. , 2019 ; Yang et al. , 2019 ; Yuan et al. , 2020 ) . In addition , the opponents may learn simultaneously , and the modeling becomes unstable and the fitted models with historical experiences lag behind . To enable the agent to continuously adapt to learning opponents , LOLA ( Foerster et al. , 2018a ) takes into account the gradients of the opponent ’ s learning for policy update , Meta-PG ( Al-Shedivat et al. , 2018 ) formulates continuous adaptation as a meta-learning problem , and Meta-MAPG ( Kim et al. , 2021a ) combines meta-learning with LOLA . However , LOLA requires knowing the learning algorithm of opponents , while Meta-PG and Meta-MAPG require all opponents use a same learning algorithm . Unlike existing work , we do not make such assumptions and focus on enabling the agent to learn effectively by directly representing the policy improvement of opponents when interacting with them , even if they may be also capable of reasoning . Inspired from the intuition that human could anticipate the future behaviors of opponents by simulating the interactions in the brain after knowing the rules and mechanics of the environment , in this paper , we propose model-based opponent modeling ( MBOM ) , which employs the environment model to predict and capture the policy improvement of opponents . By simulating the interactions in the environment model , we could obtain the best responses of opponents under the agent policy conditioned on the opponent model . Then , the opponent model can be fine-tuned using the simulated best responses to get the higher-level opponent model . The higher-level opponent model reasons more deeply and thus more competitive . By recursively repeating the simulation and fine-tuning , MBOM imagines the learning and reasoning of opponents and generates a set of opponent models with different levels , which could also be seen as recursive reasoning . However , since the learning and reasoning of opponents are unknown , a certain-level opponent model might overestimate or underestimate the opponent . To effectively and accurately represent the opponent policy , we further propose to mix the imagined opponent policies according to the similarity with the real behaviors of opponents updated by Bayesian rule . We evaluate MBOM in the two competitive tasks , Triangle Game and One-on-One , against three types of opponent , e.g. , fixed policy , naïve learner , and reasoning learner . MBOM outperforms strong baselines , especially when against naïve learner and reasoning learner , which demonstrates the effectiveness of recursive imagination and Bayesian mixing . We also show the adaptation ability of MBOM in the cooperative task Coin Game . 2 RELATED WORK . 2.1 OPPONENT MODELING . In multi-agent reinforcement learning ( MARL ) , it is a big challenge to form robust policy due to the unknown opponent policy . From the perspective of an agent , if opponents are considered as a part of the environment , the environment is unstable and complex for policy learning when the policies of opponents are also changing . If the information about opponents is included , e.g. , behaviors , goals and beliefs , the environment may become stable , and the agent could learn using single-agent RL methods . This line of research is opponent modeling . One simple idea of opponent modeling is to build a model each time a new opponent or group of opponents is encountered ( Zheng et al. , 2018 ) . However , learning a model every time is not efficient . A more computationally tractable approach is to represent an opponent ’ s policy with an embedding vector . Grover et al . ( 2018 ) uses a neural network as encoder , taking as input the trajectory of one agent . Imitation learning and contrastive learning are used to train the encoder . Then , the learned encoder can be combined with RL by feeding the generated representation into policy or/and value network . Learning of the model can also be performed simultaneously with RL , as an auxiliary task ( Jaderberg et al. , 2016 ) . Based on DQN ( Mnih et al. , 2015 ) , DRON ( He et al. , 2016 ) and DPIQN ( Hong et al. , 2018 ) use a secondary network which takes observations as input and predicts opponents ’ actions . The hidden layer of this network is used by the DQN module to condition on for a better policy . It is also feasible to model opponents using variational autoencoders ( Papoudakis & Albrecht , 2020 ) , which means the generated representations are no longer deterministic vectors , but high-dimensional distributions . ToMnet ( Rabinowitz et al. , 2018 ) tries to make agents have the same Theory of Mind ( Premack & Woodruff , 1978 ) as humans . ToMnet consists of three networks , reasoning about the agent ’ s action and goal based on past and current information . SOM ( Raileanu et al. , 2018 ) implements Theory of Mind from a different perspective . SOM uses own policy , opponent ’ s observation and opponent ’ s action to work backwards to learning opponent ’ s goal by gradient ascent . The methods aforementioned only consider opponent policies which are independent of the agent . If opponents hold belief about the agent , the agent can form higher level belief about opponents ’ belief . This process can perform recursively , termed recursive reasoning . PR2 ( Wen et al. , 2019 ) uses SAC ( Haarnoja et al. , 2018 ) to approximate opponents ’ policies conditioning on the agent ’ s action . Since PR2 uses the agent ’ s Q-function to estimate opponents ’ policies , it can only be applied in cooperative environments . Yuan et al . ( 2020 ) takes both level-0 and level-1 beliefs as input to the value function . Level-0 belief is updated according to Bayesian rule , and level-1 belief is updated using a learnable neural network . However , these methods use centralized training decentralized execution algorithms to train a set of fixed agents that can not handle diverse opponents in execution . Bayes-ToMoP ( Yang et al. , 2019 ) is developed to detect and handle different reasoning opponents , but the implementation is complex and not scalable . If the opponents are also learning , the modeling mentioned above becomes unstable and the fitted models with historical experiences lag behind . So it is beneficial to take into consideration the learning process of opponents . LOLA ( Foerster et al. , 2018a ) introduces the impact of the agent ’ s policy on the anticipated parameter update of the opponent . A neural network is used to model the opponent ’ s policy and estimate learning gradient of the opponent ’ s policy , implying that the learning algorithm used by the opponent should be known , otherwise the estimated gradient will be inaccurate . Further , the opponents may still be learning continuously during execution . Meta-PG ( Al-Shedivat et al. , 2018 ) is a method based on meta policy gradient , using trajectories come from the current opponents to do multiple meta-gradient steps and construct a policy that is good for the updated opponents . Meta-MAPG ( Kim et al. , 2021a ) extends this method by including an additional term that accounts for the impact of the agent ’ s current policy on the future policies of opponents , similar to LOLA . These meta-learning based methods require that the distribution of trajectories match across training and test , which implicitly means all opponent use a same learning algorithm . 2.2 MODEL-BASED RL AND MARL . Model-based RL allows the agent to have access to the transition function . There are two typical branches of model-based RL approaches : background planning and decision-time planning . In background planning , the agent could use the learned model to generate additional experiences for assisting learning . For example , Dyna-style algorithms ( Sutton , 1990 ; Kurutach et al. , 2018 ; Luo et al. , 2019 ) perform policy optimization on simulated experiences , and model-augmented value expansion algorithms ( Feinberg et al. , 2018 ; Oh et al. , 2017 ; Buckman et al. , 2018 ) use model-based rollouts to improve the update targets . In decision-time planning ( Chua et al. , 2018 ) , the agent could use the model to rollout the optimal action at a given state by looking forward during execution , e.g. , model predictive control . The model-based MARL methods could be divided into two classes : centralized model and decentralized model . AMLAPN ( Park et al. , 2019 ) builds a centralized auxiliary prediction network to model the environment dynamics and the opponent actions to alleviate the non-stationary dynamics . A centralized multi-step generative model ( Krupnik et al. , 2020 ) with a disentangled variational auto-encoder has been proposed for performing trajectory planning . For decentralized model , IS ( Kim et al. , 2021b ) uses the model to generate imagined trajectories for multi-agent communication to share intention . HPP ( Wang et al. , 2020 ) uses the models to propose and evaluate navigation subgoals for the rendezvous task . Our method relaxes the limitations on opponent modeling . We make no assumptions about the variation of opponents ’ policies . They could be fixed , randomly sampled from an unknowable policy set , or continuously learning using an unknowable and changeable RL algorithm , both in training and execution . We learn an environment model that allows the agent to perform recursive reasoning against opponents who may also have the same reasoning ability . 3 PRELIMINARIES . We consider an n-agent stochastic game ( S , A1 , . . . , An , P , R1 , . . . , Rn , γ ) , where S is the state space , Ai is the action space of agent i ∈ [ 1 , . . . , n ] , A = ∏n i=1Ai is the joint action space of agents , P : S ×A×S → [ 0 , 1 ] is a transition function , Ri : S ×A×S → R is the reward function of agent i , and γ is the discount factor . The policy of agent i is πi , and the joint policy of other agents is π−i ( a−i|s ) = ∏ j 6=i π j ( aj |s ) , where a−i is the joint action except agent i . All agents interact with the environment simultaneously without communication . The historical trajectory is available , i.e. , for agent i at timestep t , { s0 , ai0 , a−i0 , . . . , st−1 , ait−1 , a −i t−1 } is observable . The goal of the agent i is to maximize its expected cumulative discount rewards E st+1∼P ( ·|st , ait , a −i t ) , ai∼πi ( ·|st ) , a−it ∼π −i ( ·|st ) [ ∞∑ t=0 γtRi ( st , ait , a−it , st+1 ) ] . ( 1 ) For convenience , the learning agent treats all other agents as a joint opponent with the joint action ao ∼ πo ( ·|s ) and reward ro . The action and reward of the learning agent are denoted as a ∼ π ( ·|s ) and r , respectively . 𝜙𝑚 𝜑0 𝜑1 𝜑𝑛 Mixer …… 𝜑𝑚𝑖𝑥 𝑎𝑖 𝑎−𝑖 𝑎𝑖 𝜙0 𝜙1 𝜙𝑀 Ψ𝑖 𝑎−𝑖 𝜓0 𝜙𝑚𝑖𝑥 … … 𝑀𝐿𝑃 𝜓𝑖 … … ( 3 ) ( 5 ) … … 𝜶 ( 4 ) for m=0 to M-1 𝜙𝑚+1 𝜙𝑚 Rollout K steps 𝜋𝑖 ⋅ 𝑠 , 𝜙𝑚 ; 𝜃 𝐸𝑛𝑣𝑖𝑟𝑜𝑛𝑚𝑒𝑛𝑡 𝑚𝑜𝑑𝑒𝑙 𝑎−𝑖 Recursive Imagination 𝜙𝑚s 𝑎−𝑖∗ Mixer 𝜙0 𝜙1 𝜙𝑀 Ψ𝑖 𝑎−𝑖 𝜙𝑚𝑖𝑥 … … 𝑀𝐿𝑃 𝜓𝑖 … … ( 3 ) ( 5 ) … … 𝜶 ( 4 ) 𝜋 𝑎−𝑖~𝜌 ( ∙|𝑠 ; 𝜙𝑚 ) ( ∙ |𝑠 , 𝑎−𝑖 ; 𝜃 ) 𝜋 𝑎−𝑖~𝜌 ( ∙|𝑠 ; 𝜙𝑚𝑖𝑥 ) ( ∙ |𝑠 , 𝑎−𝑖 ; 𝜃 ) 𝜙𝑚 𝑀𝐿𝑃 Ψ ψ 𝜓0 𝜓1 𝜓𝑀

Summary: Agents learning in systems with other-agents that may be simultaneously learning presents the original agent with a dynamic learning problem. Critically, the agent does not know if the other-agents are non-learning, learning, or learning using recursive-reasoning. One such solution to this problem is to design a learning algorithm that accounts for how the other-agents may update their policy from the shared experiences, effectively eliminating the aforementioned problem. This work attempts to design such an algorithm without relying on any assumptions on the other-agent's learning algorithm. Their proposed algorithm, Model-Based Opponent Modeling (MBOM), learns parallel opponent models that are trained using various depths of recursive reasoning. Then, when played against an opponent, the agent can select the most likely opponent model (level-0: non-learning opponent, level-1: learning-opponent, level-k: k-1 level recursivly learning opponent). This algorithm assumes a centralized training and decentralized execution setting for two-player fully-observable general-sum games.