L2E: Learning to Exploit Your Opponent

1 INTRODUCTION . One core research topic in modern artificial intelligence is creating agents that can interact effectively with their opponents in different scenarios . To achieve this goal , the agents should have the ability to reason about their opponents ’ behaviors , goals , and beliefs . Opponent modeling , which constructs the opponents ’ models to reason about them , has been extensively studied in past decades ( Albrecht & Stone , 2018 ) . In general , an opponent model is a function that takes some interaction history as its input and predicts some property of interest of the opponent . Specifically , the interaction history may contain the past actions that the opponent took in various situations , and the properties of interest could be the actions that the opponent may take in the future , the style of the opponent ( e.g. , “ defensive ” , “ aggressive ” ) , or its current goals . The resulting opponent model can inform the agent ’ s decision-making by incorporating the model ’ s predictions in its planning procedure to optimize its interactions with the opponent . Opponent modeling has already been used in many practical applications , such as dialogue systems ( Grosz & Sidner , 1986 ) , intelligent tutor systems ( McCalla et al. , 2000 ) , and security systems ( Jarvis et al. , 2005 ) . Many opponent modeling algorithms vary greatly in their underlying assumptions and methodology . For example , policy reconstruction based methods ( Powers & Shoham , 2005 ; Banerjee & Sen , 2007 ) explicitly fit an opponent model to reflect the opponent ’ s observed behaviors . Type reasoning based methods ( Dekel et al. , 2004 ; Nachbar , 2005 ) reuse pre-learned models of several known opponents by finding the one which most resembles the behavior of the current opponent . Classification based methods ( Huynh et al. , 2006 ; Sukthankar & Sycara , 2007 ) build models that predict the play style of the opponent , and employ the counter-strategy , which is effective against that particular style . Some recent works combine opponent modeling with deep learning methods or reinforcement learning methods and propose many related algorithms ( He et al. , 2016 ; Foerster et al. , 2018 ; Wen et al. , 2018 ) . Although these algorithms have achieved some success , they also have some obvious disadvantages . First , constructing accurate opponent models requires a lot of data , which is problematic since the agent does not have the time or opportunity to collect enough data about its opponent in most applications . Second , most of these algorithms perform well only when the opponents during testing are similar to the ones used for training , and it is difficult for them to adapt to opponents with new styles quickly . More related works on opponent modeling are in Appendix A.1 . To overcome these shortcomings , we propose a novel Learning to Exploit ( L2E ) framework in this work for implicit opponent modeling , which has two desirable advantages . First , L2E does not build an explicit model for the opponent , so it does not require a large amount of interactive data and eliminates the modeling errors simultaneously . Second , L2E can quickly adapt to new opponents with unknown styles , with only a few interactions with them . The key idea underlying L2E to train a base policy against various styles of opponents by using only a few interactions between them during training , such that it acquires the ability to exploit different opponents quickly . After training , the base policy can quickly adapt to new opponents using only a few interactions during testing . In effect , our L2E framework optimizes for a base policy that is easy and fast to adapt . It can be seen as a particular case of learning to learn , i.e. , meta-learning ( Finn et al. , 2017 ) . The meta-learning algorithm ( c.f . , Appendix A.2 for details ) , such as MAML ( Finn et al. , 2017 ) , is initially designed for single-agent environments . It requires manual design of training tasks , and the final performance largely depends on the user-specified training task distribution . The L2E framework is designed explicitly for the multi-agent competitive environments , which generates effective training tasks ( opponents ) automatically ( c.f . , Appendix A.3 for details ) . Some recent works have also initially used meta-learning for opponent modeling . Unlike these works , which either use meta-learning to predict the opponent ’ s behaviors ( Rabinowitz et al. , 2018 ) or to handle the non-stationarity problem in multi-agent reinforcement learning ( Al-Shedivat et al. , 2018 ) , we focus on how to improve the agent ’ s ability to adapt to unknown opponents quickly . In our L2E framework , the base policy is explicitly trained such that a few interactions with a new opponent will produce an opponent-specific policy to effectively exploit this opponent , i.e. , the base policy has strong adaptability that is broadly adaptive to many opponents . In specific , if a deep neural network models the base policy , then the opponent-specific policy can be obtained by fine-tuning the parameters of the base policy ’ s network using the new interactive data with the opponent . A critical step in L2E is how to generate effective opponents to train the base policy . The ideal training opponents should satisfy the following two desiderata . 1 ) The opponents need to be challenging enough ( i.e. , hard to exploit ) . By learning to exploit these challenging opponents , the base policy eliminates its weakness and learns a more robust strategy . 2 ) The opponents need to have enough diversity . The more diverse the opponents during training , the stronger the base policy ’ s generalization ability is , and the more adaptable the base policy to the new opponents . To this end , we propose a novel opponent strategy generation ( OSG ) algorithm , which can produce challenging and diverse opponents automatically . We use the idea of adversarial training to generate challenging opponents . Some previous works have also been proposed to obtain more robust policies through adversarial training and showed that it improves the generalization ( Pinto et al. , 2017 ; Pattanaik et al. , 2018 ) . From the perspective of the base policy , giving an opponent , the base policy first adjusts itself to obtain an adapted policy , the base policy is then optimized to maximize the rewards that the adapted policy gets when facing the opponent . The challenging opponents are then adversarially generated by minimizing the base policy ’ s adaptability by automatically generating difficult to exploit opponents . These hard-to-exploit opponents are trained such that even if the base policy adapts to them , the adapted base policy can not take advantage of them . Besides , our OSG algorithm can further produce diverse training opponents with a novel diversity-regularized policy optimization procedure . In specific , we use the Maximum Mean Discrepancy ( MMD ) metric ( Gretton et al. , 2007 ) to evaluate the differences between policies . The MMD metric is then incorporated as a regularization term into the policy optimization process to obtain a diverse set of opponent policies . By training with these challenging and diverse training opponents , the robustness and generalization ability of our L2E framework can be significantly improved . To summarize , the main contributions of this work are listed bellow in four-fold : • We propose a novel learning to exploit ( L2E ) framework to exploit sub-optimal opponents without building explicit models for it . L2E can quickly adapt to a new opponent with unknown style using only a few interactions . • We propose to use an adversarial training procedure to generate challenging opponents automatically . These hard to exploit opponents help L2E eliminate its weakness and improve its robustness effectively . • We further propose a diversity-regularized policy optimization procedure to generate diverse opponents automatically . The generalization ability of L2E is improved significantly by training with these diverse opponents . • We conduct detailed experiments to evaluate the L2E framework in three different environments . The experimental results demonstrate that the base policy trained with L2E quickly exploits a wide range of opponents compared to other algorithms . 2 METHOD . In this paper , we propose a novel L2E framework to endow the agents to adapt to diverse opponents quickly . As shown in Fig . 1 , L2E mainly consists of two modules , i.e. , the base policy training part , and the opponent strategy generation part . In the base policy training part , our goal is to find a base policy that , given the unknown opponent , can fast adapt to it by using only a few interactions . To this end , the base policy is trained to be able to adapt to many opponents . In specific , giving an opponent O , the base policy B first adjusts itself to obtain an adapted policy B′ by using a little interaction data between O and B , the base policy is then optimized to maximize the rewards that B′ gets when facing O . In other words , the base policy has learned how to adapt to its opponents and exploit them quickly . The opponent strategy generation provides the base policy training part with challenging and diverse training opponents automatically . First , our proposed opponent strategy generation ( OSG ) algorithm can produce difficult to exploit opponents . In specific , the base policy B first adjusts itself to obtain an adapted policy B′ by using a little interaction data between O and B , the opponent O is then optimized to minimize the rewards that B′ gets when facing O . The resulting opponent O is hard to exploit since even if the base policy B adapts to O , the adapted policy B′ can not take advantage of O . By training with these hard to exploit opponents , the base policy can eliminate its weakness and improve its robustness effectively . Second , our OSG algorithm can further produce diverse training opponents with a novel diversity-regularized policy optimization procedure . More specifically , we first formalize the difference between opponent policies as the difference between the distribution of trajectories induced by each policy . The difference between distributions can be evaluated by the Maximum Mean Discrepancy ( MMD ) metric ( Gretton et al. , 2007 ) . Then , MMD is integrated as a regularization term in the policy optimization process to identify various opponent policies . By training with these diverse opponents , the base policy can improve its generalization ability significantly . Next , we introduce these two modules in detail . 2.1 BASE POLICY TRAINING . Our goal is to find a base policy B that can fast adapt to an unknown opponent O by updating the parameters of B using only a few interactions between B and O . The key idea is to train the base policy B against many opponents to maximize its payoffs by using only a small amount of interactive data during training , such that it acquires the ability to exploit different opponents quickly . In effect , our L2E framework treats each opponent as a training example . After training , the resulting base policy B can quickly adapt to new and unknown opponents using only a few interactions . Without loss of generality , the base policy B is modeled by a deep neural network in this work , i.e. , a parameterized function πθ with parameters θ . Similarly , the opponent O for training is also a deep neural network πφ with parameters φ . We model the base policy as playing against an opponent in a two-player Markov game ( Shapley , 1953 ) . This Markov game M = ( S , ( AB , AO ) , T , ( RB , RO ) ) consists of the state space S , the action space AB and AO , and a state transition function T : S × AB × AO → ∆ ( S ) where ∆ ( S ) is a probability distribution on S. The reward function Ri : S × AB × AO × S → R for each player i ∈ { B , O } depends on the current state , the next state and both players ’ actions . Given a training opponent O whose policy is known and fixed , this two-player Markov game M reduces to a single-player Markov Decision Process ( MDP ) , i.e. , MOB = ( S , AB , T O B , R O B ) . The state and action space of M O B are the same as in M . The transition and reward functions have the opponent policy embedded : TOB ( s , aB ) = T ( s , aB , aO ) , R O B ( s , aB , s ′ ) = RB ( s , aB , aO , s ′ ) , where the opponent ’ s action is sampled from its policy aO ∼ πφ ( · | s ) . Throughout the paper , MYX represents a single-player MDP , which is reduced from a two-player Markov game ( i.e. , player X and player Y ) . In this MDP , the player Y is fixed and can be regarded as part of the environment . Suppose a set of training opponents { Oi } Ni=1 is given . For each training opponentOi , an MDPM Oi B can be constructed as described above . The base policy B , i.e. , πθ is allowed to query a limited number of sample trajectories τ to adapt to Oi . In our method , the adapted parameters θOi of the base policy are computed using one or more gradient descent updates with the sample trajectories τ . For example , when using one gradient update : θOi = θ − α∇θLOiB ( πθ ) , ( 1 ) LOiB ( πθ ) = −Eτ∼MOiB [ ∑ t γtROiB ( s ( t ) , a ( t ) B , s ( t+1 ) ) ] . ( 2 ) τ ∼ MOiB represents that the trajectory τ = { s ( 1 ) , a ( 1 ) B , s ( 2 ) , . . . , s ( t ) , a ( t ) B , s ( t+1 ) , . . . } is sampled from the MDP MOiB , where s ( t+1 ) ∼ TOiB ( s ( t ) , a ( t ) B ) and a ( t ) B ∼ πθ ( · | s ( t ) ) . We use BOi to denote the updated base policy , i.e. , πθOi . BOi can be seen as an opponent-specific policy , which is updated from the base policy through fast adaptation . Our goal is to find a generalizable base policy whose opponent-specific policy BOi can exploit its opponent Oi as much as possible . To this end , we optimize the parameters θ of the base policy to maximize the rewards that BOi gets when interacting with Oi . More concretely , the learning to exploit objective function is defined as follows : min θ ∑N i=1 LOi BOi ( πθOi ) = min θ ∑N i=1 LOi BOi ( π θ−α∇θL Oi B ( πθ ) ) . ( 3 ) It is worth noting that the optimization is performed over the base policy ’ s parameters θ , whereas the objective is computed using the adapted based policy ’ s parameters θOi . The parameters θ of the base policy are updated as follows : θ = θ − β∇θ ∑N i=1 LOi BOi ( πθOi ) . ( 4 ) In effect , our L2E framework aims to find a base policy that can significantly exploit the opponent with only a few interactions with it ( i.e. , with a few gradient steps ) . The resulting base policy has learned how to adapt to different opponents and exploit them quickly . An overall description of the base policy training procedure is shown in Alg . 1 . The algorithm consists of three main steps . First , generating hard to exploit opponents through the Hard-OSG module . Second , generating diverse opponent policies through the Diverse-OSG module . Third , training the base policy with these opponents to obtain fast adaptability .

Summary: The authors propose an opponent modeling in 1-vs-1 games called the Learning to Exploit (L2E) framework, which exploits opponents by a few interactions with different opponents during training so that it can adapt to new opponents with unknown styles during testing quickly. In particular, the authors propose Opponent Strategy Generation (OSG) that produces effective opponents for training automatically through adversarial training for eliminating its own strategy’s weaknesses and diversity-regularized policy optimization to improve the generalization ability of L2E. Experimental results of two poker games and one grid soccer game indicate that L2E quickly adapts to diverse styles of unknown opponents.