Safe Opponent-Exploitation Subgame Refinement

1 INTRODUCTION . Behind the recent breakthroughs of superhuman AIs in Go ( Silver et al. , 2016 ; 2017 ; Schrittwieser et al. , 2020 ) , heads-up no-limit Texas hold ’ em ( HUNL ) ( Brown et al. , 2018 ; Moravcík et al. , 2017 ; Brown & Sandholm , 2019 ; Brown et al. , 2020 ) , and Hanabi ( Lerer et al. , 2020 ) , search plays a vital role . In perfect information games , Monte Carlo tree search ( MCTS ) is widely applied to improve policy ’ s strength . In zero-sum imperfect information games such as poker , search algorithms are used to find a Nash equilibrium ( NE ) approximation in subgames encountered in real time ( Brown & Sandholm , 2017 ; Burch et al. , 2014a ) . They are both theoretically sounded and empirically powerful . In fully-cooperative imperfect information games , the search algorithm proposed in Tian et al . ( 2020b ) is proved to never be detrimental to the current policy . Lerer et al . ( 2020 ) also ensures original performance asymptotically . In zero-sum games , NE-based search algorithms ( Burch et al. , 2014a ; Moravcik et al. , 2016 ; Brown & Sandholm , 2017 ; Brown et al. , 2018 ) find safe strategies with low exploitability and produce strong baselines against all opponents ( Brown & Sandholm , 2019 ) . However , it may be overly conservative confronted with opponents with limited rationality and fail to take advantage of their weaknesses to obtain higher rewards ( McCracken & Bowling , 2004 ; Johanson et al. , 2007 ; Li & Miikkulainen , 2018 ) . From the other perspective , there have been extensive studies on opponent exploitation to address the problem . Some typical works ( Carmel & Markovitch , 1996 ; Billings et al. , 2003 ; Gilpin & Sandholm , 2006 ; Li & Miikkulainen , 2018 ) model the opponent ’ s strategy based on previous observations and then search for a new strategy to exploit this model . However , these methods often neglect the significance of the strategy safety , thus being highly exploitable by the opponent . Few exceptions including Johanson et al . ( 2007 ) and Ganzfried & Sandholm ( 2015a ) aimed to search for safe and robust counter strategies , but they are computationally inefficient in an online setting where the opponent model is being updated continuously with streamed data . In this paper , we study the dilemma of safety and opponent exploitation and present a new scalable real-time search framework Safe Exploitation Search ( SES ) that smoothly interpolates between the two extremes of strategy search , hence unifying safe search and opponent exploitation . It enables computationally efficient online adaptations to a continuously changing opponent model , which is hard to address by previous safe exploitation algorithms . The safety criterion requires the refined strategy to stay close to NE , formally speaking , to expose limited exploitability against any opponents , while the opponent exploitation criterion requires the strategy to adapt to its specific opponent and to exploit its weaknesses . We propose a novel maximization objective which combines the safety objective and exploitation , controlled by the exploitation level α . We construct a new gadget game to optimize this objective , which enables our method ’ s scalability to large games such as Texas Hold ’ em . Theoretically , we prove that SES is guaranteed to outperform NE at the cost of some constant increase in its own exploitability confronted with non-NE opponents . Empirically , we evaluate the effectiveness of our search algorithm in 1 didactic matrix game 2 poker games : Leduc Hold ’ em ( Southey et al. , 2005 ) and Flop Hold ’ em Poker ( FHP ) ( Brown et al. , 2019 ) . The experiment results demonstrate that our algorithm significantly outperforms NE baselines against non-NE opponents and keeps low exploitability at the same time . Additionally , SES is much more computationally efficient than previous safe exploitation baselines . 2 RELATED WORK . This paper investigates the problem of safe opponent exploitation in two-player zero-sum imperfect information games . We propose a novel search algorithm which balances between NE and exploiting opponents . Two major relevant research areas are search algorithms in imperfect information games , and opponent exploitation . Search in imperfect information games . In recent literature , search techniques are witnessed to be important in developing strong AI strategies in both perfect and imperfect information games ( Burch et al. , 2014a ; Moravcik et al. , 2016 ; Brown & Sandholm , 2017 ) . Texas hold ’ em poker is widely employed as a benchmark for imperfect information games . A primary part of the long-term research on Texas hold ’ em poker is the evolution of subgame solving algorithms , which aim at achieving a more accurate Nash equilibrium approximation in the subgame encountered given a pre-computed strategy for the full game which we refer to as the blueprint strategy . Unsafe search ( Billings et al. , 2003 ; Ganzfried & Sandholm , 2015b ; Gilpin & Sandholm , 2006 ; 2007 ) estimates the subgame reach probability assuming the opponent follows blueprint , and searches for a refined subgame strategy . Subgame resolving ( Burch et al. , 2014a ) and maxmargin search ( Moravcik et al. , 2016 ) are theoretically sounded safe search algorithms which ensure that the subgame strategy obtained is no worse than the blueprint . They search in a gadget game and achieve safety by providing the opponent with the option not entering the current subgame . DeepStack ( Moravcík et al. , 2017 ) and Libratus ( Brown et al. , 2018 ) build strong poker AIs with the aid of search . Beyond poker , search algorithms for subgame refinement have also shown promise in improving joint strategies in cooperative imperfect information games such as Hanabi ( Lerer et al. , 2020 ) and the bidding phase of contract bridge ( Tian et al. , 2020a ) . The purpose of our search algorithm is different from previous methods in poker literature . We seek to exploit opponents while keeping exploitability low , rather than simply approximating NE . Opponent exploitation . Most previous opponent exploitation researches ( Carmel & Markovitch , 1996 ; Billings et al. , 2003 ; Gilpin & Sandholm , 2006 ; Li & Miikkulainen , 2018 ) typically model the opponent ’ s strategy based on previous observations and then search for a new strategy to exploit this model , but put little emphasis on safety . One similar work is Johanson et al . ( 2007 ) which proposes p-restricted Nash response ( RNR ) to find a safe exploitation strategy to the estimated opponent ’ s strategy . It calculates a Nash equilibrium for the whole game restricting that the opponent plays the estimated strategy σfix with probability p , and any strategy with probability 1− p. In that paper , Johanson et al . ( 2007 ) prove that a p-RNR to σfix is Pareto optimal with respect to exploitation and safety . However , it does not provide an explicit bound . Additionally , whenever the estimated opponent model changes or we want to use a different p to balance between safety and exploitation , p-RNR has to recompute the strategy for the whole game . It is computationally inefficient in an online setting , where the opponent model is updated after every round with new game data . Our algorithm instead takes modelling error into account and provides explicit bounds for both safety and exploitation . With the aid of real-time search , it only searches for strategies in subgames encountered instead of the whole game . Our experiments show that it is more efficient than Johanson et al . ( 2007 ) . Ganzfried & Sandholm ( 2015a ) study safe exploitation strategies in repeated games , which is a different setting from this paper . Intuitively , it achieves safety by risking in exploitability at most what it has earned over NE in expectation in previous rounds . Therefore , its expected value in the whole repeated game is never worse than the NE . In contrast , this paper focuses on the safety of stage game strategies . Furthermore , our algorithm is complementary to Ganzfried & Sandholm ( 2015a ) . Ganzfried & Sandholm ( 2015a ) calculate an ε-safe best response for the whole game at each iteration with LP . This procedure is one of the main limitations on the algorithm ’ s scalability . Our algorithm , which only refines strategies in subgames in real-time , can be a possible substitute for LP . To our knowledge , we are the first paper to investigate the safe opponent exploitation problem in subgame resolving schemes . Subgame resolving enables online adaptations to a continuously changing opponent model , eliminating the need to recompute a whole game strategy . It offers computational benefits in practical opponent exploitation circumstances . It is worthwhile mentioning that there also has been extensive research ( Albrecht & Stone , 2018 ) on agent modeling . However , this paper only focuses on the theoretical and empirical results of the search algorithm , but not the agent modeling techniques . We can use off-the-shelf agent modeling algorithms to estimate opponent ’ s strategies . Agent modeling provides the ability to reason about the others , making predictions of their strategies , types , etc . He et al . ( 2016 ) embeds opponent model learning into deep Q-learning , which stabilizes the training process faced with opponents whose policy is changing over time . LOLA ( Foerster et al. , 2018 ) considers the evolution of other agents in the training process of multi-agent reinforcement learning . Raileanu et al . ( 2018 ) proposes to model others from one ’ s own policy . Rabinowitz et al . ( 2018 ) uses meta learning to predict strategies of different kinds of agents . 3 NOTATIONS AND BACKGROUND . An extensive-form imperfect information game G = ( P , H , Z , A , χ , ρ , · , σc , u , I ) describes sequential interactions among agents , where agents have private information . A finite set P consists of n players and a chance node c which represents the stochastic nature of the environment . The set of non-terminal decision nodes is denoted as H , and Z is a set of terminal nodes , or leaves . The set of possible actions is A , and χ : H → 2|A| is a function which assigns to each decision node h ∈ H a set of legal actions . A player function ρ : H → P assigns to each decision node a player p ∈ P who acts at that node . If action a leads from h to h′ , we write h · a = h′ . If there exists a sequence of actions leading from h to h′ , we write h v h′ . At each node h ∈ H , the acting player p = ρ ( h ) picks an action from legal actions a ∈ χ ( h ) , and leads node h into its child h · a . The chance node always samples an action from its own distribution σc , which is common knowledge to all players . Utility functions are u = ( u1 , u2 , . . . , un ) , where ui : Z → R defines the utility of player i at terminal node z ∈ Z . The nature of imperfect information is characterized by infosets I = ( I1 , I2 , . . . , In ) , where Ii = ( Ii,1 , . . . , Ii , ki ) is a partition of H for player i . Two states in the same infoset must have the same acting player and the same legal action sets . We use I ( h ) to denote the infoset that h belongs to . A player p can not distinguish between states h1 and h2 if I ( h1 ) = I ( h2 ) , and thus should behave identically on all states in the same infoset . The strategy of a player p is σp : Ip ×A→ R , where σp ( I , a ) is a distribution over valid actions on infoset I . For simplicity , we also use σp ( h , a ) to denote σp ( I ( h ) , a ) . We use πσ ( h ) to denote the probability of reaching state h from the root when agents choose a strategy profile σ = 〈σ1 , σ2 , . . . , σn〉 . Formally , πσ ( h ) = ∏ h′·avh σρ ( h′ ) ( h ′ , a ) . We use πσ−p ( h ) = ∏ h′·avh∧ρ ( h′ ) 6=p σρ ( h′ ) ( h ′ , a ) to denote the probability of reaching h when player p always chooses the action that leads to h whenever possible . πσ ( h , h′ ) is the reaching probability of h′ from h. πσ ( h · a , h′ ) is the the probability of reaching h′ from h if action a is taken at h. These probabilities can be formally defined in a similar manner . The expected utility of player p given strategy profile σ is uσp = ∑ z∈Z π σ ( z ) up ( z ) . The counterfactual value vσp ( I , a ) is the expected utility that player p will obtain after taking action a at infoset I , given the joint policy profile is σ . Mathematically , it is the weighted sum of expected values at all states h ∈ I . vσp ( I , a ) = ∑ h∈I , z∈Z π σ −p ( h ) π σ ( h · a , z ) up ( z ) ∑ h∈I π σ −p ( h ) ( 1 ) In the rest of the paper , we focus on two-player zero-sum games with perfect recall . Zero-sum means ∀z ∈ Z , u1 ( z ) + u2 ( z ) = 0 . Perfect recall means that no player will forget the information which has been obtained previously in the game . This is a common assumption in related literature . A best response strategyBRp ( σ−p ) = arg maxσp u 〈σp , σ−p〉 p for player p is the strategy that maximize his own expected utility against fixed opponent strategy σ−p . The exploitability of strategy σp is exp ( σp ) = u σ∗ p − u 〈σp , BR−p ( σp ) 〉 p where σ∗ is the optimal strategy , and is an NE in two-player zero-sum games . It measures the performance of σp against its best response comparing with the NE . A counterfactual best response CBRp ( σ−p ) is a strategy where σp ( I , a ) > 0 if and only if vσp ( I , a ) ≥ maxb vσp ( I , b ) . Counterfactual best response is a best response , but not vice versa . The counterfactual best response value CBV σ−pp ( I ) = v 〈CBRp ( σ−p ) , σ−p〉 p is the expected utility of the counterfactual best response policy . Since we focus on two-player zero-sum games , we will use CBV σp ( I ) as a shorthand notation for CBV σp−p ( I ) . We follow the imperfect information subgame definition as in Burch et al . ( 2014b ) . An augmented infoset contains states which can not be distinguished by the remaining players . Definition 1 . An imperfect information subgame S is a forest of trees , closed under both the descendant relation and membership within augmented infosets for any player . Let Stop be the set of nodes which are roots of each tree in S .

Summary: This paper proposes a method of subgame resolving in zero-sum extensive form games, blending in elements of theoretical safety, as well as more classical approaches based on opponent modeling. The method is fairly straightforward, utilizing a modified objective balancing between the safety objective (used in Moravcik et al) and a best-response like objective use in opponent modeling. To facilitate compatibility with modern solvers such as CFR, the authors propose gadgets transforming the refinement step into another zero-sum extensive form game.