Double Neural Counterfactual Regret Minimization

1 INTRODUCTION . While significant advance has been made in addressing large perfect information games , such as Go ( Silver et al. , 2016 ) , solving imperfect information games remains a challenging task . For Imperfect Information Games ( IIG ) , a player has only partial knowledge about her opponents before making a decision , so that she has to reason under the uncertainty about her opponents ’ information while exploiting the opponents ’ uncertainty about herself . Thus , IIGs provide more realistic modeling than perfect information games for many real-world applications , such as trading , traffic routing , and politics . Nash equilibrium is a typical solution concept for a two-player perfect-recall IIG . One of the most effective approaches is CFR ( Zinkevich et al. , 2007 ) , which minimizes the overall counterfactual regret so that the average strategies converge to a Nash equilibrium . However the original CFR only works for discrete states and action spaces , and the resulting strategy is maintained as a tabular representation . Such tabular representation limits the method from being directly applied to large games . To tackle this challenge , one can simplify the game by grouping similar states together to solve the simplified ( abstracted ) game approximately via tabular CFR ( Zinkevich et al. , 2007 ; Lanctot et al. , 2009 ) . Constructing an effective abstraction , however , demands rich domain knowledge and its solution may be a coarse approximation of true equilibrium . Function approximation can be used to replace the tabular representation . Waugh et al . ( 2015 ) combines regression tree function approximation with CFR based on handcrafted features , which is called Regression CFR ( RCFR ) . However , since RCFR uses full traversals of the game tree , it is still impractical for large games . Moravcik et al . ( 2017 ) propose a seminal approach DeepStack , which uses fully connected neural networks to represent players ’ counterfactual values , tabular CFR however was used in the subgame solving . Jin et al . ( 2017 ) use deep reinforcement learning to solve regret minimization problem for single-agent settings , which is different from two-player perfect-recall IIGs . ∗The previous versions of this work were published in AAAI 2019 workshop on RLG and ICML 2019 workshop on RWSDM . It takes more than one year for us to reimplement DeepStack and evaluate our method on large-scale heads-up no-limit Texas Hold ’ em . Correspondence to : Hui Li , lihuiknight @ google.com . 1 ABS-CFR is the advanced version of HITSZ_LMW_2pn , who won the third prize of the 2018 Annual Computer Poker Competition ( ACPC ) . In our experiment , we chosen its advanced version as the benchmark . To learn approximate Nash equilibrium for IIGs in an end-to-end manner , Heinrich et al . ( 2015 ) and Heinrich & Silver ( 2016 ) propose eXtensive-form Fictitious Play ( XFP ) and Neural Fictitious Self-Play ( NFSP ) , respectively , based on deep reinforcement learning . In a NFSP model , the neural strategies are updated by selecting the best responses to their opponents ’ average strategies . These approaches are advantageous in the sense that they do not rely on abstracting the game , and accordingly their strategies can improve continuously with more optimization iterations . However fictitious play empirically converges much slower than CFR-based approaches . Srinivasan et al . ( 2018 ) use actor-critic policy optimization methods to minimize regret and achieve performance comparable to NFSP . Thus it remains an open question whether a purely neural-based end-to-end approach can achieve comparable performance to tabular based CFR approach . In the paper , we solve this open question by designing a double neural counterfactual regret minimization ( DNCFR ) algorithm 2 . To make a neural representation , we modeled imperfect information game by a novel recurrent neural network with attention . Furthermore , in order to improve the convergence of the neural algorithm , we also developed a new sampling technique which converged much more efficient than the outcome sampling , while being more memory efficient than the external sampling . In the experiment , we conducted a set of ablation studies related to each novelty . The experiments showed DNCRF converged to comparable results produced by its tabular counterpart while performing much better than NFSP . In addition , we tested DNCFR on extremely large game , heads-up no-limit Texas Hold ’ em ( HUNL ) . The experiments showed that DNCFR with only a few number of parameters achieved strong neural strategy and beat ABS-CFR . 2 BACKGROUND ℎ0 ℎ1 ℎ3 ℎ4 ℎ7𝑧1 𝑧2 F C P B P B F C ℎ2 ℎ5 ℎ6 ℎ8 F C P B P B F C 𝑧3 𝑧4 𝑧5 𝑧6 𝑧7 𝑧8 𝑧9 𝑧10 player 0 player 1 chance infoset infoset 𝑣0 𝜎 ( 𝐵|𝐼0 ) 𝑣0 𝜎 ( 𝑃|𝐼0 ) 𝑣0 𝜎 𝐼0 = 𝑣0 𝜎 𝑃 𝐼0 𝜎0 𝑃 𝐼0 + 𝑣0 𝜎𝑡 ( 𝐵|𝐼0 ) 𝜎0 ( 𝐵|𝐼0 ) 𝑟0 𝜎 ( 𝐵|𝐼0 ) =𝑣0 𝜎 𝐵 𝐼0 − 𝑣0 𝜎 ( 𝐼0 ) Figure 1 : Extensive-Form IIG and Information Set • Notation.Figure 1 illustrates an extensive game for a finite set N = { 0,1 , ... , n−1 } of n players . Define xvi as the hidden information of player i in IIG . xv−i refers to hidden variables of all players other than i. H refers to a finite set of histories . h∈H denotes a possible history ( or state ) , which consists of each player ’ s hidden variable and actions taken by all players including chance . The empty sequence ∅ is a member of H. hj v h denotes hj is a prefix of h. Z ⊆H denotes the terminal histories and any member z∈Z is not a prefix of any other sequences . A ( h ) = { a : ha∈H } is the set of available actions after non-terminal history h∈H\Z . A player function P assigns a member of N∪ { c } to each non-terminal history , where c is the chance ( we set c=−1 ) . P ( h ) is the player who takes an action after history h. For each player i , imperfect information is denoted by information set ( infoset ) Ii . All states h∈Ii are indistinguishable to i. Ii refers to the set of infosets of i . The utility function ui ( z ) defines the payoff of i at state z . See appendix B.1 for more details . Algorithm 1 : CFR Algorithm For t=1 to T do vσ t i ( Ii ) = ∑ h∈Ii , hvz , z∈Z πσ t i ( h , z ) π σt −i ( z ) ui ( z ) . ( 1 ) rσ t i ( a|Ii ) =vσ t i ( a|Ii ) −vσ t i ( Ii ) . ( 2 ) Rti ( a|Ii ) =Rt−1i ( a|Ii ) +r σt i ( a|Ii ) . ( 3 ) σt+1i ( a|Ii ) = 1 |A ( Ii ) | if ∑ a∈A ( Ii ) Rt , +i ( a|Ii ) =0 R t , + i ( a|Ii ) ∑ a∈A ( Ii ) R t , + i ( a|Ii ) otherwise . ( 4 ) St ( a|Ii ) =St−1 ( a|Ii ) +πσ t i ( Ii ) σ t i ( a|Ii ) . ( 5 ) σ̄i T ( a|Ii ) = ST ( a|Ii ) ∑ a∈A ( Ii ) S T ( a|Ii ) . ( 6 ) A strategy profile σ = { σi|σi ∈ Σi , i ∈ N } is a collection of strategies for all players , where Σi is the set of all possible strategies for player i. σ−i refers to strategy of all players other than player i . For play i ∈ N , the strategy σi ( Ii ) is a function , which assigns an action distribution over A ( Ii ) to infoset Ii . σi ( a|h ) denotes the probability of action a taken by player i at state h. In IIG , ∀h1 , h2∈Ii , we have σi ( Ii ) =σi ( h1 ) =σi ( h2 ) . For iterative method such as CFR , σt refers to the strategy profile at t-th iteration . The state reach probability of history h is denoted by πσ ( h ) if players take actions according to σ . The reach probability is also called range in DeepStack ( Moravcik et al. , 2017 ) . Similarly , πσi ( h ) refers to those for player iwhile πσ−i ( h ) refers to those for other players except for i . For an empty sequence πσ ( ∅ ) = 1 . One can also show that the reach probability of the opponent is proportional to posterior 2 Solving IIGs via function approximation methods is an important and challenging problem . In the past year , several concurrent works ( Lockhart et al. , 2019 ; Brown et al. , 2018 ; Steinberger , 2019 ) have been proposed to address this problem . We will discuss their differences in Section 6. probability of the opponent ’ s hidden variable , i.e. , p ( xv−i|Ii ) ∝πσ−i ( h ) , where xvi and Ii indicate a particular h ( proof in Appendix D.1 ) . Finally , the infoset reach probability of Ii is defined as πσ ( Ii ) = ∑ h∈Iiπ σ ( h ) . Similarly , we have πσi ( Ii ) = ∑ h∈Iiπ σ i ( h ) and π σ −i ( Ii ) = ∑ h∈Iiπ σ −i ( h ) . More details can be found in Appendix B.3 . • Counterfactual Regret Minimization . CFR is an iterative method for finding a Nash equilibrium for zero-sum perfect-recall IIGs ( Zinkevich et al. , 2007 ) ( Algorithm 1 and Figure 2 ( a ) ) . Given strategy profile σ , the counterfactual value ( CFV ) vσi ( Ii ) at infoset Ii is defined by Eq . ( 1 ) . The action CFV of taking action a is vσi ( a|Ii ) and its regret is defined by Eq . ( 2 ) . Then the cumulative regret of action a after T iterations is Eq . ( 3 ) , where R0i ( a|Ii ) = 0 . Define R t , + i ( a|Ii ) = max ( Rti ( a|Ii ) ,0 ) , the current strategy ( or behavior strategy ) at t+1 iteration will be updated by Eq . ( 4 ) . Define sti ( a|Ii ) = πσ t i ( Ii ) σ t i ( a|Ii ) as the additional strategy in iteration t , then the cumulative strategy can be defined as Eq . ( 5 ) , where S0 ( a|Ii ) =0 . The average strategy σ̄it after t iterations is defined by Eq . ( 6 ) , which approaches a Nash equilibrium after enough iterations . • Monte Carlo CFR.Lanctot et al . ( 2009 ) proposed a Monte Carlo CFR ( MCCFR ) to compute the unbiased estimation of counterfactual value by sampling subsets of infosets in each iteration . Although MCCFR still needs two tabular storages for saving cumulative regret and strategy as CFR does , it needs much less working memory than the standard CFR ( Zinkevich et al. , 2007 ) . This is because MCCFR needs only to maintain values for those visited nodes into working memory ; DefineQ= { Q1 , Q2 , ... , Qm } , whereQj∈Z is a set ( block ) of sampled terminal histories in each iteration , such thatQj spans the set Z . Define qQj as the probability of considering blockQj , where ∑m j=1qQj =1 . Define q ( z ) = ∑ j : z∈QjqQj as the probability of considering a particular terminal history z . For infoset Ii , an estimate of sampled counterfactual value is ṽσi ( Ii|Qj ) = ∑ h∈Ii , z∈Qj , hvz 1 q ( z ) π σ −i ( z ) π σ i ( h , z ) ui ( z ) . Lemma 1 ( Lanctot et al . ( 2009 ) ) The sampled counterfactual value in MCCFR is the unbiased estimation of actual counterfactual value in CFR.Ej∼qQj [ ṽ σ i ( Ii|Qj ) ] =vσi ( Ii ) . Define σrs as sampled strategy profile , where σrsi is the sampled strategy of player i and σ rs −i are those for other players except for i . The regret of the sampled action a∈A ( Ii ) is defined by r̃σi ( ( a|Ii ) |Qj ) =∑ z∈Qj , havz , h∈Iiπ σ i ( ha , z ) u rs i ( z ) − ∑ z∈Qj , hvz , h∈Iiπ σ i ( h , z ) u rs i ( z ) , where u rs i ( z ) = ui ( z ) πσ rs i ( z ) is a new utility weighted by 1 πσ rs i ( z ) . The sampled estimation for cumulative regret of action a after t iterations is R̃ti ( ( a|Ii ) |Qj ) =R̃ t−1 i ( ( a|Ii ) |Qj ) +r̃σ t i ( ( a|Ii ) |Qj ) , where R̃0i ( ( a|Ii ) |Qj ) =0 .

Summary: The authors proposed a double neural counterfactual regret minimization algorithm (DNCFR) that uses a RegretSumNetwork to approximate cumulative regret and an AvgStrategyNetwork to approximate the average strategy. To help the training, the authors use robust sampling and a mini-batch training methods. The main contributions of the paper are: First, the DNCFR algorithm and the recurrent neural network architecture; Second, an efficient sampling and training method; Third, plenty of experiments that corroborate the effectiveness of DNCFR.