Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game

1 INTRODUCTION . Policy gradient methods using deep neural networks as policy and value approximators have been successfully applied to many large-scale games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ; Ye et al. , 2020 ) . Usually , a score function representing the discounted returns is maximized by the policy , i.e. , the actor . In the meantime , a value function , known as the critic , is learned to guide the directions and magnitudes of the policy gradients . This type of actor-critic methods are efficiently scalable with regard to the game size and the amount of computational resources . However , as pointed out in Srinivasan et al . ( 2018 ) and Hennes et al . ( 2020 ) , policy gradient methods have no convergence guarantee to optimal solutions in competitive Imperfect-Information Games ( IIGs ) . The main reason is that the policy gradient theorem ( Sutton et al. , 1999 ) is established within the single agent situation , where the environment is Markovian . However , learning becomes non-stationary and non-Markovian when multiple agents learn simultaneously in a competitive environment . An optimal solution to a 2-player zero-sum IIG usually refers to a Nash Equilibrium ( NE ) , where no player could improve by unilaterally deviating to a different policy . Tremendous progress in computing such optimal solutions has been made by a family of tabular methods : Counterfactual Regret Minimization ( CFR ) ( Zinkevich et al. , 2008 ) . CFR is a type of iterative self-play algorithm based on regret minimization , and it guarantees to converge to a NE with regard to the average policy in 2-player zero-sum IIGs . A game model is required in CFR to sample many if not all actions from a state during self-play . To handle large-scale IIGs with CFR , abstractions1 , applied to either the action space or the state space , are usually employed to reduce the game to a manageable size ( Moravčík et al. , 2017 ; Brown & Sandholm , 2018 ; 2019 ) . However , abstractions are domain specific ( Waugh et al. , 2009 ; Johanson et al. , 2013 ; Ganzfried & Sandholm , 2014 ) . Besides , some large-scale IIGs may be inherently difficult to be abstracted . 1DeepStack ( Moravčík et al. , 2017 ) employs sparse lookahead trees , much like action abstraction . In this paper , we investigate a large-scale IIG , Mahjong . Li et al . ( 2020b ) has recently developed a strong 4-player Mahjong AI based on supervised learning and traditional Reinforcement Learning ( RL ) . In comparison , we study Mahjong from a game-theoretic perspective , i.e. , aiming for a NE . We focus on a 2-player ( 1-on-1 ) zero-sum Mahjong game , whose information set size and game length are much larger than poker2 . We are interested in methods using only trajectory samples to learn , as it is infeasible to consistently sample multiple actions for each state in game trees of large-scale IIGs with long episodes . Also , we employ deep neural networks to generalize across states , since the state abstraction in 1-on-1 Mahjong is not as straightforward as in poker , which is explained in the Appendix A.2 . To summarize , we make the following contributions . • Inheriting the scalability of deep RL methods and the convergence property of CFR , we develop a new actor-critic algorithm , named Actor-Critic Hedge ( ACH ) , for approaching a NE in largescale 2-player zero-sum IIGs , such as 1-on-1 Mahjong . ACH employs a deep neural network to approximate a type of weighted cumulative counterfactual regrets . In the meantime , ACH minimizes the regrets via generating self-play strategies using Hedge ( Freund & Schapire , 1997 ) . • We introduce a Neural-based Weighted CFR ( NW-CFR ) , for which we prove that the exploitability of the average strategy decreases at the rate of O ( T−1/2 ) +O ( 1 ) , where T denotes the number of iterations . ACH is a practical implementation of NW-CFR and hence theoretically justified . • To facilitate research on large-scale 2-player zero-sum IIGs and Mahjong , we propose a 1-on-1 Mahjong benchmark . The corresponding game enjoys a large population in online games . • We build a 1-on-1 Mahjong agent , named JueJong , based on ACH . In an initial evaluation against human players including a Mahjong champion3 , JueJong demonstrates superior performance . 2 NOTATIONS AND BACKGROUND . In this section , we introduce notations in the context of IIGs and NE . Related background on RL ( focusing on policy gradient methods ) and CFR are briefly described . 2.1 IMPERFECT-INFORMATION GAMES AND NASH EQUILIBRIUM . An IIG is usually represented in an extensive-form , i.e. , a game tree . A node ( history ) h ∈ H in the tree represents all information of the current situation . Given a history h , P : H → P ∪ { c } maps h to the unique player p ∈ P who acts at h or the chance player c , where the chance player is responsible for taking actions for random events . When P ( h ) ∈ P , the player P ( h ) has to take an action a ∈ A . We use A ( h ) to represent the set of legal actions of P ( h ) . The set of terminal nodes is denoted as Z . At each z ∈ Z , there is a payoff up : Z → R for each player p ∈ P . In this paper , we focus on 2-player zero-sum games , where P = { 0 , 1 } and u0 ( z ) + u1 ( z ) = 0 for each z ∈ Z . For either player p , the set of historiesH is partitioned into information sets ( infosets ) . We denote the set of infosets for player p as Ip and an infoset in Ip as Ip . Two histories h , h′ ∈ H are in the same infoset if and only if h and h′ are indistinguishable from the perspective of player p. As a result , a player p ’ s strategy πp maps an infoset to a probability distribution over legal actions . For the same reason , we define A ( Ip ) = A ( h ) , ∀h ∈ Ip and P ( Ip ) = P ( h ) , ∀h ∈ Ip . A strategy profile π is simply a tuple of strategies ( πp , π−p ) , where π−p represents the player p ’ s opponent strategy . The expected payoff for player p under π is then denoted as up ( πp , π−p ) . We use ∆ ( I ) to denote the range of payoffs reachable from a history h in infoset I . Let ∆ = maxI∈Ip , p∈P ∆ ( I ) . f π ( h ) denotes the joint reaching probability of h under π. fπp ( h ) is the contribution of player p to f π ( h ) , and fπ−p ( h ) is the contribution of the opponent and chance : fπ ( h ) = fπp ( h ) f π −p ( h ) . We focus on the perfect-recall setting , where each player recalls the sequence of their own infosets reached . We define fπp ( Ip ) = fπp ( h ) , ∀h ∈ Ip , fπ−p ( Ip ) = ∑ h∈Ip f π −p ( h ) , and f π ( Ip ) = ∑ h∈Ip f π ( h ) = fπp ( Ip ) f π −p ( Ip ) . A best response BR ( π−p ) to π−p is a player p ’ s strategy , which satisfies up ( BR ( π−p ) , π−p ) = maxπ′p up ( π ′ p , π−p ) . A NE is a strategy profile π ∗ , where each player plays a best response to the other : up ( π∗p , π ∗ −p ) = maxπ′p up ( π ′ p , π ∗ −p ) , ∀p ∈ P . The exploitability of a player ’ s strategy , 2Without specification , we mean Heads-up No-Limit Texas Hold ’ em in this paper . 3Haihua Cheng is the Competition Mahjong champion of 2014 World Mahjong Master Tournament , 2018 Tencent Mahjong Tournament , and 2019 Tencent Mahjong Tournament . denoted as e ( πp ) , measures the performance gap between πp and a NE strategy π∗p : e ( πp ) = up ( π ∗ p , π ∗ −p ) − up ( πp , BR ( πp ) ) . The average exploitability is = 1|P| ∑ p∈P e ( πp ) . 2.2 REINFORCEMENT LEARNING AND POLICY GRADIENT METHODS . RL usually assumes a Markov decision process , where the agent selects an action ai from the action set A ( si ) in state si ∈ S4 at each time step i . The agent then receives a reward ri from the environment and transitions to a new state si+1 . The objective in RL is to learn a policy that maximizes the expected discounted returns , i.e. , the state value , starting from any state s : V π ( s ) = Eπ [ ∑∞ j=i γ j−irj |si = s ] = Eπ [ G ] , with the discount factor γ ∈ [ 0 , 1 ) . A large family of methods in RL belong to policy gradient methods , in which the parameters θ of policy π ( a|s ; θ ) are updated by performing gradient ascent directly on Eπθ [ G ] . One early example is the standard REINFORCE algorithm ( Williams , 1992 ) that updates θ in the direction ∇θ log π ( a|s ; θ ) G , which is an unbiased estimate of ∇θEπθ [ G ] . To further reduce the variance of the gradient , an action-independent baseline is often subtracted from G : ∇θ log π ( a|s ; θ ) [ G−B ( s ) ] . A recent policy gradient method , named Advantage Actor-Critic ( A2C ) ( Mnih et al. , 2016 ) , learns a parameterized value function as the baseline : B ( s ) = V πθ ( s ; w ) , where w often shares some parameters with θ . Moreover , the value G−B ( s ) is replaced in A2C with the estimated advantage of action a in state s : A ( s , a ) = Q ( s , a ) − V ( s ) . Q ( s , a ) is usually estimated using sampled rewards and predicted values of future states . A2C updates parameters θ and w in a synchronous manner , while its asynchronous version is Asynchronous Advantage Actor-Critic ( A3C ) ( Mnih et al. , 2016 ) . In an asynchronous training environment , the behavioral policy is usually different from the learning policy . Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) takes this discrepancy into account by multiplying A2C gradient with an importance ratio r ( a|s ; θ ) = π ( a|s ; θ ) /π ( a|s ; θold ) , which results in r ( a|s ; θ ) ∇θ log π ( a|s ; θ ) A ( s , a ) = ∇θr ( a|s ; θ ) A ( s , a ) . Furthermore , PPO constrains the KL divergence between the learning policy πθ and the old behavioral policy πθold by clipping the ratio to a small interval around 1.0 .

Summary: This paper proposes a method somewhat similar to Deep CFR, NeuRD and RPG, where a policy is trained to predict a weighted counterfactual regret. In small openspiel experiments the method seems to outperform NeuRD and RPG, but no comparisons are made to Deep CFR/DREAM/ARMAC. Impressive results against a top Mahjong player demonstrate that this method can scale to large games.