Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game

1 INTRODUCTION . Policy gradient methods using deep neural networks as policy and value approximators have been successfully applied to many large-scale games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ; Ye et al. , 2020 ) . Usually , a score function representing the discounted returns is maximized by the policy , i.e. , the actor . In the meantime , a value function , known as the critic , is learned to guide the directions and magnitudes of the policy gradients . This type of actor-critic methods are efficiently scalable with regard to the game size and the amount of computational resources . However , as pointed out in Srinivasan et al . ( 2018 ) and Hennes et al . ( 2020 ) , policy gradient methods have no convergence guarantee to optimal solutions in competitive Imperfect-Information Games ( IIGs ) . The main reason is that the policy gradient theorem ( Sutton et al. , 1999 ) is established within the single agent situation , where the environment is Markovian . However , learning becomes non-stationary and non-Markovian when multiple agents learn simultaneously in a competitive environment . An optimal solution to a 2-player zero-sum IIG usually refers to a Nash Equilibrium ( NE ) , where no player could improve by unilaterally deviating to a different policy . Tremendous progress in computing such optimal solutions has been made by a family of tabular methods : Counterfactual Regret Minimization ( CFR ) ( Zinkevich et al. , 2008 ) . CFR is a type of iterative self-play algorithm based on regret minimization , and it guarantees to converge to a NE with regard to the average policy in 2-player zero-sum IIGs . A game model is required in CFR to sample many if not all actions from a state during self-play . To handle large-scale IIGs with CFR , abstractions1 , applied to either the action space or the state space , are usually employed to reduce the game to a manageable size ( Moravčík et al. , 2017 ; Brown & Sandholm , 2018 ; 2019 ) . However , abstractions are domain specific ( Waugh et al. , 2009 ; Johanson et al. , 2013 ; Ganzfried & Sandholm , 2014 ) . Besides , some large-scale IIGs may be inherently difficult to be abstracted . 1DeepStack ( Moravčík et al. , 2017 ) employs sparse lookahead trees , much like action abstraction . In this paper , we investigate a large-scale IIG , Mahjong . Li et al . ( 2020b ) has recently developed a strong 4-player Mahjong AI based on supervised learning and traditional Reinforcement Learning ( RL ) . In comparison , we study Mahjong from a game-theoretic perspective , i.e. , aiming for a NE . We focus on a 2-player ( 1-on-1 ) zero-sum Mahjong game , whose information set size and game length are much larger than poker2 . We are interested in methods using only trajectory samples to learn , as it is infeasible to consistently sample multiple actions for each state in game trees of large-scale IIGs with long episodes . Also , we employ deep neural networks to generalize across states , since the state abstraction in 1-on-1 Mahjong is not as straightforward as in poker , which is explained in the Appendix A.2 . To summarize , we make the following contributions . • Inheriting the scalability of deep RL methods and the convergence property of CFR , we develop a new actor-critic algorithm , named Actor-Critic Hedge ( ACH ) , for approaching a NE in largescale 2-player zero-sum IIGs , such as 1-on-1 Mahjong . ACH employs a deep neural network to approximate a type of weighted cumulative counterfactual regrets . In the meantime , ACH minimizes the regrets via generating self-play strategies using Hedge ( Freund & Schapire , 1997 ) . • We introduce a Neural-based Weighted CFR ( NW-CFR ) , for which we prove that the exploitability of the average strategy decreases at the rate of O ( T−1/2 ) +O ( 1 ) , where T denotes the number of iterations . ACH is a practical implementation of NW-CFR and hence theoretically justified . • To facilitate research on large-scale 2-player zero-sum IIGs and Mahjong , we propose a 1-on-1 Mahjong benchmark . The corresponding game enjoys a large population in online games . • We build a 1-on-1 Mahjong agent , named JueJong , based on ACH . In an initial evaluation against human players including a Mahjong champion3 , JueJong demonstrates superior performance . 2 NOTATIONS AND BACKGROUND . In this section , we introduce notations in the context of IIGs and NE . Related background on RL ( focusing on policy gradient methods ) and CFR are briefly described . 2.1 IMPERFECT-INFORMATION GAMES AND NASH EQUILIBRIUM . An IIG is usually represented in an extensive-form , i.e. , a game tree . A node ( history ) h ∈ H in the tree represents all information of the current situation . Given a history h , P : H → P ∪ { c } maps h to the unique player p ∈ P who acts at h or the chance player c , where the chance player is responsible for taking actions for random events . When P ( h ) ∈ P , the player P ( h ) has to take an action a ∈ A . We use A ( h ) to represent the set of legal actions of P ( h ) . The set of terminal nodes is denoted as Z . At each z ∈ Z , there is a payoff up : Z → R for each player p ∈ P . In this paper , we focus on 2-player zero-sum games , where P = { 0 , 1 } and u0 ( z ) + u1 ( z ) = 0 for each z ∈ Z . For either player p , the set of historiesH is partitioned into information sets ( infosets ) . We denote the set of infosets for player p as Ip and an infoset in Ip as Ip . Two histories h , h′ ∈ H are in the same infoset if and only if h and h′ are indistinguishable from the perspective of player p. As a result , a player p ’ s strategy πp maps an infoset to a probability distribution over legal actions . For the same reason , we define A ( Ip ) = A ( h ) , ∀h ∈ Ip and P ( Ip ) = P ( h ) , ∀h ∈ Ip . A strategy profile π is simply a tuple of strategies ( πp , π−p ) , where π−p represents the player p ’ s opponent strategy . The expected payoff for player p under π is then denoted as up ( πp , π−p ) . We use ∆ ( I ) to denote the range of payoffs reachable from a history h in infoset I . Let ∆ = maxI∈Ip , p∈P ∆ ( I ) . f π ( h ) denotes the joint reaching probability of h under π. fπp ( h ) is the contribution of player p to f π ( h ) , and fπ−p ( h ) is the contribution of the opponent and chance : fπ ( h ) = fπp ( h ) f π −p ( h ) . We focus on the perfect-recall setting , where each player recalls the sequence of their own infosets reached . We define fπp ( Ip ) = fπp ( h ) , ∀h ∈ Ip , fπ−p ( Ip ) = ∑ h∈Ip f π −p ( h ) , and f π ( Ip ) = ∑ h∈Ip f π ( h ) = fπp ( Ip ) f π −p ( Ip ) . A best response BR ( π−p ) to π−p is a player p ’ s strategy , which satisfies up ( BR ( π−p ) , π−p ) = maxπ′p up ( π ′ p , π−p ) . A NE is a strategy profile π ∗ , where each player plays a best response to the other : up ( π∗p , π ∗ −p ) = maxπ′p up ( π ′ p , π ∗ −p ) , ∀p ∈ P . The exploitability of a player ’ s strategy , 2Without specification , we mean Heads-up No-Limit Texas Hold ’ em in this paper . 3Haihua Cheng is the Competition Mahjong champion of 2014 World Mahjong Master Tournament , 2018 Tencent Mahjong Tournament , and 2019 Tencent Mahjong Tournament . denoted as e ( πp ) , measures the performance gap between πp and a NE strategy π∗p : e ( πp ) = up ( π ∗ p , π ∗ −p ) − up ( πp , BR ( πp ) ) . The average exploitability is = 1|P| ∑ p∈P e ( πp ) . 2.2 REINFORCEMENT LEARNING AND POLICY GRADIENT METHODS . RL usually assumes a Markov decision process , where the agent selects an action ai from the action set A ( si ) in state si ∈ S4 at each time step i . The agent then receives a reward ri from the environment and transitions to a new state si+1 . The objective in RL is to learn a policy that maximizes the expected discounted returns , i.e. , the state value , starting from any state s : V π ( s ) = Eπ [ ∑∞ j=i γ j−irj |si = s ] = Eπ [ G ] , with the discount factor γ ∈ [ 0 , 1 ) . A large family of methods in RL belong to policy gradient methods , in which the parameters θ of policy π ( a|s ; θ ) are updated by performing gradient ascent directly on Eπθ [ G ] . One early example is the standard REINFORCE algorithm ( Williams , 1992 ) that updates θ in the direction ∇θ log π ( a|s ; θ ) G , which is an unbiased estimate of ∇θEπθ [ G ] . To further reduce the variance of the gradient , an action-independent baseline is often subtracted from G : ∇θ log π ( a|s ; θ ) [ G−B ( s ) ] . A recent policy gradient method , named Advantage Actor-Critic ( A2C ) ( Mnih et al. , 2016 ) , learns a parameterized value function as the baseline : B ( s ) = V πθ ( s ; w ) , where w often shares some parameters with θ . Moreover , the value G−B ( s ) is replaced in A2C with the estimated advantage of action a in state s : A ( s , a ) = Q ( s , a ) − V ( s ) . Q ( s , a ) is usually estimated using sampled rewards and predicted values of future states . A2C updates parameters θ and w in a synchronous manner , while its asynchronous version is Asynchronous Advantage Actor-Critic ( A3C ) ( Mnih et al. , 2016 ) . In an asynchronous training environment , the behavioral policy is usually different from the learning policy . Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) takes this discrepancy into account by multiplying A2C gradient with an importance ratio r ( a|s ; θ ) = π ( a|s ; θ ) /π ( a|s ; θold ) , which results in r ( a|s ; θ ) ∇θ log π ( a|s ; θ ) A ( s , a ) = ∇θr ( a|s ; θ ) A ( s , a ) . Furthermore , PPO constrains the KL divergence between the learning policy πθ and the old behavioral policy πθold by clipping the ratio to a small interval around 1.0 .

Summary: This paper presents Actor-Critic Hedge (ACH): an actor-critic method for approximating Nash equilibrium strategies in large extensive-form games. ACH is an extension of the CFR family of algorithms that uses deep learning, and is able to learn model-free by training on trajectories and not full game traversals or subgames, which is common in much of the related CFR literature. The technique is demonstrated in toy poker domains commonly used in the literature (Kuhn and Leduc poker), Liar’s Dice, and in 1-on-1 Mahjong. In Mahjong, the ACH agent is shown to defeat several human players, including a Mahjong champion.