The Advantage Regret-Matching Actor-Critic

1 Introduction . The notion of regret is a key concept in the design of many decision-making algorithms . Regret minimization drives most bandit algorithms , is often used as a metric for performance of reinforcement learning ( RL ) algorithms , and for learning in games ( 3 ) . When used in algorithm design , the common application is to accumulate values and/or regrets and derive new policies based on these accumulated values . One particular approach , counterfactual regret ( CFR ) minimization ( 35 ) , has been the core algorithm behind super-human play in Computer Poker research ( 4 ; 25 ; 6 ; 8 ) . CFR computes an approximate Nash equilibrium by having players minimize regret in self-play , producing an average strategy that is guaranteed to converge to an optimal solution in two-player zero-sum games and single-agent games . We investigate the problem of generalizing these regret minimization algorithms over large state spaces in the sequential setting using end-to-end function approximators , such as deep networks . There have been several approaches that try to predict the regret , or otherwise , simulate the regret minimization : Regression CFR ( RCFR ) ( 34 ) , advantage regret minimization ( 17 ) , regret-based policy gradients ( 30 ) , Deep Counterfactual Regret minimization ( 5 ) , and Double Neural CFR ( 22 ) . All of these approaches have focused either on the multiagent or single-agent problem exclusively , some have used expert features , while others tree search to scale . Another common approach is based on fictitious play ( 15 ; 16 ; 21 ; 24 ) , a simple iterative self-play algorithm based on best response . A common technique is to use reservoir sampling to maintain a buffer that represents a uniform sample over past data , which is used to train a classifier representing the average policy . In Neural Fictitious Self-Play ( NFSP ) , this produced competitive policies in limit Texas Hold ’ em ( 16 ) , and in Deep CFR this method was shown to approach an approximate equilibrium in a large subgame of Hold ’ em poker . A generalization of fictitious play , policy-space response oracles ( PSRO ) ( 21 ) , stores past policies and a meta-distribution over them , replaying policies against other policies , incrementally adding new best responses to the set , which can be seen as a population-based learning approach where the individuals are the policies and the distribution is modified based on fitness . This approach only requires simulation of the policies and aggregating data ; as a result , it was able to scale to a very large real-time strategy game ( 33 ) . In this paper , we describe an approximate form of CFR in a training regime that we call retrospective policy improvement . Similar to PSRO , our method stores past policies . However , it does not store meta-distributions or reward tables , nor do the policies have to be approximate best responses , which can be costly to compute or learn . Instead , the policies are snapshots of those used in the past , which are retrospectively replayed to predict a conditional advantage , which used in a regret matching algorithm produces the same policy as CFR would do . In the single-agent setting , ARMAC is related to Politex ( 1 ) , except that it is based on regret-matching ( 14 ) and it predicts average quantities rather than explicitly summing over all the experts to obtain the policy . In the multiagent setting , it is a sample-based , model-free variant of RCFR with one important property : it uses trajectory samples to estimate quantities without requiring importance sampling as in standard Monte Carlo CFR ( 20 ) , hence it does not suffer from excessive variance in large environments . This is achieved by using critics ( value estimates ) of past policies that are trained off-policy using standard policy evaluation techniques . In particular , we introduce a novel training regime that estimates a conditional advantageWi ( s , a ) , which is the cumulative counterfactual regret Ri ( s , a ) , scaled by factor B ( s ) that depends on the information state s only ; hence , using regret-matching over this quantity yields the policy that CFR would compute when applying regret-matching to the same ( unscaled ) regret values . By doing this entirely from sampled trajectories , the algorithm is model-free and can be done using any black-box simulator of the environment ; hence , ARMAC inherits the scaling potential of PSRO without requiring a best-response training regime , driven instead by regret minimization . Problem Statement . CFR is a tabular algorithm that enumerates the entire state space , and has scaled to large games through domain-specific ( hand-crafted ) state space reductions . The problem is to define a model-free variant of CFR using only sampled trajectories and general ( domain-independent ) generalization via functional approximation , without the use of importance sampling commonly used in Monte Carlo CFR , as it can cause excessive variance in large domains . 2 Background . In this section , we describe the necessary terminology . Since we want to include the ( partiallyobservable ) multiagent case and we build on algorithms from regret minimization we use extensive-form games notations ( 29 ) . A single-player game represents the single-agent case where histories are aggregated appropriately based on the Markov property . A game is a tuple ( N , A , S , H , Z , u , τ ) , where N = { 1 , 2 , · · · , n } is the set of players . By convention we use i ∈ N to refer to a player , and −i for the other players ( N − { i } ) . There is a special player c called chance ( or nature ) that plays with a fixed stochastic strategy ( chance ’ s fixed strategy determines the transition function ) . A is a finite set of actions . Every game starts in an initial state , and players sequentially take actions leading to histories of actions h ∈ H. Terminal histories , z ∈ Z ⊂ H , are those which end the episode . The utility function ui ( z ) denotes the player i′s return over episode z . The set of states S is a partition of H where histories are grouped into information states s = { h , h′ , . . . } such that the player to play at s , τ ( s ) , can not distinguish among the possible histories ( world states ) due to private information only known by other players 1 . Let ∆ ( X ) represent all distributions over X : each player ’ s ( agent ’ s ) goal is to learn a policy πi : Si → ∆ ( A ) , where Si = { s | s ∈ S , τ ( s ) = i } . For some state s , we denote A ( s ) ⊆ A as the legal actions at state s , and all valid state policies π ( s ) assign probability 0 to illegal actions a 6∈ A ( s ) . We now show a diagram to illustrate the key ideas . Kuhn poker , shown in Figure 1 is a poker game with a 3-card deck : Jack ( J ) , Queen ( Q ) , and King ( K ) . Each player antes a single chip and has one more chip to bet with , then gets a single priavte card at random and one is left face down , and players proceed to bet ( b ) or pass ( p ) . Initially the game 1Information state is the belief about the world that a given player can infer based on her limited observations and may correspond to many possible histories ( world states ) starts in the empty history h0 = ∅ where no actions have been taken , and it is chance ’ s turn to play . Suppose chance samples , according to a fixed distribution , one of its six actions , which corresponding to one of the size-2 permutations of deals ( one card to each player ) . For example , suppose outcome 1Q2J is sampled , corresponding to the first player getting the queen and second player getting the jack . This would correspond to a new history h = ( 1Q2J ) . Label the information state corresponding to this history as s depicted by the grey joined circles : h′ = ( 1Q2K ) . At this information state s = { h , h′ } , it is the fist player ’ s turn ( τ ( s ) = 1 ) and it includes every history consistent with their information ( namely , that they were dealt the jack ) . The legal actions are now A ( s ) = { p , b } . Suppose the first player chooses p and the second player chooses b , then the history is part of s′ , the second information state shown in the figure . Finally , suppose the first player chooses to bet ( call ) , then the first player would win gaining 2 chips since they have the higher ranking card . Each player i ’ s goal is to compute πi that achieves maximal reward in expectation , where the expectation is taken over all players ’ policies , even though player i controls only their own policy . Hence , ideally , the player would learn a safe policy that guarantees the best worst-case scenario . Let π denote a joint policy . Define the state-value vπ , i ( s ) as the expected ( undiscounted ) return for player i given that state s is reached and all players follow π . Let qπ , i be defined similarly except also conditioned on player τ ( s ) taking action a at s. Formally , vπ , i ( s ) =∑ ( h , z ) ∈Z ( s ) η π ( h|s ) ηπ ( h , z ) ui ( z ) , where Z ( s ) are all terminal histories paired with their prefixes that pass through s , ηπ ( h|s ) = η π ( h ) ηπ ( s ) , where η π ( s ) = ∑ h′∈s η π ( h′ ) , and ηπ ( h , z ) is the product of probabilities of each action taken by the players ’ policies along h to z . The stateaction values qπ , i ( s , a ) are defined analogously . Standard value-based RL algorithms estimate these quantities for policy evaluation . Regret minimization in zero-sum games uses a different notion of value , the counterfactual value : vcπ , i ( s ) = ∑ ( h , z ) ∈Z ( s ) η π −i ( h ) η π ( h , z ) ui ( z ) , where ηπ−i ( h ) is the product of opponents ’ policy probabilities along h. We also write ηπi ( h ) the product of player i ’ s own probabilities along h. Under the standard assumption of perfect recall , we have that for any h , h′ ∈ s , ηπi ( h ) = ηπi ( h′ ) . Thus counterfactual values are formally related to the standard values ( 30 ) : vπ , i ( s ) = vcπ , i ( s ) β−i ( π , s ) , where β−i ( π , s ) = ∑ h∈s η π −i ( h ) . Also , qcπ , i ( s , a ) is defined similarly except over histories ( ha , z ) ∈ Z ( s ) , where ha is history h concatenated with action a. Counterfactual regret minimization ( CFR ) is a tabular policy iteration algorithm that has facilitated many advances in Poker AI ( 35 ) . On each iteration t , CFR computes counterfactual values qcπ , i ( s , a ) and vcπ , i ( s ) for each state s and action a ∈ A ( s ) and the regret of not choosing action a ( or equivalently the advantage of choosing action a at state s , rt ( s , a ) = qcπt , i ( s , a ) − vcπt , i ( s ) . CFR tracks the cumulative regrets for each state and action , RT ( s , a ) = ∑T t=1 r t ( s , a ) . Define ( x ) + = max ( 0 , x ) ; regret-matching then updates the policy of each action a ∈ A ( s ) as follows ( 14 ) : πT+1 ( s , a ) = NormalizedReLU ( RT , s , a ) = { RT , + ( s , a ) ∑ b∈A ( s ) R T , + ( s , b ) if ∑ b∈A ( s ) R T , + ( s , b ) > 0 1 |A ( s ) | otherwise , ( 1 ) In two-player zero-sum games , the mixture policy π̄T converges to the set of Nash equilibria as T →∞ . Traditional ( off-policy ) Monte Carlo CFR ( MCCFR ) is a generic family of sampling variants ( 20 ) . In outcome sampling MCCFR , a behavior policy µi is used by player i , while players −i use π−i , a trajectory ρ ∼ ( µi , π−i ) is sampled , and the sampled counterfactual value is computed : q̃cπ , i ( s , a | ρ ) = 1 η ( µi , π−i ) i ( z ) η ( µi , π−i ) i ( ha , z ) ui ( z ) , ( 2 ) if ( s , a ) ∈ ρ , or 0 otherwise . q̃cπ , i ( s , a | ρ ) is an unbiased estim . of qcπ , i ( s , a ) ( 20 , Lemma 1 ) . However , since these quantities are divided by η ( µ , π−i ) i ( z ) , the product of player i ’ s probabilities , ( i ) there can be significant variance introduced by sampling , especially in problems involving long sequences of decisions , and ( ii ) the ranges of the ṽci can vary wildly ( and unboundedly if the exploration policy is insufficiently mixed ) over iterations and states , which could make approximating the values in a general way particularly challenging ( 34 ) . Deep CFR and Double Neural CFR are successful large-scale implementations of CFR with function approximation , and they get around this variance issue by using external sampling or a robust sampling technique , both of which require a perfect game model and enumeration of the tree . This is unfeasible in very large environments or in the RL setting where full trajectories are generated from beginning to the end without having access to a generative model which could be used to generate transitions from any state .

Summary: This paper considers the problem of counterfactual regret minimization and proposes an algorithm that does not use the importance sampling procedure. The claim is that this helps in reducing the variance usually introduced by the IS procedure. They propose a new algorithm that uses the previously used policies as a buffer and replays those policies to learn a new policy. The algorithm is also claimed to be highly scalable for games with large state-action pairs.