Learning in two-player zero-sum partially observable Markov games with perfect recall

√ T where T is the number of played games . Moreover , IXOMD is computationally efficient as it needs to perform the updates only along the sampled trajectory . 1 Introduction . We study the setting of learning a Nash equilibrium ( NE , Nash Jr , 1950 ) in an imperfect information game ( IIG , Osborne and Rubinstein , 1994 ) . Precisely , we focus on two-player zero-sum IIG under the perfect-recall assumption ( Kuhn , 1953 ) . Perfect recall means that the players do not forget observations encountered or actions taken during the game . We model the game as a tabular , episodic of horizon H , partially observable Markov game ( POMG ) with a state space of size S , action spaces of size A and B for the max- and min-player respectively , and observation spaces ( i.e. , information set spaces , which are partitions of the state space ) of size X and Y for the max- and min-player . In learning by self play , we control both the max and min-player . After T episodes of the game we are asked to return a profile that is close to a NE in terms of exploitability gap ( Ponsen et al. , 2011 ) . Full feedback In case when we have perfect knowledge of the game ( i.e. , the transition probabilities and rewards ) there already exist several methods approximating the NE . The first line of work casts the setting through the sequence-form representation as a linear program which can be solved efficiently for games with moderate sizes of observation spaces X and Y ( Romanovsky , 1962 ; von Stengel , 1996 ; Koller et al. , 1996 ) . The sequence-from representation allows also to cast the setting as finding a saddle point ( Hoda et al. , 2010 ) . It is then possible to adapt first-order methods such as Nesterov ’ s smoothing ( Nesterov , 2005 ) and MirrorProx ( Nemirovski , 2004 ) to IIG , as done respectively by Hoda et al . ( 2010 ) ; Kroer et al . ( 2018 ) and Kroer et al . ( 2015 , 2020 ) . These methods have a rate of ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . convergence of order Õ ( ( X + Y ) /T ) , where Õ hides poly-log terms in eH , X , A , Y , B , T .2 Note that game-dependent exponential rate could also be obtained with first-order methods , see Gilpin et al . ( 2012 ) and Munos et al . ( 2020 ) . Another important line of work relies on minimizing the counterfactual regret ( Zinkevich et al. , 2007 ) . It uses an algorithm designed for adversarial bandits to locally minimize the regret of each player . A well-known example is CFR by Zinkevich et al . ( 2007 ) based on the regret-matching algorithm ( Hart and Mas-Colell , 2000 ; Gordon , 2007 ) . There exist many other variants of it , such as CFR+ ( Tammelin , 2014 ; Burch et al. , 2019 ) , see also Farina et al . ( 2019 , 2021a ) . These algorithms however only enjoy a ( known ) guarantee of convergence of order Õ ( ( X √ A + Y √ B ) / √ T ) . Note that the two last approaches require computing a full feedback : either some gradient for the first-order methods or the local regret for counterfactual regret minimization . Usually , this can be done by a complete traversal of the state space leading to a time-complexity of order O ( S ) . Sampling can reduce this time-complexity to O ( X + Y ) ,3 i.e. , we sample the transitions and the actions of the other player ; see for example the external-sampling MCCFR algorithm ( Lanctot et al. , 2009 ; Farina et al. , 2020 ) . Bandit feedback In this paper , we consider a more challenging setting where we only observe realizations of the games ( bandit feedback ) and do not have any prior knowledge of the game . Precisely , the rewards , the transition probabilities ( sometimes modeled as the policy of a chance player ) , the observation/state space , and its ( tree ) structure are unknown . Bandit feedback , model-based To deal with the limited bandit feedback , Zhou et al . ( 2020 ) consider model-based approach by using posterior sampling ( PS , Strens , 2000 ) to learn a model and then use the CFR algorithm in games sampled from the posterior . They obtain a convergence rate of order Õ ( max ( XA + Y B , √ S ) / √ T ) but only when the games are actually sampled according to the known prior . In addition , they still need to know the state space and its structure4 in order to instantiate the prior . Instead , Zhang and Sandholm ( 2021 ) rely on the principle of optimism in presence of uncertainty to incrementally build a model of the game . Then , they feed optimistic local regrets to a counterfactual regret minimizer algorithm such as the CFR algorithm . They prove a high-probability bound on the exploitability gap of order Õ ( ( X √ A+ Y √ B ) / √ T ) . Bandit feedback , model-free Our results follows another line of work which consider a modelfree approach . A well known algorithm of this type is outcome-sampling MCCFR ( Lanctot et al. , 2009 ; Farina et al. , 2020 ) , which builds an importance-sampling estimate of the counterfactual regret given exploration profile ( named balanced strategy by Farina et al. , 2020 ) . This exploration strategy should ensure that the players explore the information sets uniformly ( i.e. , such that all induced reach probabilities are lower-bounded by an absolute constant ) . Note that it is not clear how to find such an exploration profile without knowing the structure of the game.4 In particular , following the uniform distribution over the actions at each information set is not necessarily a good choice , e.g. , when the tree formed by the information set space is not balanced . This algorithm has a guarantee of order Õ ( ( X √ A+ Y √ B ) / √ T ) with high probability . Building on this idea , Farina and Sandholm ( 2021 ) propose to mix the exploration profile with one produced by a counterfactual regret minimizer such as CFR . They prove a high-probability bound on the exploitability gap of order Õ ( poly ( X , A , Y , B ) /T 1/4 ) . Note that this bound is a consequence of a bound on the regret of both players ( see Section 2 ) that holds even in the non-stochastic setting where an adversary picks a new game at each episode . Closer to our approach , Farina et al . ( 2021b ) recast the setting to an adversarial bandit linear optimization ( Flaxman et al. , 2005 ; Abernethy et al. , 2008 , see also Section 3.1 ) . Precisely , they use the online mirror descent ( OMD ) algorithm with the dilated entropy distance-generating function ( Hoda et al. , 2010 ; Kroer et al. , 2015 ) as regularizer . Then , OMD is fed with an estimate of the losses of the reformulated adversarial bandit linear instance . The estimator is a generalization of the typical one-point linear regression ( Dani et al. , 2008 ) . They obtain a rate of order Õ ( ( XA + Y B ) / √ T ) , which is , similarly as done by Farina and Sandholm ( 2021 ) , derived from a regret bound valid in the adversarial setting . However , their bound holds only in expectation and not in high probability . 2Therefore , we hide polynomial dependence on the horizon H . 3Note that O ( X + Y ) is at most O ( S ) . 4By structure we refer to the tree structure of the state space or observations spaces , see Section 2 . 1 Only in expectation according to a known prior on the game . 2 Only in expectation . To obtain high-probability bound , we instead propose to use an importance sampling estimator of the losses with implicit exploration ( Kocák et al. , 2014 ; Neu , 2015 ) . Indeed , the implicit bias of this estimator allows to effortlessly control the variance of the estimate , see Lattimore and Szepesvári ( 2020 , Chapter 12 ) for an in-depth discussion . Using this estimator , we give the Implicit Exploration Online Mirror Descent ( IXOMD ) based on OMD with the dilated entropy distance-generating function ( using uniform weights ) as a regularizer and add implicit exploration in the importance sampling estimator of the losses . Using our new analysis of this particular combination , we prove a highprobability bound on the exploitability gap of the average profile of order Õ ( ( X √ A+ Y √ B ) / √ T ) ; cf . Table 1 to see how our result compares to the prior work mentioned above . Precisely , our bound is obtained by bounding the regret of each player if they both follow the policy prescribed by IXOMD . Note that the regret bound , e.g. , of the max-player , of order Õ ( X √ AT ) , remains valid if the opponent ’ s policy and the game are picked by an adversary at each episode . IXOMD shares some similarities with the approach of Jin et al . ( 2020 ) designed for a different setting ( see Remark 1 ) . A notable difference is that we use the dilated entropy distance-generating function as a regularizer instead of the un-normalized Kullback-Leibler divergence ( Rosenberg and Mansour , 2019 ) . Our choice of regularizer allows an efficient update of the current policy with a O ( HA ) time-complexity per episode ( see Section 3.3 ) . In particular , our result answers the open problem raised by Farina et al . ( 2021b ) and Farina and Sandholm ( 2021 ) of providing an algorithm with high-probability regret bound scaling with √ T with O ( HA ) computations per episode . Interestingly , we can also update the average profile ( which will be returned at the end of the learning , see Section 3.3 ) in an online fashion . As consequence , IXOMD enjoys an overall time-complexity of O ( TH ( A+B ) + min ( TH , X ) A+ min ( TH , Y ) B ) and space-complexity of order O ( min ( TH , X ) A+ min ( TH , Y ) B ) . Moreover , IXOMD requires almost no prior knowledge of the game . In particular , we do not need to know the list of information sets in advance . We only require an oracle providing the possible actions at encountered information sets and a bound on A , B , and H to optimally5 tune the learning rate , see Remark 4 . We highlight our main contributions : • We give the IXOMD algorithm that learns a NE of an IIG in self-play with limited feedback . It has a provably high-probability convergence rate of order Õ ( ( X √ A+ Y √ B ) / √ T ) . The time-complexity of IXOMD is of order O ( TH ( A+B ) + min ( TH , X ) A+ min ( TH , Y ) B ) with a space-complexity of order O ( min ( TH , X ) A+ min ( TH , Y ) B ) . • If only one player follows IXOMD , e.g. , the max-player , then its regret is w.h.p . at most Õ ( X √ AT ) . The important property of our result is that it remains valid even if the policy and the game are picked by an adversary at each episode . Furthermore , the time-complexity 5Precisely , with this knowledge we obtain a regret bound , e.g . for the max-player , of order Õ ( X √ AT ) ; whereas we get Õ ( XA √ T ) without it . of IXOMD per episode is of order O ( HA ) . This answers an open problem of Farina et al . ( 2021b ) ; Farina and Sandholm ( 2021 ) . • IXOMD only needs to know the possible actions at the encountered information sets and a bound on A , B , and H to tune the learning rate . In particular , we do not need to know the list of information sets in advance .

Summary: This paper studies the problem of learning Nash equilibrium policies in two-player zero-sum extensive games, under the assumption of bandit feedback. The authors propose an algorithm that is based on Implicit Exploration and OMD, and they provide an high-probability regret bound that scales sub-linearly in the number of iterations and polynomially in the size of the game. Interestingly, the regret bound holds even in an online setting (not in self-play) in which both the policies of the opponent and the game structure are chosen adversarially.