Learning in two-player zero-sum partially observable Markov games with perfect recall

√ T where T is the number of played games . Moreover , IXOMD is computationally efficient as it needs to perform the updates only along the sampled trajectory . 1 Introduction . We study the setting of learning a Nash equilibrium ( NE , Nash Jr , 1950 ) in an imperfect information game ( IIG , Osborne and Rubinstein , 1994 ) . Precisely , we focus on two-player zero-sum IIG under the perfect-recall assumption ( Kuhn , 1953 ) . Perfect recall means that the players do not forget observations encountered or actions taken during the game . We model the game as a tabular , episodic of horizon H , partially observable Markov game ( POMG ) with a state space of size S , action spaces of size A and B for the max- and min-player respectively , and observation spaces ( i.e. , information set spaces , which are partitions of the state space ) of size X and Y for the max- and min-player . In learning by self play , we control both the max and min-player . After T episodes of the game we are asked to return a profile that is close to a NE in terms of exploitability gap ( Ponsen et al. , 2011 ) . Full feedback In case when we have perfect knowledge of the game ( i.e. , the transition probabilities and rewards ) there already exist several methods approximating the NE . The first line of work casts the setting through the sequence-form representation as a linear program which can be solved efficiently for games with moderate sizes of observation spaces X and Y ( Romanovsky , 1962 ; von Stengel , 1996 ; Koller et al. , 1996 ) . The sequence-from representation allows also to cast the setting as finding a saddle point ( Hoda et al. , 2010 ) . It is then possible to adapt first-order methods such as Nesterov ’ s smoothing ( Nesterov , 2005 ) and MirrorProx ( Nemirovski , 2004 ) to IIG , as done respectively by Hoda et al . ( 2010 ) ; Kroer et al . ( 2018 ) and Kroer et al . ( 2015 , 2020 ) . These methods have a rate of ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . convergence of order Õ ( ( X + Y ) /T ) , where Õ hides poly-log terms in eH , X , A , Y , B , T .2 Note that game-dependent exponential rate could also be obtained with first-order methods , see Gilpin et al . ( 2012 ) and Munos et al . ( 2020 ) . Another important line of work relies on minimizing the counterfactual regret ( Zinkevich et al. , 2007 ) . It uses an algorithm designed for adversarial bandits to locally minimize the regret of each player . A well-known example is CFR by Zinkevich et al . ( 2007 ) based on the regret-matching algorithm ( Hart and Mas-Colell , 2000 ; Gordon , 2007 ) . There exist many other variants of it , such as CFR+ ( Tammelin , 2014 ; Burch et al. , 2019 ) , see also Farina et al . ( 2019 , 2021a ) . These algorithms however only enjoy a ( known ) guarantee of convergence of order Õ ( ( X √ A + Y √ B ) / √ T ) . Note that the two last approaches require computing a full feedback : either some gradient for the first-order methods or the local regret for counterfactual regret minimization . Usually , this can be done by a complete traversal of the state space leading to a time-complexity of order O ( S ) . Sampling can reduce this time-complexity to O ( X + Y ) ,3 i.e. , we sample the transitions and the actions of the other player ; see for example the external-sampling MCCFR algorithm ( Lanctot et al. , 2009 ; Farina et al. , 2020 ) . Bandit feedback In this paper , we consider a more challenging setting where we only observe realizations of the games ( bandit feedback ) and do not have any prior knowledge of the game . Precisely , the rewards , the transition probabilities ( sometimes modeled as the policy of a chance player ) , the observation/state space , and its ( tree ) structure are unknown . Bandit feedback , model-based To deal with the limited bandit feedback , Zhou et al . ( 2020 ) consider model-based approach by using posterior sampling ( PS , Strens , 2000 ) to learn a model and then use the CFR algorithm in games sampled from the posterior . They obtain a convergence rate of order Õ ( max ( XA + Y B , √ S ) / √ T ) but only when the games are actually sampled according to the known prior . In addition , they still need to know the state space and its structure4 in order to instantiate the prior . Instead , Zhang and Sandholm ( 2021 ) rely on the principle of optimism in presence of uncertainty to incrementally build a model of the game . Then , they feed optimistic local regrets to a counterfactual regret minimizer algorithm such as the CFR algorithm . They prove a high-probability bound on the exploitability gap of order Õ ( ( X √ A+ Y √ B ) / √ T ) . Bandit feedback , model-free Our results follows another line of work which consider a modelfree approach . A well known algorithm of this type is outcome-sampling MCCFR ( Lanctot et al. , 2009 ; Farina et al. , 2020 ) , which builds an importance-sampling estimate of the counterfactual regret given exploration profile ( named balanced strategy by Farina et al. , 2020 ) . This exploration strategy should ensure that the players explore the information sets uniformly ( i.e. , such that all induced reach probabilities are lower-bounded by an absolute constant ) . Note that it is not clear how to find such an exploration profile without knowing the structure of the game.4 In particular , following the uniform distribution over the actions at each information set is not necessarily a good choice , e.g. , when the tree formed by the information set space is not balanced . This algorithm has a guarantee of order Õ ( ( X √ A+ Y √ B ) / √ T ) with high probability . Building on this idea , Farina and Sandholm ( 2021 ) propose to mix the exploration profile with one produced by a counterfactual regret minimizer such as CFR . They prove a high-probability bound on the exploitability gap of order Õ ( poly ( X , A , Y , B ) /T 1/4 ) . Note that this bound is a consequence of a bound on the regret of both players ( see Section 2 ) that holds even in the non-stochastic setting where an adversary picks a new game at each episode . Closer to our approach , Farina et al . ( 2021b ) recast the setting to an adversarial bandit linear optimization ( Flaxman et al. , 2005 ; Abernethy et al. , 2008 , see also Section 3.1 ) . Precisely , they use the online mirror descent ( OMD ) algorithm with the dilated entropy distance-generating function ( Hoda et al. , 2010 ; Kroer et al. , 2015 ) as regularizer . Then , OMD is fed with an estimate of the losses of the reformulated adversarial bandit linear instance . The estimator is a generalization of the typical one-point linear regression ( Dani et al. , 2008 ) . They obtain a rate of order Õ ( ( XA + Y B ) / √ T ) , which is , similarly as done by Farina and Sandholm ( 2021 ) , derived from a regret bound valid in the adversarial setting . However , their bound holds only in expectation and not in high probability . 2Therefore , we hide polynomial dependence on the horizon H . 3Note that O ( X + Y ) is at most O ( S ) . 4By structure we refer to the tree structure of the state space or observations spaces , see Section 2 . 1 Only in expectation according to a known prior on the game . 2 Only in expectation . To obtain high-probability bound , we instead propose to use an importance sampling estimator of the losses with implicit exploration ( Kocák et al. , 2014 ; Neu , 2015 ) . Indeed , the implicit bias of this estimator allows to effortlessly control the variance of the estimate , see Lattimore and Szepesvári ( 2020 , Chapter 12 ) for an in-depth discussion . Using this estimator , we give the Implicit Exploration Online Mirror Descent ( IXOMD ) based on OMD with the dilated entropy distance-generating function ( using uniform weights ) as a regularizer and add implicit exploration in the importance sampling estimator of the losses . Using our new analysis of this particular combination , we prove a highprobability bound on the exploitability gap of the average profile of order Õ ( ( X √ A+ Y √ B ) / √ T ) ; cf . Table 1 to see how our result compares to the prior work mentioned above . Precisely , our bound is obtained by bounding the regret of each player if they both follow the policy prescribed by IXOMD . Note that the regret bound , e.g. , of the max-player , of order Õ ( X √ AT ) , remains valid if the opponent ’ s policy and the game are picked by an adversary at each episode . IXOMD shares some similarities with the approach of Jin et al . ( 2020 ) designed for a different setting ( see Remark 1 ) . A notable difference is that we use the dilated entropy distance-generating function as a regularizer instead of the un-normalized Kullback-Leibler divergence ( Rosenberg and Mansour , 2019 ) . Our choice of regularizer allows an efficient update of the current policy with a O ( HA ) time-complexity per episode ( see Section 3.3 ) . In particular , our result answers the open problem raised by Farina et al . ( 2021b ) and Farina and Sandholm ( 2021 ) of providing an algorithm with high-probability regret bound scaling with √ T with O ( HA ) computations per episode . Interestingly , we can also update the average profile ( which will be returned at the end of the learning , see Section 3.3 ) in an online fashion . As consequence , IXOMD enjoys an overall time-complexity of O ( TH ( A+B ) + min ( TH , X ) A+ min ( TH , Y ) B ) and space-complexity of order O ( min ( TH , X ) A+ min ( TH , Y ) B ) . Moreover , IXOMD requires almost no prior knowledge of the game . In particular , we do not need to know the list of information sets in advance . We only require an oracle providing the possible actions at encountered information sets and a bound on A , B , and H to optimally5 tune the learning rate , see Remark 4 . We highlight our main contributions : • We give the IXOMD algorithm that learns a NE of an IIG in self-play with limited feedback . It has a provably high-probability convergence rate of order Õ ( ( X √ A+ Y √ B ) / √ T ) . The time-complexity of IXOMD is of order O ( TH ( A+B ) + min ( TH , X ) A+ min ( TH , Y ) B ) with a space-complexity of order O ( min ( TH , X ) A+ min ( TH , Y ) B ) . • If only one player follows IXOMD , e.g. , the max-player , then its regret is w.h.p . at most Õ ( X √ AT ) . The important property of our result is that it remains valid even if the policy and the game are picked by an adversary at each episode . Furthermore , the time-complexity 5Precisely , with this knowledge we obtain a regret bound , e.g . for the max-player , of order Õ ( X √ AT ) ; whereas we get Õ ( XA √ T ) without it . of IXOMD per episode is of order O ( HA ) . This answers an open problem of Farina et al . ( 2021b ) ; Farina and Sandholm ( 2021 ) . • IXOMD only needs to know the possible actions at the encountered information sets and a bound on A , B , and H to tune the learning rate . In particular , we do not need to know the list of information sets in advance .

Summary: This paper proposes Implicit Exploration Online Mirror Descent (IXOMD) algorithm that achieves sublinear regret in unknown extensive-form two-player zero-sum games with high probability. The setup is that player learn rewards only when actions are taken at some step, and do not learn about counterfactual reward they could have earned, have they made a different choice in the iteration. IXOMD is an adaptation of OMD, it uses importance sampling estimator of the losses with implicit exploration (to appropriately weigh the reward of the action taken and reduce its possibly high variance) and dilated entropy distance-generating function as a regularizer for the computation of next-step strategies. Authors prove that IXOMD achieves sublinear regret with appropriate setting of the algorithm hyperparameters - the learning rate and the exploration penalty. Thus the algorithm can be used to approximate Nash equilibria (if both players use the algorithm) with high probability.