Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent

Abstract: A conceptually appealing approach for learning Extensive-Form Games (EFGs) is to convert them to Normal-Form Games (NFGs). This approach enables us to directly translate state-of-the-art techniques and analyses in NFGs to learning EFGs, but typically suffers from computational intractability due to the exponential blow-up of the game size introduced by the conversion. In this paper, we address this problem in natural and important setups for the \emph{$\Phi$-Hedge} algorithm---A generic algorithm capable of learning a large class of equilibria for NFGs. We show that $\Phi$-Hedge can be directly used to learn Nash Equilibria (zero-sum settings), Normal-Form Coarse Correlated Equilibria (NFCCE), and Extensive-Form Correlated Equilibria (EFCE) in EFGs. We prove that, in those settings, the \emph{$\Phi$-Hedge} algorithms are equivalent to standard Online Mirror Descent (OMD) algorithms for EFGs with suitable dilated regularizers, and run in polynomial time. This new connection further allows us to design and analyze a new class of OMD algorithms based on modifying its log-partition function. In particular, we design an improved algorithm with balancing techniques that achieves a sharp $\widetilde{\mathcal{O}}(\sqrt{XAT})$ EFCE-regret under bandit-feedback in an EFG with $X$ information sets, $A$ actions, and $T$ episodes. To our best knowledge, this is the first such rate and matches the information-theoretic lower bound.

Reviews and author responses:

 I agree! Thanks for your response.

 Thank you for your valuable reviews and positive feedback to our paper! We respond to the questions as follows.

---“Model-free learning within bandit feedback… the decision space of each player (treeplex) needs to be discovered as part of the learning process… for instance [1] and [2]... Do you think that your approach could be used to address that setting as well?”  
Currently, our result for the bandit-feedback case (Algorithm 5, Theorem 7) does require knowing the structure of each player’s decision space (treeplex) beforehand, as the balanced exploration policies (Definition E.1) used in the Algorithm depends on that structure. Therefore our algorithm does not yet address the unknown tree structure setting like [1,2]. We believe that extending to that setting, e.g. by “learning” an exploration policy on-the-fly that is as good as the balanced one for our purpose, would be an interesting open question. 


 Thank you for your valuable reviews and positive feedback to our paper! 

Re. the typo in Eq. (1): Right, the equation was missing a summation; we have added that in our revision.

 Thank you for your valuable reviews and positive feedback to our paper!

 This work addresses the task of developing an efficient algorithm for $\Phi$-regret minimization in extensive-form games (EFGs). As a first step toward this goal, the authors consider the problem of regret minimization in the normal-form representation of the EFG that is known to have an exponential dimension with respect to the ATG size.  In such a scenario, they provide a computationally efficient version of the algorithm $\Phi$-Hedge that, under a suitable definition of the set $\Phi$ can be used to approximate equilibria like for instance the EFCE.
In the second part of the paper, the authors show that their version of $\Phi$-Hedge is equivalent to FTRL and OMD with suitably defined dilated entropies and divergences. Based on this equivalence, they recover near-optimal regret bounds for both the full feedback case and the bandit feedback one (in the latter case regret bounds are in high probability). Strengths: 
The paper provides a solid contribution to the field of regret minimization in games. The writing is clear, overall resulting in a very good quality of presentation. As far as I'm concerned the theoretical claims are sound, novel, and non-trivial. The review of the related works seems accurate. For such reasons, I believe that this is a work that is ready to be published at a venue like NeurIPS.

Weaknesses: 
Since the claims of the paper are those of an efficient algorithm, an experimental evaluation comparing their solution with previous state-of-the-art for computing EFCE (e.g., [1], [2]) could have been proposed. However, I understand that the contribution that this paper intends to propose is mainly theoretical, hence I would classify such weakness as a minor. 

[1] Farina et al. "Simple Uncoupled No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium". 
[2] Celli et al. "No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium". I do not have any question for the authors.  This work is theoretical, hence it has no potential negative societal impact.

 The paper studies the problem of computing an EFCE in extensive-form games. First, it presents an efficient implementation of $\Phi$-Hedge which minimizes a suitable notion of regret (i.e., trigger regret). Then, the authors show that such algorithm is equivalent to an OMD framework with dilated regularizers (EFCE-OMD). The authors provide regret upper bounds for EFCE-OMD in the full feedback and bandit feedback setting. Such bounds can be further improved via a modified version of the algorithm (Balanced EFCE-OMD) to match the information-theoretic lower bound. 
 The paper is well-written and technically solid. The contributions span several interesting directions related to the problem of computing correlated equilibria in extensive-form games. 

A small typo: eq. (1) is probably missing a summation. None. N/A

 The paper shows gives uncoupled no-regret dynamics for EFCE in extensive-form games with improved sample complexity both under full-information and (adversarial) bandit feedback. The regret bound in the full-information case matches concurrent work by Song et al.; the bandit case yields state-of-the-art results in that it applies even to adversarial opponents.

The proposed algorithm, from a conceptual standpoint, takes the extensive-form game and converts it into a normal-form game. Then, it applies the Phi-hedge algorithm for the choice of trigger deviation functions. The main hurdle is sidestepping the exponential blowup due to the normal-form conversion. In that sense, the paper provides a positive result that complements well the recent result by Farina et al. on kernelizability of OMWU (KOMWU) learning dynamics.

As an additional result, the authors point out that both KOMWU and their algorithm can be implemented in extensive-form games as specific instances of OMD with dilated entropy regularizers. I think the paper is interesting and well organized. The results are important and fit well within the current efforts on the topic and the existent literature. I also found the remark on KOMWU being equivalent on OMD+dilated entropy regularization very interesting. Overall, I have a very positive opinion of the paper.

Originality: The paper presents original research and is technically strong. None of the results can be classified as groundbreaking, but at least one of the result was unexpected to me (equivalence between KOMWU and OMD+dilated regularizer), and the other results represent new state-of-the-art bounds.

Significance: the results are natural and well-motivated. I am confident that the paper will be well-received within the learning-in-games community.

Quality and clarity: the paper represents high-quality work. Both the body and the supplementary material are curated and easy to follow, at least for an expert in the field. In the past, certain results have focused on model-free learning within the bandit feedback. Effectively, what that means is that the structure of the decision space of each player (treeplex) needs to be discovered as part of the learning process. See for instance [1] and [2]. Because the structure of the decision problem is not known upfront, it is not immediately clear that one can instantiate a balanced exploration strategy. Do you think that your approach could be used to address that setting as well?


[1] Farina and Sandholm. Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games.

[2] Kozuno, Menard, Munos, and Valko. Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall. No unaddressed concerns.

Meta-review: All reviews are strongly positive for this paper. The reviewers appreciate the
many interesting contributions made by this paper. For my own part, I concur
with the reviewers: this is a very nice paper with several interesting results that deepen our understanding of several aspects of EFGs.

Decision: Accept