Title: Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent
Authors: Yu Bai
Chi Jin
Song Mei
Ziang Song
Tiancheng Yu
Venue: NeurIPS 2022
Keywords: extensive-form games
regret minimization
correlated equilibria
multi-agent reinforcement learning
reinforcement learning theory

Abstract: A conceptually appealing approach for learning Extensive-Form Games (EFGs) is to convert them to Normal-Form Games (NFGs). This approach enables us to directly translate state-of-the-art techniques and analyses in NFGs to learning EFGs, but typically suffers from computational intractability due to the exponential blow-up of the game size introduced by the conversion. In this paper, we address this problem in natural and important setups for the \emph{$\Phi$-Hedge} algorithm---A generic algorithm capable of learning a large class of equilibria for NFGs. We show that $\Phi$-Hedge can be directly used to learn Nash Equilibria (zero-sum settings), Normal-Form Coarse Correlated Equilibria (NFCCE), and Extensive-Form Correlated Equilibria (EFCE) in EFGs. We prove that, in those settings, the \emph{$\Phi$-Hedge} algorithms are equivalent to standard Online Mirror Descent (OMD) algorithms for EFGs with suitable dilated regularizers, and run in polynomial time. This new connection further allows us to design and analyze a new class of OMD algorithms based on modifying its log-partition function. In particular, we design an improved algorithm with balancing techniques that achieves a sharp $\widetilde{\mathcal{O}}(\sqrt{XAT})$ EFCE-regret under bandit-feedback in an EFG with $X$ information sets, $A$ actions, and $T$ episodes. To our best knowledge, this is the first such rate and matches the information-theoretic lower bound.

TL;DR: We show a new class of algorithms for learning extensive-form games, arising from connections to normal-form games, admits efficient implementation and sharp rates. 

Review discussion:
I think the paper is interesting and well organized. The results are important and fit well within the current efforts on the topic and the existent literature. I also found the remark on KOMWU being equivalent on OMD+dilated entropy regularization very interesting.
Overall, I have a very positive opinion of the paper.
Originality: The paper presents original research and is technically strong. None of the results can be classified as groundbreaking, but at least one of the result was unexpected to me (equivalence between KOMWU and OMD+dilated regularizer), and the other results represent new state-of-the-art bounds.
Significance: the results are natural and well-motivated. I am confident that the paper will be well-received within the learning-in-games community.
Quality and clarity: the paper represents high-quality work. Both the body and the supplementary material are curated and easy to follow, at least for an expert in the field.
In the past, certain results have focused on model-free learning within the bandit feedback. Effectively, what that means is that the structure of the decision space of each player (treeplex) needs to be discovered as part of the learning process. See for instance [1] and [2]. Because the structure of the decision problem is not known upfront, it is not immediately clear that one can instantiate a balanced exploration strategy. Do you think that your approach could be used to address that setting as well?
[1] Farina and Sandholm. Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games.
[2] Kozuno, Menard, Munos, and Valko. Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall.
No unaddressed concerns.
No
Thank you for your valuable reviews and positive feedback to our paper! We respond to the questions as follows.
---“Model-free learning within bandit feedback… the decision space of each player (treeplex) needs to be discovered as part of the learning process…
for instance [1] and [2]... Do you think that your approach could be used to address that setting as well?”
Currently, our result for the bandit-feedback case (Algorithm 5, Theorem 7) does require knowing the structure of each player’s decision space (treeplex) beforehand, as the balanced exploration policies (Definition E.1) used in the Algorithm depends on that structure.
Therefore our algorithm does not yet address the unknown tree structure setting like [1,2]. We believe that extending to that setting, e.g. by “learning” an exploration policy on-the-fly that is as good as the balanced one for our purpose, would be an interesting open question.
I agree! Thanks for your response.
The paper is well-written and technically solid. The contributions span several interesting directions related to the problem of computing correlated equilibria in extensive-form games.
A small typo: eq. (1) is probably missing a summation.
None.
N/A
No
Thank you for your valuable reviews and positive feedback to our paper!
Re. the typo in Eq. (1): Right, the equation was missing a summation; we have added that in our revision.
Strengths:
The paper provides a solid contribution to the field of regret minimization in games. The writing is clear, overall resulting in a very good quality of presentation. As far as I'm concerned the theoretical claims are sound, novel, and non-trivial. The review of the related works seems accurate. For such reasons, I believe that this is a work that is ready to be published at a venue like NeurIPS.
Weaknesses:
Since the claims of the paper are those of an efficient algorithm, an experimental evaluation comparing their solution with previous state-of-the-art for computing EFCE (e.g., [1], [2]) could have been proposed.
However, I understand that the contribution that this paper intends to propose is mainly theoretical, hence I would classify such weakness as a minor.
[1] Farina et al. "Simple Uncoupled No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium".
[2] Celli et al. "No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium".
I do not have any question for the authors.
This work is theoretical, hence it has no potential negative societal impact.
No
Thank you for your valuable reviews and positive feedback to our paper!

Meta-review:
role: AC
data: metareview: All reviews are strongly positive for this paper. The reviewers appreciate the
many interesting contributions made by this paper. For my own part, I concur
with the reviewers: this is a very nice paper with several interesting results that deepen our understanding of several aspects of EFGs.
role: PC

Decision: Accept