Game-Theoretic Decision Making inImperfect-Information Games
Learning Dynamics, Equilibrium Computation, and Complexity
Gabriele Farina
CMU-CS-23-117
May 2023

Computer Science Department
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA

Thesis Committee
Tuomas Sandholm, Chair
Vincent Conitzer
Geoffrey Gordon
J. Zico Kolter
Avrim Blum (Toyota Technological Institute at Chicago)
Constantinos Daskalakis (Massachusetts Institute of Technology)

Submitted in partial fulfillment of the requirements
for the degree of Doctor of Philosophy

Copyright © 2023 Gabriele Farina
Parts of this research was sponsored by the National Science Foundation under grant numbers IIS-1718457, IIS-1617590, IIS-1901403, and CCF-1733556; the ARO under awards
W911NF-20-1-0081, W911NF-17-1-0082, and W911NF-22-1-0266; a 2019-2020 Facebook Research
fellowship. The views and conclusions contained in this document are those of the author andshould not be interpreted as representing the official policies, either expressed or implied, of any
sponsoring institution, the U.S. government or any other entity.

Keywords: computational game theory, extensive-form games, learning dynamics, equilibriumcomputation.

To the many scientists and scholars whose work laid the foundations for this dissertation.
Your ingenuity, innovation, and tireless pursuit of knowledge have been a great personal inspiration.

Abstract
Designing machines that can reason strategically and make optimal decisions even in thepresence of imperfect information and adversarial behavior is a fundamental goal of artificial
intelligence. Our understanding of game-theoretic decision-making is fairly advanced innormal-form games, strategic interactions in which all players act once and simultaneously,
selecting (possibly randomizing their choice) an action from a pre-determined finite setof possibilities. In contrast, this dissertation focuses on the more realistic imperfectinformation extensive-form games, tree-form decision-making problems in which decision
makers can face multiple decisions interleaved with observations about the way previous
decisions affected the (partially hidden and possibly adversarial) multiagent environment.
This dissertation seeks to close some recognized open problems in this area, and provide
solid theoretical and algorithmic foundations for strategic, game-theoretic decision-making
in imperfect-information extensive-form games. We highlight the following contributions.
• We close the long-standing open question of whether extensive-form correlated equilibria can arise from efficient, uncoupled learning dynamics, establishing unconditional
positive results.
• We settle the complexity of computing extensive-form perfect equilibria in two-player
games, a recognized open problem, showing it matches that of Nash equilibrium.
• We establish learning dynamics with state-of-the-art OT (log T /T ) convergence to coarse
correlated equilibrium, improving as a corollary known bounds for normal-form games.
• We uncover a kernelized version of the classic multiplicative weights update algorithm. This algorithm leads to learning dynamics for coarse correlated equilibria withtheoretical state-of-the-art dependence on the size of the imperfect-information game.
• We give fundamental results about the geometry of correlated strategies, identifying
new important classes of games where optimal (e.g., welfare-maximizing) correlated
solution concepts can be computed in polynomial time, yielding positive complexity
results for extensive-form correlation.
• We give the first learning algorithm with guaranteed convergence to welfare-maximizing
correlated solution concepts, and show their state-of-the-art empirical performance.
• We give state-of-the-art algorithms that enable, for the first time, finding exact tremblinghand refinements of Nash equilibrium in large games with up to half a billion terminal
states.
• We study the problem of computing optimal team strategies in zero-sum adversarial
team games, and give state-of-the-art algorithms and positive complexity results forthat setting.
• We give learning dynamics that can incorporate a model of the player to produce
strong, human-compatible strategies. These dynamics were a key component in
training artificial intelligence for the game of Diplomacy, a 7-player game involving
both cooperation and competition with humans.

Acknowledgments
Before delving into the contents of the dissertation, I would like to take a moment to express
my heartfelt appreciation to everyone who has been a part of my academic journey and has
contributed to the completion of this dissertation.
To my advisor, Tuomas Sandholm, thank you for the guidance, encouragement, and support
throughout my graduate studies. Your mentorship has helped me grow both as a researcher andas a person. I am grateful for the countless intellectually stimulating discussions we had, and forthe great deal of academic freedom you have always entrusted me with.
To my roommates over the years, Zach, Jay, and Camille, thank you for the laughter, support,
and fun memories we have shared. From impromptu BBQs to late-night chats, you havecomplemented grad school in so many ways. You have often challenged my perspective andhelped me adjust my focus on the things that matter. I was lucky to have you as a second family.
To Margarida, thank you for your unwavering love and support. Your patience, kindness, andhelp have been a source of inspiration and have meant the world to me. I am grateful for every
moment we have shared together.
To my committee members, colleagues, mentors, and peers, thank you for your feedback,insights, and collaborations. Your contributions have broadened my perspective and enriched myresearch. I am grateful for the friendships and connections we have formed along the way. I am
especially indebted to my mentors whose example and guidance lit the way: Christian, Nicola,
Noam, Vince, Zico, and Costis. A special thanks also to the many awesome collaborators I was
lucky to work with and learn from: Ioannis Anagnostides, Brian, Stephen, Haipeng, Ioannis
Panageas, Keegan, Misha, Sam, Chun Kai, Fei, John, Chung-Wei, Andrea, Andy, Alberto, Robin,
Luca, Federico, Tommaso, Jacob, Noah, Max, Marc, Anton, Adam, Alex, and more.
To the awesome Pittsburgh friends I have not mentioned yet, including Arman, Anna, Pietro,
Antonis, Luca, Derya, and more, thanks for all the fun, food, and drinks, and for your warmth,
company, and support. A special thanks also to my lab mates and office mates: Ellen, Sid, Carlos,
Brandon, and Roger.
To the Computer Science Department at CMU, and to Meta, thank you for your generous
financial support, which has allowed me to pursue my research interests and complete thisdissertation.
Finally, to my loving family I express my heartfelt gratitude for your steadfast support.

Contents

I

Introduction, notation, and background1

1

Introduction
3
1.1 Structure and summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 Research discussed in this dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Mathematical notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2Imperfect-information extensive-form games
2.1 Game trees and information sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 The player’s perspective: Tree-form decision processes . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Extracting a tree-form decision process from the game tree . . . . . . . . . . . . . . . .
2.2.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Strategies and sequence-form representation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Sequence-form representation of strategies . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2 Deterministic sequence-form strategies and Kuhn’s theorem . . . . . . . . . . . . . . .

17
18
21
22
24
26
27
29

3

No-regret learning in games
3.1 Hindsight rationality and Φ-regret . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 Definition of Φ-regret for a player in the repeated game . . . . . . . . . . . . . . . . .
3.1.2 Relationship between Φ-regret and game-theoretic equilibrium . . . . . . . . . . . . .
3.1.3 Feedback available to the learning player . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Mathematical abstraction of a predictive no-Φ-regret algorithm . . . . . . . . . . . . . . . . .
3.2.1 The canonical optimistic learning setup (COLS) for games . . . . . . . . . . . . . . . .
3.2.2 The important special case of external regret minimization . . . . . . . . . . . . . . . .
3.2.3 Reducing Φ-regret minimization to external regret minimization . . . . . . . . . . . .
3.2.4 Degrees of predictivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 No-external-regret algorithms for probability simplexes . . . . . . . . . . . . . . . . . . . . .
3.3.1 Multiplicative weights update (MWU) and its predictive variant (OMWU) . . . . . . . . . .
3.3.2 Regret matching (RM), regret matching+ (RM+ ), and variants . . . . . . . . . . . . . . .

31
32
32
33
34
35
38
39
39
40
41
41
44

ii

II
4

CONTENTS

Computation of coarse-correlated and Nash equilibria
Composability of learning dynamics and predictive counterfactual regret minimization√
Dynamics with OT (1/ T ) convergence to coarse-correlated equilibrium, linear-time iterations,state-of-the-art practical performance in many game instances

49

4.1
4.2

49
51
51
52
54
57
59
63
67
68

4.3

5

47

Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Regret circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Pictorial depiction of regret circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.2 Cartesian product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.3 Convex hull . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.4 Affine transformation and Minkowski sum . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.5 Scaled extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predictive counterfactual regret minimization paradigm . . . . . . . . . . . . . . . . . . . . .
4.3.1 Predictive RM (PRM) and predictive RM+ (PRM+ ) algorithms . . . . . . . . . . . . . . . . .
4.3.2 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Notions of distance for sequence-form strategies, and prox methods
Notion of distance for sequence-form strategies with both linear-time projections and polynomial
diameter • Exact polynomial-time algorithm for Euclidean projections • RVU-predictive dynamics
with linear-time iterations, OT (1/T 3/4 ) convergence to coarse-correlated equilibrium in multiplayer
games, and OT (1/T ) convergence to Nash equilibrium in two-player zero-sum games

71

5.1
5.2

Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2.1 Distance-generating functions and proximal setups . . . . . . . . . . . . . . . . . . . . 73
5.2.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.3 Online mirror descent and follow-the-regularized-leader . . . . . . . . . . . . . . . . . 75
5.2.4 Bilinear saddle points: Excessive gap technique and mirror prox . . . . . . . . . . . . 76
5.3 Euclidean distance-generating function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.3.1 Exact Euclidean projection algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.4 Distance-generating functions with linear-time projections . . . . . . . . . . . . . . . . . . . . 84
5.4.1 Dilated distance-generating functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.4.2 Dilatable global entropy distance-generating function . . . . . . . . . . . . . . . . . . 88
5.5 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.5.1 MPROX and EGT without aggressive stepsizing . . . . . . . . . . . . . . . . . . . . . . . 99
5.5.2 EGT with aggressive stepsizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.A Appendix: Properties of SMPL functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6Strongly predictive learning dynamics, and OT (log T /T ) convergence in self play
Strongly predictive dynamics for imperfect-information extensive-form games • State-of-the-art
OT (log T /T ) convergence to coarse-correlated equilibrium in self play
6.1
6.2
6.3

105

Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Setup and algorithm pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

CONTENTS

iii

6.4
6.5
6.6

Regret analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Connecting strong predictivity and logarithmic regret . . . . . . . . . . . . . . . . . . . . . . 111
Implementation and iteration complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.6.1 Local proximal oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.6.2 Linear maximization oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.7 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.A Appendix: Proof details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.A.1 Proof of Proposition 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.A.2 Proof of Proposition 6.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.A.3 Proof of Corollary 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.A.4 Proof of Theorems 6.2 and 6.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7

State-of-the-art regret dependence on game size via kernelization
Kernelized version of the multiplicative weights update algorithm enables reducing learning inimperfect-information extensive-form games to learning in simultaneous game • Dynamics withtheoretical state-of-the-art dependence on game size, OT (log4 T /T ) convergence to coarse-correlated
equilibria in self play, and linear-time iterations

131

7.1
7.2
7.3

Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
A natural reduction: Running OMWU on the vertices of the strategy set . . . . . . . . . . . . . . 133
Kernelized multiplicative weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.3.1 The sequence-form kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.3.2 Using the kernel to simulate the Vertex OMWU algorithm . . . . . . . . . . . . . . . . . . 136
7.4 Efficient evaluation of the sequence-form kernel . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.4.1 Worst-case linear complexity for a single evaluation . . . . . . . . . . . . . . . . . . . . 138
7.4.2 Batched computation and amortized complexity . . . . . . . . . . . . . . . . . . . . . 139
7.5 Regret bound of Kernelized OMWU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.6 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.A Appendix: Proof details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.A.1 Proof of Theorem 7.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.A.2 Proof of Proposition 7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

III
8Computation of extensive-form correlated and team equilibria

151

Uncoupled learning of extensive-form correlated equilibrium
Efficient, uncoupled no-regret learning dynamics for extensive-form correlated equilibria in imperfectinformation extensive-form games • Practical algorithm for EFCE

153

8.1
8.2

8.3

Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Extensive-form correlated equilibrium and its relation with Φ-regret . . . . . . . . . . . . . . 155
8.2.1 Trigger agents and trigger deviation functions . . . . . . . . . . . . . . . . . . . . . . . 155
8.2.2 Convergence to the set of EFCEs via no-Φ-regret dynamics . . . . . . . . . . . . . . . . 157
Construction of no-trigger-regret dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.3.1 Canonical trigger deviation matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.3.2 Structural decomposition of canonical trigger deviation matrices . . . . . . . . . . . . 163

iv

CONTENTS

8.3.3
8.4

Complete algorithm and analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Final remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

8.A Appendix: Inductive computation of fixed points of trigger deviation matrices . . . . . . . . 169
9

Geometry of correlated strategies, and positive complexity results for optimal EFCE

181

Positive complexity results around optimal correlated solution concepts in imperfect-information
extensive-form games • Enables computation of the polytope of expected utilities that can be reachedvia EFCE
9.1

Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

9.2

Preliminaries, notation, and prior work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.2.1

Polytope of correlation plans Ξ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

9.2.2

Optimal EFCE as a linear program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

9.2.3

The von Stengel-Forges polytope V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

9.2.4

von Stengel and Forges (2008)’s result for two-player games without chance . . . . . . 186

9.3

Characterization of the relationship between Ξ and V . . . . . . . . . . . . . . . . . . . . . . . 187

9.4

Scaled-extension-based structural decomposition for V . . . . . . . . . . . . . . . . . . . . . . 188
9.4.1

Examples and intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

9.4.2

Triangle-freeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

9.4.3

Two-player games with public chance moves are triangle-free . . . . . . . . . . . . . . 193

9.4.4

Computation of the decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

9.4.5

Integrality of the vertices of V in triangle-free games . . . . . . . . . . . . . . . . . . . 202

9.5

Beyond triangle-freeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

9.6

Experimental investigation of utilities reached by EFCE . . . . . . . . . . . . . . . . . . . . . . 205
9.6.1

Two-player general-sum games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

9.6.2

Three-player zero-sum games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

9.A Appendix: Additional lemmas on the structure of V . . . . . . . . . . . . . . . . . . . . . . . . 207
9.B Appendix: Optimal EFCE as a linear program . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
10 Learning optimal extensive-form correlated equilibria

213

Learning algorithm converging to optimal (e.g., social-welfare-maximizing) EFCE in imperfectinformation extensive-form games • Theoretical and practical state-of-the-art technique for learning
optimal EFCE
10.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
10.2 Optimal EFCE as a bilinear saddle-point problem . . . . . . . . . . . . . . . . . . . . . . . . . 214
10.3 No-regret learning algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
10.3.1 Construction of weakly-predictive algorithms via regret circuits (Chapter 4) . . . . . . 218
10.3.2 Faster rates using RVU-predictive algorithms (Chapter 5) . . . . . . . . . . . . . . . . 219
10.3.3 Remarks on last-iterate convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
10.4 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

CONTENTS

11 Learning dynamics for team coordination and collusionv

223

Models of team coordination and collusion • Practical state-of-the-art learning algorithm converging
to team maxmin equilibria with correlation device (TMECor) in two-team zero-sum games • Positive
complexity results to TMECor
11.1 Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
11.2 Failure of minmax theorem for team maxmin equilibrium . . . . . . . . . . . . . . . . . . . . 225
11.3 TMECor as a bilinear saddle-point problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
11.3.1 Realization vectors and low-dimensional parameterization . . . . . . . . . . . . . . . . 229
11.3.2 Connection with correlation plans and triangle-freeness . . . . . . . . . . . . . . . . . 231
11.4 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

IV

Beyond perfect rationality

12 Positive complexity results for trembling-hand perfect equilibria

235
237

Settles the complexity of computing extensive-form perfect equilibria in two-player games, showing it
is not harder than Nash12.1 Preliminaries on trembling-hand refinements . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
12.2 Contributions and related results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
12.3 Positive complexity results for two-player EFPE . . . . . . . . . . . . . . . . . . . . . . . . . . 240
12.3.1 Behavioral perturbation matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
12.3.2 EFPE as a trembling linear complementarity problem (LCP) . . . . . . . . . . . . . . . 242
12.3.3 Existence of a negligible positive perturbation (NPP) . . . . . . . . . . . . . . . . . . . 245
12.3.4 Computation of extensive-form perfect equilibria . . . . . . . . . . . . . . . . . . . . . 249
12.3.5 Polynomial-time computation in zero-sum games . . . . . . . . . . . . . . . . . . . . . 252
12.A Appendix: Undomination does not prevent sequential irrationality . . . . . . . . . . . . . . . 253
13 Computing exact trembling-hand refinements in two-player zero-sum games at scale

255

Scalable algorithm to compute exact trembling-hand refinements in large-scale games
13.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
13.2 Refined strategies as solutions to trembling linear programs . . . . . . . . . . . . . . . . . . . 256
13.2.1 Extensive-form perfect equilibria as trembling linear programs . . . . . . . . . . . . . 257
13.2.2 Quasi-perfect equilibria: Definition and formulation . . . . . . . . . . . . . . . . . . . 258
13.2.3 One-sided quasi-perfect equilibrium: Definition and formulation . . . . . . . . . . . . 260
13.2.4 Formulations with sparsified payoff matrices . . . . . . . . . . . . . . . . . . . . . . . 262
13.3 Basis stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
13.3.1 Analytic basis stability condition and existence of stable bases . . . . . . . . . . . . . . 264
13.3.2 Existence of stable bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
13.4 A practical algorithm for finding a TLP limit solution . . . . . . . . . . . . . . . . . . . . . . . 268
13.4.1 Basis-stability oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
13.4.2 Oracle for non-singular basis matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
13.4.3 Oracle for singular basis matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
13.4.4 Limit of strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

vi

CONTENTS

13.5 Experimental evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
13.5.1 Experiments on small and medium-sized benchmark games . . . . . . . . . . . . . . . 275
13.5.2 Experiments on real-world poker endgames . . . . . . . . . . . . . . . . . . . . . . . . 277
14 Quantal response and regularization towards human play
Learning dynamics biased towards to a given model of behavior • Learning dynamics for logit quantal
response equilibria in two-player zero-sum games • Construction of AI players for the game of no-press
Diplomacy279

14.1 Contributions and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
14.2 Logit quantal responses and KL-anchored responses . . . . . . . . . . . . . . . . . . . . . . . 282
14.2.1 Logit quantal responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
14.2.2 Logit quantal response as an instances of KL-anchored response . . . . . . . . . . . . 283
14.2.3 Learning dynamics for KL-anchored equilibria . . . . . . . . . . . . . . . . . . . . . . 284
14.2.4 Imitation-anchored responses in imperfect-information extensive-form games . . . . . 285
14.3 Modeling uncertainty on the anchoring coefficients . . . . . . . . . . . . . . . . . . . . . . . . 286
14.3.1 A technical lemma needed in the analysis . . . . . . . . . . . . . . . . . . . . . . . . . 288
14.3.2 Regret analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
14.3.3 Last-Iterate Convergence in Two-Player Zero-Sum Games . . . . . . . . . . . . . . . . 293
14.4 Experimental evaluation in no-press Diplomacy . . . . . . . . . . . . . . . . . . . . . . . . . . 300
14.4.1 Background on Double Oracle Reinforcement learning for Action exploration (DORA) 300
14.4.2 Training of our bot Diplodocus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
14.4.3 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
14.4.4 Performance compared to prior algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 302
14.4.5 Experiments against human players . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303

Conclusions and future work

305

Bibliography

308

Appendices

324

A Description of benchmark games used in experimental evaluations
325
A.1 Description of game instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
A.1.1 Battleship ( B ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
A.1.2 Liar’s dice ( D ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
A.1.3 Goofspiel ( G ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
A.1.4 Limited-information Goofspiel ( GL ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
A.1.5 Kuhn poker ( K ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
A.1.6 Leduc poker ( L ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
A.1.7 Pursuit-evasion ( P ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
A.1.8 River Endgame ( REL ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
A.1.9 Sheriff ( S ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

CONTENTS

vii

A.1.10 Double-dummy bridge endgame ( T , TP ) . . . . . . . . . . . . . . . . . . . . . . . . 329
A.1.11 Ridesharing game ( RS ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
A.1.12 Small matrix ( SM ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
A.2 Game dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
B Summary of notation
333
B.1 Tree-form decision processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
B.2 Correlated strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
B.3 Mathematical notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335

viii

CONTENTS

List of Algorithms

3.1
3.2
3.3
3.4
3.5

Predictive multiplicative weights update algorithm (OMWU) . . . . . . . . . . . . . . . . . . . . .
Regret matching (RM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Regret matching+ (RM+ ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Discounted regret matching (Discounted RM) . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linear regret matching (Linear RM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

42
44
44
45
45

4.1
4.2
4.3

52
55

4.5
4.6

Regret circuit for X × Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Regret circuit for co{X , Y} . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predictive CFR (weakly-predictive no-external-regret algorithm for sequence-form strategy
polytope) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
(Continued) Predictive CFR (weakly-predictive no-external-regret algorithm for sequence-form
strategy polytope) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predictive regret matching (PRM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predictive regret matching+ (PRM+ ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1
5.2

Predictive FTRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Predictive OMD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.1
6.2

Log-Regularized Lifted Optimistic FTRL (LRL-OFTRL) . . . . . . . . . . . . . . . . . . . . . . . . 107
Proximal Newton method (Tran-Dinh, Kyrillidis, and Cevher, 2015) . . . . . . . . . . . . . . . 115

7.1
7.2

Vertex OMWU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Kernelized OMWU (KOMWU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

8.1
8.2
8.3
8.4
8.5

No-external-regret algorithm Rσ̂ for set Λσ̂ := {Tσ̂→xσ̂ : xσ̂ ∈ Q≽j } . . . . . . . . . . . . . . . 164
No-regret algorithm R̃ for the set co Ψ = co{Λσ̂ : σ̂ ∈ Σ} . . . . . . . . . . . . . . . . . . . . . 166
No-trigger-regret algorithm for sequence-form strategy polytope Q . . . . . . . . . . . . . . . 167
Extend(T, J, j ∗ , x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
FixedPoint(T) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

9.1
9.2

FillOutRow((σ1 , σ2 ), j1 , S,
. .D)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
FillOutColumn((σ1 , σ2 ), j2 , S,
. D)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

4.4

65
66
68
68

x

LIST OF ALGORITHMS

9.3

Decompose((σ1 , σ2 ), S,. D). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

12.1 Find-EFPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
13.1 Naïve algorithm for finding a limit solution to a TLP P (ϵ). . . . . . . . . . . . . . . . . . . . . . 268
14.1 DiL-piKL algorithm, for a generic player i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

List of Figures

2.1

Game tree for the game of Kuhn poker. Utilities are shown for Player 1 only; utilities forPlayer 2 are the opposite Player 1’s. Due to space constraints, we used ‘chk.’ as an abbreviation
for the ‘check’ action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2

Tree-form decision process faced by Player 1 in the game of Kuhn poker. . . . . . . . . . . . . 22

2.3

Small game tree used in the example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.4

(Left) Tree-form decision process considered in the example. (Right) The constraints thatdefine the sequence-form polytope Q for Player 1 (besides nonnegativity) in the TFDP shownon the left. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5

(Right) Examples of deterministic sequence-form strategies for the small tree-form decision
process on the left. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1

Regret circuit for the image H(X ) of X under the affine transformation H. . . . . . . . . . . . 58

4.2

Regret circuit for the Minkowski sum X + Y.

. . . . . . . . . . . . . . . . . . . . . . . . . . . 59

h4.3

Regret circuit for the scaled extension X ◁ Y. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.4

Sequential decision-making problem used in the example. . . . . . . . . . . . . . . . . . . . . 63

4.5

Performance of PCFR+ , CFR+ , DCFR, and LCFR on nine games. In all plots, the x axis is thenumber of iterations of each algorithm. For each game, the top plot shows that the Nash gap
on the y axis (on a log scale), the bottom plot shows and the average prediction error (on a log
scale). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.1

Performance of the EGT algorithm instantiated with the two entropy DGFs across nine games.
The x-axis shows the number of EGT iterations, and the y-axis shows the distance to Nashequilibrium. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.2

Performance of the MPROX algorithm instantiated with the two entropy DGFs across nine games.100

5.3

Performance of the EGT/AS algorithm instantiated with the two entropy DGFs, as well as
aggressive stepsizing, µ balancing, and initial µ fitting. . . . . . . . . . . . . . . . . . . . . . . 101

6.1

The lifting operation performed by the LRL-OFTRL algorithm on the strategy set.

6.2

The regret of the players when they follow our learning dynamics, LRL-OFTRL. The x-axis
indexes the iteration, while the y-axis the regret. The scale on the x-axis is logarithmic. We
observe that the regret of each player grows as OT (log T ), verifying Theorem 6.3. . . . . . . . 117. . . . . . . 108

xii

LIST OF FIGURES

7.1
7.2
7.3

Overview of reduction template from Q to ∆Π . The matrix V has the vertices Π of Q as columns.134
Experimental comparison of KOMWU with CFR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Experimental comparison of KOMWU with DOMWU for different choices of learning rates. . . . . . 144

8.1

Canonical trigger deviation matrices. Entries shaded with dark gray represent the entries ofthe matrix defined in the second case of Equation (8.5). Given trigger sequence σ̂ ∈ Σ∗ , allindices (σr , σc ) such that σr , σc ≽ σ̂ are shaded with light gray. . . . . . . . . . . . . . . . . . 161
Pictorial depiction of our no-(co Ψ)-reget algorithm for the set of sequence-form strategies Q.
For notational convenience we let Σ∗ := {1, . . . , m}. . . . . . . . . . . . . . . . . . . . . . . . . 168

8.2

9.1
9.2

9.3

9.4

Overview of the connections among this chapter’s results. . . . . . . . . . . . . . . . . . . . . 189
Three examples of extensive-form games with increasingly complex information partitions.
As usual, the crossed nodes belong the chance player, the black round nodes belong toPlayer 1, the white round nodes belong to Player 2, the gray round sets define information
sets, and the white squares denote terminal nodes (payoffs are omitted as they are irrelevant).
The numbers along the edges define the action names; for clarity we assign unique action
names at each information set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Polytope of expected utilities for the players that can be reached via extensive-form correlated
equilibria in three standard two-player general-sum imperfect-information extensive-form
games. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Polytope of expected utilities for the players that can be reached via extensive-form correlated
equilibria in three standard three-player zero-sum imperfect-information extensive-form games.207

11.1 Matching pennies game. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
12.1 Perfect-information game in which a sequentially-irrational Nash equilibrium is highlighted. 237
12.2 Small imperfect-information game in which a sequentially-irrational Nash equilibrium ishighlighted. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
12.3 Modified Guess-the-Ace games. The highlighted equilibrium is undominated, and yet
sequentially irrational. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
13.1 High-level overview of the steps of our practical algorithm for finding a TLP limit solution. . 269
13.2 Example of a situation where the basis matrix B(ϵ) is singular at ϵ = 0. . . . . . . . . . . . . 273
A.1 The graph on which the search game is played. . . . . . . . . . . . . . . . . . . . . . . . . . . 327
A.2 Left: Graph configuration 1, used for RS212 RS213 . Right: Graph configuration 2, used
for RS222 RS223 . In both cases the position of the two drivers is randomly chosen at the
beginning of the game, edge costs are unitary, and the reward for each node is indicatedbetween curly brackets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

List of Tables

2.1

Summary of basic notation for TFDPs. In cases where it is important to specify the player towhich the different quantities belong, a subscript with the player will be added. . . . . . . . . 26

4.1

Bottom-up construction rules for sequence-form strategy spaces. ei denotes the i-th indicator
vector, that is, the vector whose entries are all 0 except for the entry in position i, which is set
to 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

5.1(A) Various measures of the size of each of the games that we test algorithms on. (B), (C) The
magnitude of the dilated entropy DGF and dilatable global entropy DGF weights. . . . . . . 97

7.1

Properties of various no-regret algorithms for imperfect-information extensive-form games.
All algorithms take linear time to perform an iteration. The first set of rows are for nonpredictive algorithms. The second set of rows are for predictive algorithms. The regretbounds are per player and apply to multiplayer general-sum games. They depend on the
maximum number of actions A available at any decision node, the maximum ℓ1 norm
∥Q∥1 = maxq∈Q ∥q∥1 over the player’s sequence-form strategy polytope Q, the depth Dof the decision polytope, and the number of players m. Optimistic algorithms have better
asymptotic regret, but worse dependence on the game constants m, A, and ∥Q∥1 . Notethat our algorithms achieve better dependence on ∥Q∥1 compared to all existing algorithms.
†
Last-iterate convergence results are for two-player zero-sum games, and some results relyon the assumption of a unique Nash equilibrium—see Section 7.5 for details. ‡ We remarkthat LRL-OFTRL enjoys polynomial-time iterations (ignoring a log log T dependence), but notlinear-time iterations unlike the other methods in the table. ∗ See C.-W. Lee, Kroer, and Luo(2021). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

9.1

Additional notation used when dealing with correlation of strategy spaces. . . . . . . . . . . 183

10.1 Experimental comparison between our learning-based approach (‘Ours’, Section 10.3.1) andusing the commercial LP solver Gurobi on the linear programming formulation of optimal
EFCE (Proposition 9.1), both for computing an optimal EFCE within an optimality andfeasibility tolerance set to 1% of the payoff range of the game. . . . . . . . . . . . . . . . . . . 221

xiv

LIST OF TABLES

10.2 Experimental comparison between our learning-based approach (‘Ours’, Section 10.3.1) andusing the commercial LP solver Gurobi on the linear programming formulation of optimal
EFCE (Proposition 9.1), both for computing an optimal EFCE within an optimality andfeasibility tolerance set to 0.1% of the payoff range of the game. . . . . . . . . . . . . . . . . . 222
11.1 Comparison between TME, TMECor and Team Nash equilibrium. . . . . . . . . . . . . . . . . 224
11.2 Runtime of our learning algorithm (column ‘This chapter’), compared to prior state-of-the-art
algorithms based on linear programming (‘ZS22’, B. H. Zhang and Sandholm, 2022b) andcolumn generation (‘ZFCS22’, B. H. Zhang, Farina, Celli, and Sandholm, 2022) respectively,
on several standard parametric benchmark games. ‘—’: Missing or unknown value. . . . . . 233
12.1 Complexity of computing trembling-hand equilibrium refinements in two-player games. . . 240
13.1 Comparison between different methods of computing Nash equilibrium refinements in
two-player zero-sum small and medium-sized benchmark games benchmark games. . . . . . 276
13.2 Unsparsified and sparsified size of River endgames encountered by Libratus during the“Brains vs AI” competition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
13.3 Computation of refined Nash equilibria in real poker endgames using our algorithm. . . . . 278
14.1 Performance of different algorithms. Agents above the line were retrained. Agents below theline were evaluated using the models and the parameters provided by the authors. The ±shows one standard error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
14.2 Performance of four different agents in a population of human players, ranked by Elo, among
all 43 participants who played at least 5 games. The ± shows one standard error. . . . . . . . 304
A.1 Dimensions of game instances used in this dissertation. . . . . . . . . . . . . . . . . . . . . . . 331
A.2 (Continued) Dimensions of game instances used in this dissertation. . . . . . . . . . . . . . . 332

Part I

Introduction, notation, andbackgroundChapter 1

Introduction
Designing machines that can reason strategically and make optimal decisions even in the presenceof imperfect information and adversarial behavior is a fundamental goal of artificial intelligence.
The study of strategic decision-making is a rich discipline involving models and techniques fromgame theory, optimization, statistics, and others, and with applications ranging from markets(including financial markets, auctions, etc.), to defense (including cybersecurity, airport security,
patrolling routes, etc.), to recreational games, to logistics and procurement, and more.
Our understanding of the computational aspects of game-theoretic decision-making is fairly
advanced in strategic interactions in which all players act once and simultaneously, selecting(possibly randomizing their choice) an action from a pre-determined finite set of possibilities.
These interactions, called normal-form games, are arguably the best studied in game theory, andhave been the standard model in the theory of learning in games and online learning moregenerally for a long time.
In contrast, this dissertation focuses on the more realistic and challenging tree-form decisionmaking problems in which decision makers can face multiple decisions interleaved with observations about the way previous decisions affected the (partially hidden and possibly adversarial)
multiagent environment. These games go under the name of imperfect-information extensive-form
games—where the term extensive-form is a standard term in the literature, meaning tree-form—andcan model both sequential and simultaneous moves, stochastic events (such as drawing a random
card from a deck), and uncertainty about the state of the game. By allowing interleaved decisions
and observations, imperfect-information extensive-form games present unique challenges compared to (and require different techniques from) normal-form games. For one, due to the presenceof observations, the number of strategies in imperfect-information extensive-form games is usually
exponential in the number of possible choices, resulting in additional computational challenges
and a more nuanced complexity landscape than interactions that terminate after a single action.
I believe that tackling the added complexity of imperfect-information extensive-form games

4§1. INTRODUCTION

is necessary to bring strategic decision-making to the real world, where unlike single-choiceinteractions, a decision maker rarely has to face just one choice, much less independently of any
observation about the state of the environment.
This dissertation seeks to provide solid theoretical and algorithmic foundations for strategic,
game-theoretic decision-making in imperfect-information extensive-form games. We identify
three main areas of contributions (a more detailed description of the contributions of each chapter,as well as of the dependency relations among the chapters is given below, in Section 1.1).

New and state-of-the-art learning dynamics for imperfect-information games Learning
dynamics provide positive, constructive answers to the following fundamental question: Can aplayer that repeatedly plays a game follow rules to refine their own strategy after each match,so as to guarantee reaching a form of game-theoretic equilibrium in the long run? Due to their
uncoupled nature, learning dynamics are typically extremely practical methods to compute
game-theoretic solutions to large games. While the existence of powerful learning dynamics
leading to the most important notions of game-theoretic equilibrium in general normal-form
games (such as correlated and coarse-correlated equilibrium) had been a celebrated result for along time, much less was known in general imperfect-information games.
This dissertation provides several contributions to the theory of learning dynamics inmultiplayer imperfect-information games. Here we mention four; more work on learning inimperfect-information extensive-form games is mentioned later in this section.
• We construct the first efficient (i.e., with polynomial-time iteration complexity) learning
dynamics for extensive-form correlated equilibrium in imperfect-information extensive-form
games. These dynamics resolve a long-standing open problem in the theory of learningin imperfect-information extensive-form games, and beget the practical state of the art
for computing extensive-form correlated equilibria in multiplayer imperfect-information
extensive-form games.
• We show that learning coarse-correlated equilibria in imperfect-information extensive-form
games can be efficiently reduced to learning in normal-form games by means of a kernelizedversion of the classic multiplicative weights update algorithm. This reduction enablestransferring results that were previously known to hold only in normal-form games togeneral imperfect-information extensive-form games, thereby shortening the gap between
the two domains. The existence of such a reduction contradicts decades of popular beliefs,
and leads to regret bounds with state-of-the-art dependence on the game size, improving
on all prior learning algorithms for imperfect-information extensive-form games. This
technique is also the first to show that OT (log4 T /T ) convergence to coarse-correlated
equilibrium can be achieved in imperfect-information extensive-form games while retaining
linear-time strategy updates. Before this result, the bound was only known to be possible for§1. INTRODUCTION

5normal-form games (Daskalakis, Fishelson, and Golowich, 2021), and the best-known result√for imperfect-information extensive-form games was of order OT (1/ T ), exponentiallyworse.
• Using a different technique, we further improve the rate of convergence to coarse-correlated
equilibrium in both imperfect-information extensive-form games and normal-form gameswith any number of players. Specifically, we construct learning dynamics leading to coarsecorrelated equilibrium in multiplayer imperfect-information extensive-form games at thetheoretical state-of-the-art, near-optimal rate of OT (log T /T ), while enjoying OT (log log T )per-iteration complexity. We remark that the technique we use to establish the result applies
to the larger class of concave games (of which imperfect-information extensive-form gamesand normal-form games are instances), establishing, for the first time, the existence ofnear-optimal polynomial-time learning dynamics for any such games.
• Finally, we propose new learning dynamics with state-of-the-art empirical convergence
to Nash equilibrium in a large number of two-player zero-sum imperfect-information
extensive-form games.

Structure of correlated play in imperfect-information extensive-form games Several problems
of interest, including the computation of welfare-maximizing correlated solution concepts andoptimal strategies for teams in the absence of communication, can be formulated as linear
optimization problems over the set of correlated strategies of (possibly subsets of) the players. To
enable progress on those problems, a part of this dissertation focuses on developing new techniqueand foundational results regarding the structure and geometry of correlated distributions of play
between players. Here, we highlight the most significant findings.
• We give positive complexity results for the problem of optimizing a linear function over thepolytope of correlated strategies between two players. Specifically, we isolate a condition—
called triangle freeness and automatically satisfied by all two-player games whose chancemoves are publicly observed—that guarantees that the set of correlated strategies can be
represented via polynomially many linear constraints, rendering optimization tractable.
When triangle freeness is not satisfied, we also discuss how the techniques can be generalized
to yield state-of-the-art parameterized complexity results. These complexity thresholds are
unique to imperfect-information extensive-form games and do not have an equivalent innormal-form games, highlighting the intricate and fascinating structure of the former. We
also remark that the techniques we introduce in this dissertation also enable to study, for the
first time, what equilibrium points can be supported by extensive-form correlated equilibria,
highlighting the richness of behavior that arises from introducing a correlation device inimperfect-information extensive-form games.

6§1. INTRODUCTION• We leverage our results regarding the structure of the polytope of correlated strategies toconstruct the first no-regret learning algorithm with guaranteed convergence to the set of
optimal (for example, welfare-optimizing) extensive-form correlated equilibria, leading to
the practical state-of-the-art algorithm for the problem.
• Finally, we draw new connections between the study of correlated strategies and the studyof optimal strategies in two-team zero-sum imperfect-information extensive-form games.
By leveraging these connections, we give the first positive complexity result for the latterproblem, as well as the practical state-of-the-art learning algorithm.

Learning and equilibrium computation in the presence of imperfect players Finally, the lastpart of this dissertation focuses on learning and computational techniques for imperfect (that is,
not perfectly rational) players.
• We settle the complexity of computing extensive-form perfect equilibria in two-player games,a recognized open problem (Miltersen and Sørensen, 2006), showing that it is not harderthan computing a generic Nash equilibrium. Extensive-form perfect equilibria (EFPEs) are
trembling-hand equilibrium refinements, subsets of Nash equilibria that satisfy some additional
robustness guarantees to small mistakes of the opponents. In particular, EFPEs guaranteethat the equilibrium strategies are optimal even in parts of the game tree that are reachedonly if a player has made a mistake, and are therefore appealing concepts for AI players thatneed to be play against human players.
• We also give practical state-of-the-art algorithms that enable, for the first time, to find exact
trembling-hand refinements of Nash equilibrium (including EFPE and other) at scale in twoplayer zero-sum imperfect-information extensive-form games. As we show, our algorithm isable to scale to real games with up to half a billion terminal nodes, an unprecedented scalefor equilibrium refinements.
• Finally, we introduce a methodology for constraining learning dynamics to remain close to agiven anchor policy for each player. As a special case, such anchored learning algorithms canbe used to compute logit quantal response equilibria in imperfect-information extensive-form
games. We empirically investigate the use of these learning algorithms in the constructionof AI agents for the game of no-press Diplomacy, showing that our approach enables the AI
agents to comply with human conventions. In a 200-game no-press Diplomacy tournament
involving 62 human participants spanning skill levels from beginner to expert, two AI botstrained with our algorithm both achieved a higher average score than all other participants
who played more than two games, and ranked first and third according to an Elo ratingsmodel. We also remark that the same methodology was recently used in building Cicero, anAI agent for playing the full version of Diplomacy using natural language communication.

§1. INTRODUCTION

1.1

7Structure and summary of contributions

We now summarize the structure and main contributions of each chapter of this dissertation. A
detailed list of contributions presented in each chapter is placed at the beginning of the chapter
itself, together with a discussion of the most relevant related research.
Chapters 2 and 3 give an approachable overview of imperfect-information extensive-form games
and learning in games. When presenting the key ideas of the theory of learning in games,
we opted for a modern approach, introducing the concept of predictivity early on as astandard aspect in the definition of a no-regret algorithm rather than as a later development.
Chapter 4 introduces regret circuits, a methodology for constructing predictive no-regret algorithms for composite sets obtained via convexity-preserving operations. We use theframework to construct a predictive version of the counterfactual regret minimization (CFR)
algorithm, a no-external-regret and parameter-free algorithm for the strategy set of a player
in an imperfect-information extensive-form games. We find that our predictive algorithms
obtained via regret circuits achieve state-of-the-art empirical convergence to the set of Nashequilibria in two-player zero-sum games in a majority of instances we test on. In addition,
the material presented in the chapter will find applications in several other chapters, suchas in the construction of the first learning dynamics for extensive-form correlated equilibria(Chapter 8), welfare-maximizing equilibria (Chapter 10), and team strategies (Chapter 11).
Chapter 5 investigates several fundamental questions about the metric structure of strategy
spaces in imperfect-information extensive-form games, with applications to online andoffline optimization methods. The contributions are multiple, spanning different notionsof distance between strategies that arise in practice, and laying foundations applicable toa wide range of optimization techniques. Perhaps most notably, the chapter introduces
the first notion of distance in imperfect-information games that enables projecting points
onto the strategy polytope in linear time (a key operation needed in projected optimization
methods such as mirror descent or projected gradient descent), while at the same time
guaranteeing polynomial (in the game tree size) distance between any two strategies.
Chapter 6 provides state-of-the-art regret bounds for learning in imperfect-information extensiveform games with an arbitrary number of players, constructing learning dynamics withguaranteed convergence to coarse correlated equilibria at a rate of OT (log T /T ), all while
enjoying an OT (log log T ) iteration complexity. This settles, for the positive, the question ofwhether near-optimal no-regret learning can be achieved in general imperfect-information
extensive-form games, and improves the state of the art for nonsequential games as well. To
establish the result, we design new techniques that are of independent interest, as they apply
to the larger class of concave games, which contains imperfect-information extensive-form

8§1.1. STRUCTURE AND SUMMARY OF CONTRIBUTIONS

games. This chapter depends on certain parts of Chapter 5 as prerequisites, as it builds
on new results related to efficient proximal updates for strategies in imperfect-information
extensive-form games.
Chapter 7 shows that learning in imperfect-information extensive-form games can be efficiently
reduced to learning in nonsequential games by means of a kernelized version of the classic
multiplicative weights update algorithm. This reduction enables transferring, in a black-box
fashion, results that were previously known to apply only in nonsequential games to generalimperfect-information extensive-form games, thereby shortening the gap between the twodomains. The existence of such a reduction might come as a surprise, as it contradicts
decades of popularly held beliefs. Even more surprisingly, it leads to regret bounds withstate-of-the-art dependence on the game size, improving on all prior algorithms, despite
these being designed specifically for imperfect-information extensive-form games. The
algorithm presented in this chapter is also the first to show that polylogarithmic regret canbe achieved in imperfect-information extensive-form games while retaining linear-time
strategy updates.
Chapter 8 provides the first uncoupled, polynomial-time learning dynamics leading to extensiveform correlated equilibrium (EFCE), the natural generalization of correlated equilibrium toimperfect-information extensive-form games. The material presented in this chapter closes
an longstanding open problem in the literature, and parts of it were recognized with apaper award at NeurIPS and publication in the Journal of the ACM. This chapter makes useof the formalism of regret circuits introduced in Chapter 4.
Chapter 9 focuses on the structure and geometry of correlated distributions of play between
players. As we show in this chapter and in the following ones, several problems of interest,
including the computation of welfare-maximizing correlated solution concepts and optimal
strategies for teams in the absence of communication, can all be formulated as linear
optimization problems over the set of correlated strategies of (possibly subsets of) the
players. One of the key results of the chapter is establishing new positive description
complexity results, isolating a condition—called triangle freeness and automatically satisfied
by all two-player games whose chance moves are publicly observed—that guarantees thatthe set of correlated strategies can be represented via polynomially many linear constraints,
rendering optimization tractable. Along the way, several new concepts and techniquesof likely independent interest are introduced to study the combinatorial structure ofcorrelation. Beyond triangle freeness, we also discuss how the ideas of the chapter can beextended to yield state-of-the-art parameterized complexity results for those optimization
problems over correlated strategies. The complexity thresholds presented in this chapter
are very different from nonsequential games and highlight the intricate and fascinating
structure of imperfect-information extensive-form games. At the end of the chapter, we use§1. INTRODUCTION

9

our techniques to study, for the first time, what equilibrium points can be supported byextensive-form correlated equilibria, highlighting the richness of behavior that arises fromintroducing a correlation device in imperfect-information extensive-form games.
Chapter 10 combines the structural understanding of correlated strategies gained from Chapter 9
together with the formalism of regret circuits introduced in Chapter 4 to design thefirst learning algorithm with guaranteed convergence to optimal (for example, welfaremaximizing) extensive-form correlated equilibrium in imperfect-information extensive-form
games. As we show in the experimental evaluation at the end of the chapter, the use oflearning begets the practical state-of-the-art approach for computing optimal correlated
solution concepts in multiplayer imperfect-information extensive-form games.
Chapter 11 studies the problem of constructing optimal strategies in two-team zero-sum games,
where the players cannot communicate during the game. After discussing alternative
notions of game-theoretic optimality for the setting, we isolate one, TMECor, as the mostappropriate for the setting. Then, we establish strong ties between the computation of
TMECor strategies for a team and the geometry of correlation plans studied in Chapter 9.
By leveraging that connection, we provide the current state-of-the-art complexity result forthe problem of computing optimal team strategies, as well as the practical state-of-the-art
learning algorithm for TMECor. This chapter combines ideas presented in Chapters 4, 9
and 10.
Chapter 12 settles the complexity of computing an extensive-form perfect equilibrium—arguably
the best-known and most-studied sequentially-rational refinement of the Nash equilibrium—in two-player games, a recognized problem. Extensive-form perfect equilibrium (EFPE)
resolves certain unsatisfactory behaviors that can be prescribed by Nash equilibrium when
the opponent makes a mistake. This is especially relevant when using Nash equilibriumas the optimization target for AI bots playing games against humans. Our results show
a positive result: computing an EFPE is not harder, from a computational complexity
standpoint, than computing a Nash equilibrium, showing that the benefits of EFPE come at
no additional (theoretical) cost. However, we remark that the technique we use to establish
the complexity result in this chapter is hardly scalable, and devote the next chapter toconstructing scalable algorithms that can compute EFPE and other equilibrium refinements
at scale for the first time.
Chapter 13 presents the first algorithm able to compute exact trembling-hand equilibriumrefinements in large imperfect-information extensive-form games, significantly extending
some key ideas presented in Chapter 12. We demonstrate the scalability of our algorithm
for computing exact extensive-form perfect, quasi-perfect, and one-sided quasi-perfect (anew solution concept we introduce), in poker endgames that were encountered as part10

§1.2. RESEARCH DISCUSSED IN THIS DISSERTATIONof the Brains-vs-AI competition. These are games with hundreds of million of terminal
states, and provide a natural testing ground for research on refinements robust to small
mistakes of the (human) players. The scale of the games that can be handled using thetechnology developed in this chapter is several order of magnitude larger that anything thatwas possible before, and enables—for the first time—studying the empirical performanceof equilibrium refinements in large games.
Chapter 14 introduces a methodology for anchoring learning dynamics to remain close to given
strategies for each player. As a special case, such anchored learning algorithms can beused to compute logit quantal response equilibria in imperfect-information extensive-form
games. We empirically investigate the use of these learning algorithms in the constructionof AI agents for the game of no-press Diplomacy, showing that our approach enables
the AI agents to comply with human conventions. This research was recognized with an
outstanding paper honorable mention at ICLR. As mentioned earlier in the introduction, in
a 200-game no-press Diplomacy tournament involving 62 human participants spanningskill levels from beginner to expert, two AI bots trained with our algorithm both achieved
a higher average score than all other participants who played more than two games, andranked first and third according to an Elo ratings model. In the chapter, we also remarkthat the same methodology was recently used in building Cicero, an AI agent for playing
the full version of Diplomacy using natural language communication, which was recently
featured on Science.

1.2

Research discussed in this dissertation

This dissertation combines and builds on a selection of articles that I have published during
my doctoral studies. In addition, this dissertation includes a number of improvements to the
published material, with the goal of offering a cohesive and approachable treatment of the subject.
These additions include new text, numerous examples, new figures and diagrams, remarks about
connections across different chapters, and technical improvements that I discovered in hindsightand never published. Through this additional amount of work I hope that this dissertation will
help future researchers interested in computational aspects of imperfect-information extensiveform games, serving as an approachable reference and partially filling a current void in theliterature.
Parts of this dissertation have appeared in the literature as follows (in chronological order foreach chapter).
Chapter 4.

§1. INTRODUCTION

11• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019c). “Regret Circuits: Composability of Regret Minimizers”. In: Proceedings of the International Conference on Machine
Learning (ICML).
• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2021b). “Faster Game Solving via
Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent”. In:
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Chapter 5.
• Kroer, Christian, Gabriele Farina, and Tuomas Sandholm (2018b). “Solving Large Sequential
Games with the Excessive Gap Technique”. In: Proceedings of the Annual Conference on Neural
Information Processing Systems (NeurIPS).
• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019b). “Optimistic Regret
Minimization for Extensive-Form Games via Dilated Distance-Generating Functions”. In:
Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2021a). “Better Regularizationfor Sequential Decision Spaces: Fast Convergence Rates for Nash, Correlated, and Team
Equilibria”. In: ACM Conference on Economics and Computation.
Chapter 6.
• Anagnostides, Ioannis, Gabriele Farina, Christian Kroer, Andrea Celli, and Tuomas Sandholm(2022). “Faster No-Regret Learning Dynamics for Extensive-Form Correlated and Coarse
Correlated Equilibrium”. In: Proceedings of the ACM Conference on Economics and Computation(EC).
• Anagnostides, Ioannis, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, andTuomas Sandholm (2022). “Uncoupled Learning Dynamics with O(log T ) Swap Regret in
Multiplayer Games”. In: Proceedings of the Annual Conference on Neural Information Processing
Systems (NeurIPS).
• Anagnostides, Ioannis, Gabriele Farina, and Tuomas Sandholm (2023). “Near-Optimal
Φ-Regret Learning in Extensive-Form Games”. In: Proceedings of the International Conference
on Machine Learning (ICML).
Chapter 7.

12§1.2. RESEARCH DISCUSSED IN THIS DISSERTATION• Farina, Gabriele, Chung-Wei Lee, Haipeng Luo, and Christian Kroer (2022). “Kernelized
Multiplicative Weights for 0/1-Polyhedral Games: Bridging the Gap Between Learning in
Extensive-Form and Normal-Form Games”. In: Proceedings of the International Conference on
Machine Learning (ICML).
Chapter 8.
• Celli, Andrea, Alberto Marchesi, Gabriele Farina, and Nicola Gatti (2020). “No-Regret
Learning Dynamics for Extensive-Form Correlated Equilibrium”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).
• Farina, Gabriele, Tommaso Bianchi, and Tuomas Sandholm (2020). “Coarse Correlationin Extensive-Form Games”. In: Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).
• Farina, Gabriele, Andrea Celli, Alberto Marchesi, and Nicola Gatti (2022). “Simple Uncoupled
No-regret Learning Dynamics for Extensive-form Correlated Equilibrium”. In: Journal of theACM 69.6. url: https://dl.acm.org/doi/10.1145/3563772.
Chapter 9.
• Farina, Gabriele, Chun Kai Ling, Fei Fang, and Tuomas Sandholm (2019a). “Correlation in
Extensive-Form Games: Saddle-Point Formulation and Benchmarks”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).
• Farina, Gabriele, Chun Kai Ling, Fei Fang, and Tuomas Sandholm (2019b). “Efficient Regret
Minimization Algorithm for Extensive-Form Correlated Equilibrium”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).
• Farina, Gabriele and Tuomas Sandholm (2020). “Polynomial-Time Computation of Optimal
Correlated Equilibria in Two-Player Extensive-Form Games with Public Chance Moves andBeyond”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Chapter 10.
• Farina, Gabriele, Andrea Celli, Nicola Gatti, and Tuomas Sandholm (2021). “Connecting
Optimal Ex-Ante Collusion in Teams to Extensive-Form Correlation: Faster Algorithms
and Positive Complexity Results”. In: Proceedings of the International Conference on Machine
Learning (ICML).

§1. INTRODUCTION

13• Zhang, Brian Hu, Gabriele Farina, Andrea Celli, and Tuomas Sandholm (2022). “Optimal
Correlated Equilibria in General-Sum Extensive-Form Games: Fixed-Parameter Algorithms,
Hardness, and Two-Sided Column-Generation”. In: Proceedings of the ACM Conference on
Economics and Computation (EC).
• Zhang, Brian Hu, Gabriele Farina, Ioannis Anagnostides, Federico Cacciamani, Stephen
McAleer, Andreas Haup, Andrea Celli, Nicola Gatti, Vincent Conitzer, and Tuomas Sandholm(2023). Learning and Steering toward Optimal Equilibria and Mechanisms.
Chapter 11.
• Farina, Gabriele, Andrea Celli, Nicola Gatti, and Tuomas Sandholm (2018). “Ex Ante Coordination and Collusion in Zero-Sum Multi-Player Extensive-Form Games”. In: Proceedings of
the Annual Conference on Neural Information Processing Systems (NeurIPS).
• Zhang, Brian Hu, Gabriele Farina, and Tuomas Sandholm (2023). “Team Belief DAG Form:
A Concise Representation for Team-Correlated Game-Theoretic Decision Making”. In:
Proceedings of the International Conference on Machine Learning (ICML).
Chapter 12.
• Farina, Gabriele and Nicola Gatti (2017). “Extensive-Form Perfect Equilibrium Computationin Two-Player Games”. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Chapter 13.
• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2017). “Regret Minimization in
Behaviorally-Constrained Zero-Sum Games”. In: Proceedings of the International Conference on
Machine Learning (ICML).
• Kroer, Christian, Gabriele Farina, and Tuomas Sandholm (2017). “Smoothing Method forApproximate Extensive-Form Perfect Equilibrium”. In: Proceedings of the International Joint
Conference on Artificial Intelligence (ĲCAI).
• Farina, Gabriele, Nicola Gatti, and Tuomas Sandholm (2018). “Practical Exact Algorithm for
Trembling-Hand Equilibrium Refinements in Games”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NeurIPS).
• Farina, Gabriele and Tuomas Sandholm (2021a). “Equilibrium Refinement for the Age of
Machines: The One-Sided Quasi-Perfect Equilibrium”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NeurIPS).

14§1.2. RESEARCH DISCUSSED IN THIS DISSERTATION• Farina, Gabriele and Tuomas Sandholm (2022). “Fast Payoff Matrix Sparsification Techniques
for Structured Extensive-Form Games”. In: Proceedings of the AAAI Conference on Artificial
Intelligence (AAAI).
Chapter 14.
• Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019a). “Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games”. In: Proceedings of the
AAAI Conference on Artificial Intelligence (AAAI).
• Bakhtin, Anton, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried,
Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik
Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya
Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, DavidWu, Hugh Zhang, and Markus Zĳlstra (2022). “Human-level play in the game of Diplomacy
by combining language models with strategic reasoning”. In: Science 378.6624. url: https://www.science.org/doi/pdf/10.1126/science.ade9097.
• Jacob, Athul Paul, David J. Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin,
Jacob Andreas, and Noam Brown (2022). “Modeling Strong and Human-Like Gameplay with
KL-Regularized Search”. In: Proceedings of the International Conference on Machine Learning(ICML).
• Bakhtin, Anton, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele
Farina, Alexander H Miller, and Noam Brown (2023). “Mastering the Game of No-Press
Diplomacy via Human-Regularized Reinforcement Learning and Planning”. In: Proceedingsof the International Conference on Learning Representations (ICLR).

Mathematical notation
Throughout the dissertation, we make use of the following mathematical notational conventions.
Vectors and matrices• Vectors and matrices are marked in bold.
• Given a finite set S = {s1 , . . . , sn }, we denote as ℝS (resp., ℝS⩾0 ) the set of real (resp.,
nonnegative real) |S|-dimensional vectors whose entries are denoted as x[s1 ], . . . , x[sn ].
′′• Similarly, given finite sets S, S ′ , we denote as ℝS×S (resp., ℝS×S
⩾0 ) the set of real (resp.,′
nonnegative real) S × S square matrices M whose entries are denoted as M[sr , sc ] (sr ∈S, sc ∈ S ′ ), where sr corresponds to the row index and sc to the column index.
Standard sets
• We denote the set of real numbers as ℝ, the set of nonnegative real numbers as ℝ⩾0 , and theset {1, 2, . . . } of positive integers as ℕ.
• The set {1, . . . , n}, where n ∈ ℕ, is compactly denoted as [[n]].
• The empty set is denoted as { }.
• Given a finite set S, we denote by ∆S the simplex ∆S := {x ∈ ℝS⩾0 :symbol ∆n , with n ∈ ℕ, is used to mean ∆[[n]] .

Ps∈S x[s] = 1}. The• Given a finite set S, we use the symbol 𝕊S ⊆ ℝS×S
⩾0 to denote the set of stochastic matrices,that is, nonnegative square matrices whose columns all sum up to 1. The symbol 𝕊n , wheren ∈ ℕ, is used to mean 𝕊[[n]] .
Operations on sets
• Given a set S, we denote its convex hull with the symbol co S. The convex hull of the unionof finitely many sets S1 , . . . , Sm is denoted co{S1 , . . . , Sm }.

16

MATHEMATICAL NOTATION• Disjoint union of set is denoted with the symbol ⊔.
Functions• Given two functions f : X → Y and g : Y → Z, we denote by g ◦ f : X → Z theircomposition x 7→ g(f (x)).
• Given a set S and a function f , the image of S via f is denoted as f (S) := {f (s) : s ∈ S}.
• Given a proposition p, we denote with 𝟙p the indicator function of that proposition:

1
𝟙p =
0

if p is true
otherwise.

Partial orders• Given a partially ordered set (S, ≺) and two elements s, s′ ∈ S, we use the standard derived
symbols s ≼ s′ to mean that (s = s′ ) ∨ (s ≺ s′ ), s ≻ s′ to mean that s′ ≺ s, and s ≽ s′ tomean that s′ ≼ s.
Asymptotic notation• We use the symbols O, Ω, Θ to denote the usual asymptotic notation.
• The symbol OT is used to denote that only dependence on the parameter T is highlighted,
treating all other parameters as constants.
Additional notation about games and other objects used throughout the dissertation will be
introduced in later chapters. All notation is summarized in Appendix B to help the reader.

Chapter 2

Imperfect-information
extensive-form games
Imperfect-information extensive-form[2.a] games model tree-form strategic interactions in which
not all actions might be observed by all players. They represent an ample majority of strategicinteractions encountered in the real world, ranging from recreational games such as poker, tonegotiation, and auctions.
The standard representation of an imperfect-information extensive-form game is through its
game tree, which formalizes the interaction of the players as a directed tree. In the game tree,
each non-terminal node belongs to exactly one player, who acts at the node by picking one of theoutgoing edges (each labeled with an action name). Imperfect information is captured in thisrepresentation by partitioning the nodes of each player into sets (called information sets) of nodes
that are indistinguishable to that player given his or her observations. We will describe in detailthis representation in Section 2.1.
It should be stressed that one of the distinguishing features of the game tree representation ofgames is that it encodes the dynamics of the interaction for all players, without taking the side of
any one player in particular. This makes the game tree a suboptimal representation for shedding
light on how individual players can learn in the game—that is, iteratively refine their strategies
to improve performance until eventually, in most cases, game-theoretic equilibrium is reached.
Instead, in most of this dissertation we will find it convenient (and insightful) to express learningand optimization of strategies from the point of a particular player as a procedure on the player’s
tree-form decision process (TFDP). As the name suggests, the TFDP expresses the interaction fromthe point of view of the player of interest, making a distinction between the decisions that theplayer faces, and the observations that the player makes. We introduce TFDPs in Section 2.2,
showing how the TFDP of any player can be extracted from the game tree.
[2.a] The term extensive-form is a standard term in the game theory literature, meaning tree-form.

18§2.1. GAME TREES AND INFORMATION SETS

2.1

Game trees and information sets

LegendIn this section, we detail how the game tree representation captures the dynamics of a game,and how imperfect information is encoded in this representation. As a running example, wewill illustrate the representation by analyzing the game tree of a simplified two-player variant ofpoker (perhaps the archetype of imperfect-information extensive-form games), known as Kuhn
poker (Kuhn, 1950). Additional examples will be presented in Section 2.2.

A

chk.
P

QJ

QK

JK

KJ

B

bet

chk.

NaturePlayer 2
Y

Terminal
Information set

JQ

C

bet

chk.
R

Q

KQ

Player 1

bet

chk.

bet

chk.
T

S

bet

chk.

bet

U

chk. bet

fold call chk. bet

fold call chk. bet

fold call chk. bet

fold call chk. bet

fold call chk. bet

fold call−1 D+1 −2 −1 E

+1 −2 +1

+1 +2 +1 F

+1 +2 +1

+1 +2 −1

+1 −2

fold callfold callfold callfold callfold callfold call

−1 −2

−1 −2

−1 +2

−1 +2

−1 +2

−1 −2

Figure 2.1: Game tree for the game of Kuhn poker. Utilities are shown for Player 1 only; utilities forPlayer 2 are the opposite Player 1’s. Due to space constraints, we used ‘chk.’ as an abbreviation
for the ‘check’ action.

Histories, actions, and payoffs The game tree represents the strategic interaction of players asa finite directed tree. The nodes of the game tree are called histories. Each history that is not aleaf of the game tree is associated with a unique acting player. In an n-player game, the set ofvalid players is the set [[n]] ∪ {c} = {1, . . . , n, c}, where c denotes the chance (or nature) player—afictitious player that selects actions according to a known fixed probability distribution andmodels exogenous stochasticity of the environment (for example, a roll of the dice, or drawing acard from a deck). The player is free to pick any one of the actions available at the history, whichcorrespond to the outgoing edges at the histories.
The players keep acting until a leaf of the game tree—called a terminal history—is reached.
Terminal histories are not associated with any acting player; the set of terminal histories is denoted
Z. When the game transitions to a terminal node z ∈ Z, each player i ∈ [[n]] receives a payoff
according to the payoff function ui : Z → ℝ.

§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMES

19

Example 2.1 (Kuhn poker). In the game tree of Kuhn poker (Figure 2.1), the root history of thetree (the first move in the game) belongs to the nature player c. It models a dealer that privately
deals one card to each player from a shuffled deck containing cards {Jack, Queen, King}. The
actions of the nature player correspond to the six possible assignments of two cards from thedeck, which are annotated on the edges; for example, the leftmost edge JK corresponds to thecase in which Player 1 is dealt a Jack and Player 2 is dealt a King. Since the deck is shuffled,
each of the six actions are selected with probability 1⁄6 by the nature player.
No matter the action selected by the dealer, the game transitions to a history of Player 1,
which marks the beginning of what in poker is called a “betting round”. First, Player 1
decides to either check (continue without betting any money) or bet $1. Then,• If Player 1 checks, Player 2 can either check, or bet $1.
– If Player 2 checks, the game terminates with a showdown: the player with the higher
card receives from the other player whatever amount the other player bet, plus anante amount of $1.
– If, instead, Player 2 bets the additional $1, then Player 1 can either fold his hand or call,that is, raise his bet by $1.
* If Player 1 folds, he has to give Player 2 only the $1 ante;* if Player 1 calls, a showdown with the same dynamics as before.
• If Player 1 bets the $1, Player 2 can either fold her hand or call.
– If Player 2 folds her hand, Player 2 gives Player 1 the $1 ante.
– If, instead, Player 2 calls the bet, she increases her bet by $1 and a showdown occurs,with the same dynamics as before.

Imperfect information and information sets To model imperfect information, the histories
of each player i ∈ [[n]] are partitioned into a collection Ii of so-called information sets. Each
information set I ∈ Ii groups together histories that Player i cannot distinguish between when heor she acts there. In the limit case in which all information sets are singleton, the player neverhas any uncertainty about which history they are acting at, and the game is said to have perfect
information.
Since a player always knows what actions are available at a decision node, any two historiesh, h′ belonging to the same information set I must have the same set of available actions.
Correspondingly, we can write AI to denote the set of actions available at any node that belongs
to information set I.
Example 2.1 (Continued; Kuhn poker). In Kuhn poker, each player observes their own private
card and the actions of the opponent, but not the opponent’s private card. The twelve

20§2.1. GAME TREES AND INFORMATION SETS

information sets indicated in Figure 2.1 (six for Player 1 denoted A through F, and six forPlayer 2 denoted P through U) reflect this partial information.
For example, Player 1’s histories following actions QK and QJ of the nature player (thedealer) are part of the same information set B, in that Player 1 cannot distinguish between thetwo histories, having observed only their private Queen card.
As another example, Player 2’s information set P captures the uncertainty the player hason the underlying history after having observed a private King card, and a check from Player 1.

JK

QK

Q

QK

JQ

KQ

JQ

KJ

JK

KQ

KJ

QK

KJ

JK

J

Example 2.2. To further illustrate how information sets capture private information, in thisexample we speculate on how different rules for Kuhn poker would translate into different
information set structures.

KQ

JQ■ First variation

Q

J

Player 1 is revealed the private
card of Player 2 by the dealer.

■ Second variation

Q

J

Player 2 does not get to observe
her private card.

■ Third variation

Player 1 is allowed to look at his
private card only if he decides tocheck.

Perfect recall As is standard in the literature, we assume that the game has perfect recall, thatis, information sets satisfy the fact that that no player forgets about their actions, and about
information once acquired. This condition is formalized as follows.

§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMES

21

Definition 2.1 (Perfect recall). A player i ∈ [[n]] is said to have perfect recall if, for any
information set I ∈ Ii , for any two histories h, h′ ∈ I the sequence of Player i’s actions
encountered along the path from the root to h and from the root to h′ must coincide(or otherwise Player i would be able to distinguish among the histories, since the player
remembers all of the actions they played in the past). The game is perfect recall if all players
have perfect recall.

2.2 The player’s perspective: Tree-form decision processes
The game tree representation introduced in Section 2.1 provides a description of the global dynamics
of the game, without taking the side of any player in particular.
In this section, we will lay foundations and tools for operating with the tree-form decision process(TFDP) that an individual player faces in an imperfect-information extensive-form game. The
TFDP provides a more natural formalism for defining player-specific quantities and procedures,such as strategies and learning algorithms, that inherently refer to the decision space that one player
faces while playing the game. Most of the content of this dissertation, being generally concernedwith how individual players can optimize or refine their strategies so that equilibrium is reached,will find its natural formalization in the language of TFDPs.
Example 2.3 (Player 1’s decision process in Kuhn poker). As an example, consider Player 1
in Kuhn poker (Example 2.1). From the player’s point of view, playing the game could besummarized as follows:• As soon as the game starts, the player observes a private card that has been dealt tothem; the set of possible signals is {Jack, Queen, King}.
• No matter the card observed, the player now needs to select one action from the set{check, bet}.
– If the player bets, the player does not have a chance to act further– Otherwise, if the player checks, the player will then observe whether the opponent
checks (at which point the interaction terminates) or bets. In the latter case, a newdecision needs to be made, between folding the hand, or calling the bet. In either
case, after the action has been selected, the interaction terminates.
By arranging the structure of decisions and observations along a tree as in Figure 2.2, we
obtain the tree-form decision process for Player 1.

22§2.2.THE PLAYER’S PERSPECTIVE:TREE-FORM DECISION PROCESSES

chk.

chk.

bet

fold call

chk.

bet

chk.

King

Queen

bet

fold call

chk.

bet

chk.

bet

bet

fold callLegendJack

Decision nodeObservation node
End of process

Figure 2.2: Tree-form decision process faced by Player 1 in the game of Kuhn poker.

The tree-form decision process lays out the player’s opportunities to act. Unlike the game tree,
in which each node belongs to one of many players, the tree-form decision process is a directed
tree made of only two types of nodes: decision nodes, at which the player must act by picking anaction from a set of legal actions, and observation nodes, at which the player does not act but rather
observes a signal drawn from a set of possible signals. Furthermore, the information structureof the player, previously defined indirectly through information sets, is captured directly in the
TFDP representation.
While the process that led us to Figure 2.2 was heuristic and based on our intuitive understanding of Kuhn poker, in the next subsection we discuss how a TFDP can be extracted from agame tree in any perfect-recall imperfect-information extensive-form game.

2.2.1

Extracting a tree-form decision process from the game tree

In some cases, like in Example 2.3, it is straightforward to compile the tree-form decision process
faced by a player starting from our intuitive understanding of the game. In this subsection we
discuss how the TFDP for the player can be constructed programmatically starting from the gametree when such an understanding is missing. We assume that an n-player imperfect-information
extensive-form game with perfect recall and a player i ∈ [[n]] of interest, have been fixed.
The set of decision nodes Ji of the player’s TFDP coincides with the set of his or her information
sets, that is, Ji = Ii . This is consistent with the fact that the player cannot condition their behavior
on anything other than their information set, given that they cannot distinguish between histories
in the same information set. Furthermore, the set of actions available at any decision nodej = I ∈ Ii coincides with the set of actions AI available at any history in information set I.
Fix an information set I for the player, and for any history h ∈ I, imagine walking the pathfrom the root of the game tree to h, keeping track of all the information sets and actions ofPlayer i encountered along the path—let us call this the trace corresponding to history h. Because
the player recalls their past actions and information sets, and yet all h ∈ I are by definitionindistinguishable to the player, it follows immediately that the traces of all histories h ∈ I must23

§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMES

coincide. Hence, the notion of trace of I, defined as the trace of any h ∈ I, is well defined. We
give two examples illustrating traces.
Example 2.4. Consider Kuhn poker (Figure 2.1) from the point of view of Player 1.
• The trace of any history in A is the sequence (A).
• The trace of any history in E is the sequence (B, chk., E).
From the point of view of Player 2, the trace of any history in R is the sequence (R).

Example 2.5. Consider the small game tree given in Figure 2.3.
Legend

A
1

2
Q

P

B

C

3 4

5 6

Player 1

Terminal history

Player 2
Y

Information set

D
789

789

Figure 2.3: Small game tree used in the example.

Taking the side of Player 1, the trace of the only history in B is the sequence (A, 1, B), thetrace of any history in D is (A, 2, D), and the trace of the only history in A is (A). Taking theside of Player 2, the trace of the only history in P is (P), and the trace of the only history in Q
is (Q).
Traces implicitly encode a notion of partial chronological ordering between information sets,of which the player has recall—see Definition 2.1. Hence, for the TFDP of Player i to be an accurate
representation of the decision process the player faces while playing the game, it is necessary that
traces of the information sets are the same in the game tree and in the TFDP. In other words, we requirethat decision points in the TFDP be structured so as to satisfy that the trace of any information set
I matches the sequence of information sets and actions encountered from the root of the TFDP todecision node I.
Definition 2.2 (Tree-form decision process). Fix the game tree of an n-player imperfect
information game, and a player i ∈ [[n]]. A tree-form decision process (TFDP) for Player i is adirected rooted tree made of decision, observation, and terminal nodes, satisfying the following
properties.
• The set of decision nodes Ji of the TFDP is equal to the set Ii of information sets.
• The set of actions available at each decision node j = I ∈ Ii (i.e., the set of outgoing24§2.2.THE PLAYER’S PERSPECTIVE:TREE-FORM DECISION PROCESSES

edges from the decision node) is equal to the set of actions AI available at any historyh ∈ I in the game tree.
• Given any decision node j = I ∈ Ii , the sequence of decision nodes and actions
encountered from the root of the TFDP to j is equal to the trace of any history h ∈ I.
We remark that Definition 2.2 leaves the labeling and structure of observation nodes unspecified.
In fact, a player might have multiple TFDPs that satisfy the definition, and differ in how theobservation nodes are placed. We illustrate this in the next example.
Example 2.6. The following are both valid TFDPs capturing Player 1’s decision process whenplaying Kuhn poker.

B

chk.

chk.

bet

C
chk.

bet

bet

D

E

F

fold callfold callfold callLegendA

A

B

chk. bet

chk. bet

Cchk. bet

DE

F

fold callfold callfold call

Decision node

Terminal nodeObservation node

Example 2.7. A valid TFDP representing the decision process of Player 1 in the small gameof Figure 2.3, reproduced below on the left, is shown below on the right.

1

2

−→

Q

P

1

LegendA

A

2

Decision nodeObservation node
Terminal node

D
789

B

C

3 4

5 6

2.2.2

D
789

789

B

C

3 4

5 6

Notation

Most of the algorithms and procedures we will introduce in this dissertation are best understoodas operating on the tree-form decision process of a player. In this section we introduce additional
notation and conventions related to such objects, which will be used throughout the document.
A summary of the notation is given in Table 2.1 and in Appendix B.

§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMES

25

Convention regarding players In this dissertation we use the general convention of subscripting
quantities relative to a Player i with a lowercase i whenever important. This is the case, forexample, of the information sets of the players, which we have denoted I1 , . . . , In in the past.
However, as most of the algorithms we will discuss in this dissertation assume that a TFDP
for a generic Player i has been fixed, it will be typical to omit the lowercase i from most of the
discussion. The notation summarized in Table 2.1 has the player omitted.
Decision and observation nodes, transition function:• We denote the set of decision nodes in the TFDP as J , and the set of observation nodes as K.
At each decision node j ∈ J , the player selects an action from the set Aj of available actions.
At each observation node k ∈ K, the player observes a signal sk from the environment outof a set of possible signals Sk .
• We denote ρ the transition function of the process. Picking action a ∈ Aj at decision nodej ∈ J results in the process transitioning to ρ(j, a) ∈ J ∪ K ∪ {⊥}, where ⊥ denotes the endof the decision process. Similarly, the process transitions to ρ(k, s) ∈ J ∪ K ∪ {⊥} after theplayer observes signal s ∈ Sk at observation node k ∈ K.
Sequences, which identify actions at decision nodes, and will be key in defining sequence-form
strategies in the next section:• A pair (j, a) where j ∈ J and a ∈ Aj is called a non-empty sequence. The set of all non-empty
sequences is denoted as Σ∗ := {(j, a) : j ∈ J , a ∈ Aj }. For notational convenience, we willoften denote an element (j, a) in Σ as ja without using parentheses, especially when used
as a subscript.
• The symbol ∅ denotes a special sequence called the empty sequence. The set of all sequences,including the empty one, is denoted Σ.
• Given a decision node j ∈ J , we denote by pj its parent sequence, defined as the last sequence(that is, decision point-action pair) encountered on the path from the root of the decision
process to j. If the player does not act before j (that is, j is the root of the process or only
observation nodes are encountered on the path from the root to j), we let pj = ∅.
• Given a sequence σ ∈ Σ, we denote with Cσ the set of decision nodes j whose parent
sequence is σ: Cσ := {j ∈ J : pj = σ}.
Subtrees and descendancy relationships Finally, we introduce the following symbols to establish
descendancy relationships:• Given two decision nodes j, j ′ , we write j ≺ j ′ (or equivalently j ′ ≻ j) to mean that j is anancestor of j ′ in the TFDP and that j ̸= j ′ . The symbol j ≼ j ′ (or equivalently j ′ ≽ j) meansthat j ≺ j ′ or j = j ′ .
• Given two sequences ja, j ′ a′ , we write ja ≼ j ′ a′ (or equivalently j ′ a′ ≽ ja) to mean that(unique) path from the root to action a′ at j ′ passes through action a at j.

26§2.3. STRATEGIES AND SEQUENCE-FORM REPRESENTATION• The overloaded notation σ ≽ j (or equivalently j ≼ σ), defined for any j ∈ J , and sequence
σ = (j ′ , a′ ) ∈ Σ∗ , denotes that j ′ ≽ j.
• Finally, we let Σj := {σ ∈ Σ∗ : σ ≽ j} ⊆ Σ∗ be the set of sequences at or below a given j ∈ J .
SymbolDescriptionJ
Aj

Set of decision nodes
Set of legal actions at decision node j ∈ J

KSk

Set of observation nodes
Set of possible signals at observation node k ∈ KρTransition function:• given j ∈ J and a ∈ Aj , ρ(j, a) returns the next point v ∈ J ∪ K in thedecision tree that is reached after selecting legal action a in j, or ⊥ if thedecision process ends;• given k ∈ K and s ∈ Sk , ρ(k, s) returns the next point v ∈ J ∪ K in thedecision tree that is reached after observing signal s in k, or ⊥ if the decision
process endsΣ∗Σ

Set of non-empty sequences, defined as Σ∗ := {(j, a) : j ∈ J , a ∈ Aj }Set of sequences, defined as Σ := Σ∗ ∪ {∅} where the special element ∅ iscalled the empty sequence
Parent sequence of decision node j ∈ J , defined as the last sequence (decisionnode-action pair) on the path from the root of the TFDP to decision node j; if
the player does not act before j, pj = ∅Decision nodes j ∈ J with parent sequence σ ∈ Σ: Cσ := {j ∈ J : pj = σ}

pj

Cσ
′

j ≼j
j a ≼ ja
σ≽j
′ ′

Σ≽j

j ′ ∈ J is on the path from the root to j ∈ J
Action a′ at j ′ is on the path from the root to action a at j ∈ J
Shorthand for σ = j ′ a′ with j ′ ≽ jSequences at or below j: Σ≽j := {σ ∈ Σ∗ : σ ≽ j}

Table 2.1: Summary of basic notation for TFDPs. In cases where it is important to specify the player towhich the different quantities belong, a subscript with the player will be added.

2.3

Strategies and sequence-form representation

Consider the tree-form decision process faced by a player in an imperfect-information extensiveform game. Conceptually, a strategy for a player corresponds to a choice of distribution over theset of actions Aj at each decision node j ∈ J . So, perhaps the most intuitive representation of astrategy, called a behavioral strategy, is as a vector x ∈ ℝΣ
⩾0 indexed over sequences assigning toeach action a at decision node j the probability of picking that action at that decision node. The setof all possible behavioral strategies is clearly convex, as it is the Cartesian product of probability

27§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMES

simplexes—one per each decision node. However, that representation has a major drawback: theprobability of reaching a particular terminal state in the decision process is the product of allactions on the path from the root to the terminal state. This makes many expressions of interest
that depend on the probability of reaching terminal states (including crucially the expected utility
in the game) non-convex.

2.3.1

Sequence-form representation of strategies

The sequence-form representation (Romanovskii, 1962; Koller, Megiddo, and von Stengel, 1996; vonStengel, 1996) soundly resolves the issue of non-convexity. In the sequence-form representation, astrategy is a vector x ∈ ℝΣ⩾0 whose entries are indexed by Σ. However, the entry x[ja] contains
the product of the probabilities of all actions at all decision nodes on the path from the root of theprocess to action a at decision node j. In order to be a valid sequence-form strategy, the entries inx must therefore satisfy the following probability-flow-conservation constraints:x[∅] = 1,X

x[ja] = x[pj ] ∀j ∈ J .

(2.1)

a∈Aj

Conversely, it is easy to see that any x that satisfies the above constraints is the sequence-form
representation of at least one behavioral strategy.
Example 2.8. Consider the TFDP introduced in Example 2.7, reproduced below.



1.00


A1  0.50 
A2  0.50 
B3  0.25 
0.25 
q := B4 
C5  0.10 
C6  0.40 
D7  0.00 
D8  0.50 
D9 0.00
∅

A
1

2
D
789

B

C

3 4

5 6

Let us consider the mixed sequence-form strategy q ∈ Q defined alongside the TFDP
above. We have that q[A1] = q[A2] = 0.5, and therefore Player 1 will select between actions 1
and 2 at decision node A uniformly at random.
Suppose Player 1 selects action 1. If Player 1 reached decision node B, she would selectactions 3 and 4 with probability 0.25/0.5 = 0.5 each. On the other hand, if Player 1 reacheddecision node C, she would choose action 5 with probability 0.1/0.5 = 0.2, and action 6 withprobability 0.4/0.5 = 0.8.

28

§2.3. STRATEGIES AND SEQUENCE-FORM REPRESENTATION

Analogously, if Player 1 played action 2 at decision node A, upon reaching decision node
D she would play action 8 with probability 0.5/0.5 = 1.
In general, the probability of playing action a at a generic decision node j can be obtainedby dividing q[ja] by q[pj ].
The set of all sequence-form strategies will be denoted with the symbol Q. Since (2.1) defines
linear constraints, Q is a convex polytope.
Definition 2.3. The polytope of sequence-form strategies of a TFDP is the convex polytope

Q :=




x ∈ ℝΣ
⩾0 :

x[∅] = 1,Xa∈Aj

x[ja] = x[pj ] ∀j ∈ J




.



Example 2.9. Consider the tree-form decision process faced by Player 1 in the small game of
Example 2.5, which is reproduced in Figure 2.4 (Left).
The decision process has four decision nodes J = {A, B, C, D} and nine sequences
including the empty sequence ∅. For decision node D, the parent sequence is pD = A2; for B
and C it is pB = pC = A1; for A it is the empty sequence pA = ∅. The constraints that define
the sequence-form polytope (Definition 2.3) are shown in Figure 2.4 (Right).
A
1

2
D
789

B

C

3 4

5 6

Sequence-form constraints:x[∅] = 1,





x[A1]
+
x[A2]
= x[∅],
x[B3] + x[B4] = x[A1],x[C5] + x[C6] = x[A1], x[D7] + x[D8] + x[D9] = x[A2].

Figure 2.4: (Left) Tree-form decision process considered in the example. (Right) The constraints
that define the sequence-form polytope Q for Player 1 (besides nonnegativity) in the TFDP
shown on the left.

This dissertation will almost exclusively work with strategies represented in sequence form.
The polytope of sequence-form strategies possesses a strong combinatorial structure thatenables speeding up several common optimization procedures and will be crucial in developing
efficient algorithms to converge to equilibrium.
Sometimes, we will find it important to consider partial strategies that only specify behavior
at a decision node j and all of its descendants j ′ ≻ j. We make that formal through the following
definition.

29§2. IMPERFECT-INFORMATION EXTENSIVE-FORM GAMESDefinition 2.4. The set of sequence-form strategies for the subtree[2.b] rooted at j, denoted Q≽j , isΣ
the set of all vectors x ∈ ℝ⩾0≽j such that probability-mass-conservation constraints hold at
decision node j and all of its descendants j ′ ≻ j, specificallyQ≽j :=





2.3.2

Σ

x ∈ ℝ⩾0≽j :Xa∈Ajx[ja] = 1,andXa∈Aj ′


x[j ′ a] = x[pj ′ ] ∀ j ′ ≻ j .


(2.2)

Deterministic sequence-form strategies and Kuhn’s theorem

Out of the polytope of all possible strategies in the decision process, deterministic strategies are
extremely important. Deterministic strategies are strategies that select exactly one action at eachdecision node, without ever randomizing the choice. In other words, deterministic strategies
assign probability either 0 or 1 to each action at each decision node. Hence, when using thesequence-form representation, the set of all deterministic strategies—denoted Π—corresponds tothe subset of Q whose components are 0 or 1.
Definition 2.5. The set of deterministic sequence-form strategies is the setΠ := Q ∩ {0, 1}Σ .
Similarly, the set of deterministic sequence-form strategies for the subtree rooted at j isΠ≽j := Q≽j ∩ {0, 1}Σ≽j .

We provide examples of deterministic sequence-form strategies next.
Example 2.10 (Deterministic sequence-form strategies). Continuing Example 2.7, in Figure 2.5(Right) we provide five deterministic sequence-form strategies π 135 , π 136 , π 145 , π 27 , π 28 ∈ Π.

[2.b] The term “subtree” does not refer to a subtree of the game tree, but rather to a subtree of the partially ordered set(J , ≺). In other words, the term subtree here refers to the fact that the quantities are specified only at decision node j andall of its descendants.

30§2.3. STRATEGIES AND SEQUENCE-FORM REPRESENTATION

π 135
A



1.0
 
A1  1.0 
 
A2  0.0  
B3  1.0 
 
B4  0.0  
C5  1.0 
 
C6  0.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅

1

2
D
789

B

C

3 4

5 6

π 136




π 145


1.0
 
A1  1.0 
 
A2  0.0  
B3  1.0 
 
B4  0.0  
C5  0.0 
 
C6  1.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅



π 27


1.0
 
A1  1.0 
 
A2  0.0  
B3  0.0  
B4  1.0 
 
C5  1.0 
 
C6  0.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅



π 28


1.0
 
A1  0.0  
A2  1.0  
B3  0.0  
B4  0.0  
C5  0.0 
 
C6  0.0 
 
D7  1.0 
 
D8  0.0 
D9 0.0
∅




1.0
 
A1  0.0  
A2  1.0  
B3  0.0  
B4  0.0  
C5  0.0 
 
C6  0.0 
 
D7  0.0  
D8  1.0 
D9 0.0∅Figure 2.5: (Right) Examples of deterministic sequence-form strategies for the small tree-form
decision process on the left.

One can check that these vectors are valid sequence-form strategies by verifying that theprobability-mass-conservation constraints of Definition 2.3 hold.
As an illustration, consider the deterministic sequence-form strategy π 136 . When playing
according to that strategy, the agent will always choose action 1 at decision node A, action 3 atdecision node B, and action 6 at decision node C. It is impossible for the player to reach decisionnode D given her strategy at A and correspondingly π 136 [D7] = π 136 [D8] = π 136 [D9] = 0.
Kuhn’s theorem (Kuhn, 1953) establishes the connection between the sequence-form strategy
polytope and the set of deterministic sequence-form strategies.
Theorem 2.1 (Kuhn’s theorem). The sequence-form polytope is the convex hull of the set ofdeterministic sequence-form strategies, that is,
Q = co Π,andQ≽j = co Π≽j∀j ∈ J.

Chapter 3

No-regret learning in games
One of the key focuses of this dissertation is providing positive, constructive answers to thefollowing fundamental question:
Can a player that repeatedly plays an imperfect-information extensive-form game follow rules torefine their strategy after each match, so as to guarantee mastering the game in the long run?
Throughout the different chapters, we will investigate different angles of the question: How does
one define “mastering” the game? What kind of feedback is collected by each player after eachmatch? How fast can the agent master the game?
To attach a quantitative measure to the goal of learning, we will operate within the frameworkof regret minimization. Regret minimization is founded on the idea that learning players should
keep under control (specifically, keep as low as possible) their regret—the difference between
the reward they accumulated through the actions they played, and the reward they would
have accumulated in hindsight had they consistently modified their behavior according to somestrategy transformation function. The size of the set of transformation functions considered byeach learning player determines a natural notion of rationality of that player.
An extremely important feature of regret minimization is that it connects a local notion of
optimality, the minimization of the regret incurred by each learning player, to a global notion of
optimality, convergence to game-theoretic equilibrium. This connection informs a methodology
for computing a variety of equilibria that is core to this dissertation. As of today, algorithms
based on learning dynamics are by far the most scalable techniques for computing many notionsof game-theoretic equilibrium in large games, and have played a central role in most recent
game-playing AI breakthroughs. Compared to other traditional optimization techniques forequilibrium computation, which tend to define a more centralized notion of update, learning
dynamics have the advantage that the optimization is decentralized (each player updates theirown strategy incrementally), and uncoupled (each player only uses feedback about their own

§3.1. HINDSIGHT RATIONALITY AND Φ-REGRET

32

utility function).

3.1

Hindsight rationality and Φ-regret

No-regret learning algorithms make the objective of “learning to play the game” concrete bymeans of the quantitative metric called regret. Intuitively, as the name suggests, regret tracks thedifference between the utility that the algorithm accumulated over time, and the utility that it
could have cumulated in hindsight by employing a different strategy. More specifically, given a
generic player whose set of strategies is X , we consider a necessary condition for saying that theplayer has “learnt” to play the game when looking back at the history of play, the player cannot
think of any transformation ϕ : X → X of their strategies that when applied at the whole historyof play would have given strictly better utility to the player. The size of the set Φ of transformations
ϕ : X → X considered by the player defines a natural notion of how “hindsight-rational” the agent
is. As we will see in Section 3.1.2, higher hindsight rationality is tied to the ability to recover moresophisticated game-theoretic notion from no-regret strategies.

3.1.1

Definition of Φ-regret for a player in the repeated game

In order to formally define Φ-regret, consider a generic n-player imperfect-information extensiveform game, and let u1 , . . . , un and Q1 , . . . , Qn denote the expected utility functions and sequenceform strategy spaces of the players, respectively. Furthermore, let Φ be a compact set of functions
ϕ : Qi → Qi for a generic player i in the game. Suppose that at all times (repetition of the game)(t)t = 1, 2, . . . , Player i plays according to some strategy xi , while all other players play according
(t)
(t)
(t)
(t)
(t)
to strategies (x1 , . . . , xi−1 , xi+1 , . . . , xn ) =: x−i . Then, for any time horizon T , the Φ-regret ofPlayer i up to time T is defined as the difference between what Player i could have accumulated
in hindsight had they consistently modified their strategy according to the best ϕ ∈ Φ, and the
expected utility that they actually accumulated; in symbols


(t)
(t)
Φ -Regi {xi , x−i }Tt=1

:= max
ϕ̂∈Φ

( T
X



(t)
(t)
ui ϕ̂(xi ), x−i





(t)
(t)
− ui xi , x−i



)
.

t=1

It is clear from the definition that regret can grow at most linearly as a function of the time
horizon T . When the player plays in a way that regret is guaranteed to grow sublinearly in T , wesay that the player is a no-Φ-regret learner. As we show in the next subsection, sublinear growth ofregret is enough to guarantee ergodic convergence to game-theoretic equilibrium.

§3. NO-REGRET LEARNING IN GAMES

3.1.2

33

Relationship between Φ-regret and game-theoretic equilibrium

An important feature of the framework of Φ-regret minimization is its ability to connect theindividual notion of Φ-regret with global notions of game-theoretic optimality. The specific notionof game-theoretic equilibrium that can be recovered from no-Φ-regret learning players depends
on the class of deviations Φ with respect to which the agents seek to minimize regret. As theset Φ grows, so does intuitively the rationality of the agents, and more sophisticated notions ofequilibrium are approached. Below, we mention without proof a few landmark results showingthat some of the best-studied notions of equilibrium can indeed be reached via learning dynamics.
Constant transformations (External regret) In this case, we are only requiring that the player
does not regret substituting all of the strategies they played with a unique, constant strategyx̂ in hindsight. Despite the seemingly restrictive notion of rationality, regret minimization
with respect to constant transformation is extremely powerful.
Connection 3.1 (D. Foster and Vohra, 1997). When all players in a multiplayer game playso that their Φ-regret with respect to the of constant transformations grows sublinearlyin the number of repetitions of the game T , their average joint strategy[3.a] converges tothe set of coarse correlated equilibria of the game as T → ∞.

Connection 3.2 (Folklore). When the two players in a two-player zero-sum game play sothat their Φ-regret with respect to the set of constant transformations grows sublinearly
(T )
(T )
in the number of repetitions of the game T , the average strategies (x̄1 , x̄2 ) :=
P
P
(t)
(t)
T
T
( T1 t=1 x1 , T1 t=1 x2 ) converge to the set of Nash equilibria of the game as T → ∞.

Even more, we will show in Section 3.2.3 that the minimization of regret with respect toconstant transformations forms the backbone of learning with respect to any richer set ofdeviations. Φ-regret measured with respect to constant transformations is often called
external regret or static regret in the literature.
Mass-transferring deviations (Internal regret) These deviations apply only to nonsequential
games, where each player’s strategy space is the probability simplex over their finite set ofactions A. In this context, the set of mass-transferring deviations is the set of transformation
functions that intuitively transfer the probability mass given to an action a to another actionb; formally, Φ := {ϕa→b : a, b ∈ A}, where, for all k ∈ A,
[3.a] That is, at all times T , the correlated distribution of play µ(T ) that selects (x(t) , . . . , x(t) ) with probability 1/T .
n1

§3.1. HINDSIGHT RATIONALITY AND Φ-REGRET

34

ϕa→b (x)[k] :=




x[k]
0





if k ∈/ {a, b}if k = ax[a] + x[b] if k = b.

Φ-regret measured with respect to mass-transferring transformations is often called internal
regret in the literature. It can be easily shown that an internal-regret-minimizing agent is
automatically external-regret-minimizing; so, Connections 3.1 and 3.2 apply to internalregret-minimizing agents too. In addition, internal-regret-minimizing agents recovercorrelated equilibria of the game, as the following celebrated result states.
Connection 3.3 (Hart and Mas-Colell, 2000). When all players in a multiplayer
nonsequential game play so that their internal regret grows sublinearly as a function of
the number of repetitions of the game T , their average joint strategy[3.a] converges tothe set of correlated equilibria of the game.

Trigger deviation functions (Trigger regret) In Chapter 8 of this dissertation we will introduce
a generalization of the mass-transferring deviations mentioned above, called trigger deviation functions. These deviation functions coincide with mass-transferring deviations innonsequential games, and are well-defined in any extensive-form game. In this case, wewill show how one can construct learning players that are able to update their strategies
efficiently while guaranteeing that their Φ-regret bounded with a polynomial dependence
on the size of the game tree. Most importantly, we will show that trigger deviation functions
are related to solution concept called “extensive-form correlated equilibrium”, introducedby von Stengel and Forges (2008).
Connection 3.4 (Chapter 8). When all players in a multiplayer extensive-form gameplay so that their Φ-regret relative to the set of all trigger deviation functions grows
sublinearly in the number of repetitions of the game T , their average joint strategy[3.a]converges to the set of extensive-form correlated equilibria of the game.
Φ-regret measured with respect to trigger deviation functions takes the name of trigger
regret.

3.1.3

Feedback available to the learning player

The previous subsections justify keeping regret as small as possible (and, in particular, at mostsublinear) as valuable goal with important connections to the computation of game-theoretic
equilibrium. In order for the player to learn and satisfy the sublinear regret requirement, it is35§3. NO-REGRET LEARNING IN GAMES

clearly important to define what kind of input (that is, feedback) the learning player has at hisor her own disposal. In this dissertation we will consider three models of feedback that eachlearning player might use to progressively refine their strategies.
Gradient feedback In this model, after the t-th match (repetition of the game) is completed, the(t) (t)learning player receives as input a vector, which represents the gradient ∇xi ui (xi , x−i ) of
the player’s expected utility evaluated in the current strategy of each player.
Trajectory bandit feedback In this feedback model, it is assumed that the game is simulated byhaving the players sample actions according to their current strategies. The learning player
observes the realized trajectory of play (the sequence of actions and observations from theroot of the player’s tree-form decision problem down to a terminal node), as well as therealized utility.
Bandit optimization feedback In this feedback model, the learning player only observes therealized utility, but not the realized trajectory of play. One way to think about this feedback
model is that at the beginning of each match, each player writes down their strategy, andsends it to a third party. The third party simulates the game on behalf of the players, andinforms every player of their expected utility. It might come as a surprise to the reader thateven from this extremely limited feedback, learning and convergence to equilibrium arepossible.
The majority of this dissertation will consider learning algorithms that operate with gradient
feedback, for the important reason that learning under trajectory bandit and bandit optimization
feedback can be reduced to it. Specifically, it is known that the feedback of the latter two models
can be used to construct an unbiased estimator of the gradient of the utility function, making it
feasible to retrofit a learning algorithm designed for gradient feedback into one for the trajectorybandit or bandit optimization feedback.

3.2

Mathematical abstraction of a predictive no-Φ-regret algorithm

We have seen in Section 3.1.1 that a desirable goal for a generic player i in a repeated game is toensure that the Φ-regret


(t)
(t)
Φ -Regi {xi , x−i }Tt=1

:= max
ϕ̂∈Φ

( T
X



(t)
(t)
ui ϕ̂(xi ), x−i





(t)
(t)
− ui xi , x−i



)
(3.1)

t=1

grows sublinearly as a function of the number of repetitions of the game T , no matter the strategies(t)x−i of the opponents. By construction of the sequence form, each player’s utility is linear the36§3.2. MATHEMATICAL ABSTRACTION OF A PREDICTIVE NO-Φ-REGRET ALGORITHM

player’s strategy, and in particular
(t)

(t)

ui ( · , x−i ) = ⟨∇xi ui (x1 , . . . , x(t)n ), · ⟩.
Hence, we can rewrite (3.1) to express the goal of Player i as ensuring that the quantity


(t)
(t)
Φ -Regi {xi , x−i }Tt=1

:= max
ϕ̂∈Φ

( T
XD

(t)
(t)
(t)
∇xi ui (x1 , . . . , x(t)n ), ϕ̂(xi ) − xi

)
E

,

(3.2)

t=1
(t)

grows sublinearly in the number of repetitions T , no matter the strategies x−i played by the
opponents. The mathematical abstraction of a no-Φ-regret algorithm generalizes (3.2), by asking
that
T D


E
X
(t)
(t)
(t)
(t)
Φ -Regi {xi , ui }Tt=1 := max
u(t) , ϕ̂(xi ) − xi ,ϕ̂∈Φ t=1

no matter the sequence of (potentially adversarially chosen) linear utility vectors u(t) , each of whichis assumed to be drawn from some bounded set. Moving away from the concrete choice of linear(t)(t)
utility vector ∇xi ui (x1 , . . . , xn ) to a generic u(t) without assumptions other than bounded
norm comes with advantages and disadvantages.
• Decoupling u(t) from the strategies of the opponents, and requiring that the no-Φ-regret
algorithm for Player i be able to guarantee sublinear Φ-regret no matter the sequence of u(t) ,removes Player i’s opponents from the picture. In other words, learning defined in thisabstracted way is properly a per-player endeavor.
• Another advantage is in that the generality afforded by the u(t) , which are now not constrained(t)(t)
to be in the form u(t) = ∇xi ui (x1 , . . . , xn ), enables the use of no-Φ-regret algorithms fortasks other than learning in games. In this sense, one can think of a no-Φ-regret algorithm
as an online optimization algorithm (Zinkevich, Bowling, Johanson, and Piccione, 2007).
• On the flip side, the generality comes at the cost of losing the ability to exploit any special(t)(t)
structure enjoyed by utility vectors of the form u(t) = ∇xi ui (x1 , . . . , xn ), as the no-Φregret algorithm has in general no assumption about the nature of the utility vectors it could
receive next.
The last point is not just of theoretical interest. When no assumptions are available about the√utility vectors u(t) (other than bounded norm), a lower bound of ΩT ( T ) on the regret is known(t)(t)(Shalev-Shwartz, 2012). Yet, the utilities u(t) = ∇xi ui (x1 , . . . , xn ) that arise while learning
games are structured and often have nice properties—for example changing slowly over time.
It is then natural to wonder what improved guarantees can be achieved in games specifically,§3. NO-REGRET LEARNING IN GAMES

37

and consequently how fast convergence to equilibrium can be guaranteed. This fundamental
question was first formulated and addressed by Daskalakis, Deckelbaum, and Kim (2011) within
the context of zero-sum games. Since then, there has been a considerable interest in extending theirguarantee to more general settings (S. Rakhlin and Sridharan, 2013; Syrgkanis, Agarwal, Luo,and Schapire, 2015; D. J. Foster, Li, Lykouris, Sridharan, and Tardos, 2016; Chen and Peng, 2020;
Daskalakis and Golowich, 2022; Piliouras, Sim, and Skoulakis, 2022). In particular, Daskalakis,
Fishelson, and Golowich (2021) established that when all players in a general nonsequential games
employ a variant of multiplicative weights update (MWU) (see Section 3.3.1), the regret of each player
grows nearly-optimally as OT (log4 T ) after T repetitions of the game, leading to an exponential
improvement over the guarantees obtained using traditional techniques within the no-regret
framework. In Chapters 6 and 7 of this dissertation, we will look into this question, extending
polylogarithmic guarantees to the significantly more challenging setting of imperfect-information
extensive-form games, and further improving on the bounds known in the literature, establishing
OT (log T ) guarantees.
Predictions The preceding discussion about the advantages and disadvantages of abstracting
learning in games via the framework of no-Φ-regret algorithms begs the natural question as towhat is the best compromise, or middle ground, between the appealing decoupled formulationand the ability to not lose track of the positive properties of game-induced utilities. In this thesis
we adopt the middle ground of learning with predictions, pioneered in the works of Chiang, Yang,
C.-J. Lee, Mahdavi, C.-J. Lu, R. Jin, and Zhu (2012) and S. Rakhlin and Sridharan (2013).
The framework of learning with predictions does not introduce any assumption on theutility vectors u(t) , but rather introduces the possibility that the no-Φ-regret algorithm is given a
prediction of the next utility function before the strategy is output. In this model the prediction
can either be received by an oracle as a generic input to the algorithm, or more typically it isassumed that it is constructed by the learning algorithm itself. In the latter case, a standard
choice is to use as prediction the last-received utility vector, as we will recall in Section 3.2.1. In
particular, this latter choice will be used in Chapters 6 and 7 and will lead to ÕT (1/T ) convergenceto equilibria. We formalize this choice in what we call the canonical optimistic learning setup (COLS)
in Section 3.2.1.
Definition of no-Φ-regret algorithm We are now ready to formalize in the concept of ano-Φ-regret algorithm in a definition.
Definition 3.1 (No-Φ-regret algorithm). Given a convex set X and a set Φ of linear
transformations ϕ : X → X , a No-Φ-regret algorithm for the set X is a model for a decision
maker that repeatedly interacts with a black-box environment. At each time t, the algorithm
interacts with the environment through two operations:38§3.2. MATHEMATICAL ABSTRACTION OF A PREDICTIVE NO-Φ-REGRET ALGORITHM• NextStrategy(m(t) ) informs the algorithm of the prediction vector m(t) . The algorithm
will output the next strategy x(t) ∈ X ;• ObserveUtility(u(t) ) provides the environment’s feedback to the no-regret algorithm, inthe form of a linear utility function X ∋ x 7→ ⟨u(t) , x⟩. The utility vector u(t) can depend
adversarially on the outputs x(1) , . . . , x(t) if the no-regret algorithm is deterministic(i.e., does not use randomness internallya ).
The Φ-regret cumulated up to any time T compared to a transformation ϕ̂ ∈ Φ is defined as
the quantityΦ -Reg(T ) (ϕ̂) :=T D
EXu(t) , ϕ̂(x(t) ) − x(t) ,

(3.3)

t=1

whereas the Φ-regret cumulated by to time T by the algorithm is the maximum Φ-regret
cumulated compared to any of the transformations in Φ,Φ -Reg(T ) := max Φ -Reg(T ) (ϕ̂).

(3.4)

ϕ̂∈Φ
a When randomness is involved, the utility vector cannot depend adversarially on x(t) or guaranteeing sublinear
regret would be impossible. Rather, u(t) must be conditionally independent on x(t) , given all past random outcomes.

Calls to NextStrategy and ObserveUtility keep alternating to each other: first, the no-regret
algorithm will output a point x(1) , then it will receive feedback u(1) from the environment, then it
will output a new point x(2) , and so on. The decision-making encoded by the no-regret algorithm
is online, in the sense that at each time t, the output of the no-regret algorithm can depend on theprior outputs x(1) , . . . , x(t−1) and corresponding observed utility vectors u(1) , . . . , u(t−1) , but noinformation about future utilities is available.

3.2.1

The canonical optimistic learning setup (COLS) for games

As mentioned in the discussion preceding Definition 3.1, a no-Φ-regret algorithm is an abstract
learning algorithm that outputs strategies (taking into account a prediction vector m(t) ), andreceives as feedback a utility vector u(t) , guaranteeing that the x(t) accumulate sublinear Φ-regretwith respect to the u(t) over time. We now reconnect the definition to the context of learningin games, by specifying how the m(t) and u(t) are defined. In particular, we will refer to these
specific choices as defining the canonical optimistic learning setup (COLS).
(t)
In the COLS, at all times t ∈ {1, 2, . . . } each player i ∈ [[n]] picks mixed strategies xi according(t)(t)
to some no-Φ-regret learning algorithm Ri .NextStrategy(mi ), where the prediction vector mi
(t)
(t−1)
(1)
is defined as the previous loss mi := ui
if t ⩾ 2, and mi := 0 otherwise. Then, all players
(t)
(t)
(t)
receive as feedback the utility vectors ui := ∇xi ui (x1 , . . . , xm ), that is, the gradient of their39§3. NO-REGRET LEARNING IN GAMES

own utility evaluated in the mixed strategies last output by the learning players. In this setup,(t)(t)
the Φ-regret accumulated by each player i ∈ [[n]] is equal to the quantity Φ -Regi ({xi , x−i }Tt=1 )defined in (3.1), enabling the wealth of connections between Φ-regret and equilibrium computation
laid out in Section 3.1.2.

3.2.2

The important special case of external regret minimization

The special case where Φ is chosen to be the set of constant transformations is so important that it
warrants its own definition and notation.
Definition 3.2 (No-regret algorithm). Given a set X , an no-external-regret algorithm for X , orsimply “no-regret algorithm for X ”, is a no-Φconst -regret algorithm, where Φconst is the set of all
constant transformations
Φconst := {ϕx̂ : x̂ ∈ X },where ϕx̂ : X ∋ x 7→ x̂.

Its corresponding Φconst -regret (Equations (3.3) and (3.4)) is called “external regret” or simply“regret”. We will often indicate external regret with the symbol Reg(T ) rather than Φconst -Reg.
In particular, we will let
Reg(T ) (x̂) :=TX⟨u(t) , x̂ − x(t) ⟩,
t=1

3.2.3

Reg(T ) := max Reg(T ) (x̂).

(3.5)

x̂∈X

Reducing Φ-regret minimization to external regret minimization

As mentioned, a no-external-regret algorithm is a particular case of a no-Φ-regret algorithm.
Then, the problem of constructing a no-external-regret algorithm for a set X cannot be harderthan the problem of constructing a no-Φ-regret algorithm for X for a richer set of transformation
functions Φ. It might then seem surprising that there exists a construction that reduces Φ-regret
minimization to regret minimization. More precisely, a result by G. J. Gordon, A. Greenwald, andMarks (2008) gives a way to construct a no-Φ-regret algorithm for a generic set X , starting fromany no-external-regret algorithm for the set of functions Φ. We present the result next, as it will be
an important component in the construction of learning algorithms that guarantee convergenceto the set of extensive-form correlated equilibria (Chapter 8).
Theorem 3.1 (G. J. Gordon, A. Greenwald, and Marks, 2008). Let R be a deterministic no-regret
algorithm for the set of transformations Φ whose (external) cumulative regret Reg(T ) grows
sublinearly in T , and assume that every ϕ ∈ Φ admits a fixed point ϕ(x) = x ∈ X . Then, a40§3.2. MATHEMATICAL ABSTRACTION OF A PREDICTIVE NO-Φ-REGRET ALGORITHM

no-Φ-regret algorithm RΦ can be constructed starting from R as follows:• Each call to RΦ .NextStrategy first calls NextStrategy on R to obtain the next transformation
ϕ(t) . Then, a fixed point x(t) = ϕ(t) (x(t) ) is computed and output.
• Each call to RΦ .ObserveUtility(u(t) ) with linear utility vector u(t) first constructs thelinear utility vector L(t) corresponding to the linear map ϕ 7→ ⟨u(t) , ϕ(x(t) )⟩, where x(t)is the last-output strategy. Then, it passes L(t) to R by executing R.ObserveUtility(L(t) ).b
Furthermore, the Φ-regret Φ -Reg(T ) cumulated up to time T by RΦ we have just defined isexactly equal to the (external) cumulative regret Reg(T ) cumulated by R:Φ -Reg(T ) (ϕ̂) = Reg(T ) (ϕ̂)∀ ϕ̂ ∈ Φ, T = 1, 2, . . . .

So, because the regret cumulated by R grows sublinearly by hypothesis, then so does theΦ-regret cumulated by RΦ .
b On the surface, it might look like L(t) is independent on the last output ϕ(t) of the no-regret algorithm R, andthus, that it trivially satisfies the requirements of Definition 3.2. However, that is not true: x(t) is a fixed point of
ϕ(t) , and since x(t) enters into the definition of L(t) , if R picks ϕ(t) randomly, it might very well be that L(t) is notconditionally independent on ϕ(t) . We sidestep this issue by requiring that R is deterministic (cf. Footnote a).

Proof. We propose an independent proof of the above statement given the brevity and eleganceof the argument. By construction, the algorithm R outputs transformations ϕ(1) , ϕ(2) , · · · ∈ Φand receives the linear utilities ϕ 7→ ⟨L(t) , ϕ⟩ := ⟨u(t) , ϕ(x(t) )⟩. Hence, its cumulative external
regret Reg(T ) is by definition
Reg(T ) (ϕ̂) =

T D
X

E
u(t) , ϕ̂(x(t) ) − ϕ(t) (x(t) ) .

t=1

Since by construction x(t) is a fixed point of ϕ(t) , then ϕ(t) (x(t) ) = x(t) , and therefore
Reg(T ) (ϕ̂) =T D
X

E(3.3)
u(t) , ϕ̂(x(t) ) − x(t) = Φ -Reg(T ) (ϕ̂),

t=1

as we wanted to show.

3.2.4

Degrees of predictivity

In this dissertation we propose a systematic classification of predictive regret minimization
algorithms on the basis of how effectively they are able to take into account predictions. In
particular, we introduce the following nomenclature.

§3. NO-REGRET LEARNING IN GAMES

41

Definition 3.3. A no-Φ-regret algorithm for a set X is said to be:• Weakly predictive relative to learning rate η > 0 and norm ∥ · ∥, if its Φ-regret satisfies a
relation of the form
!
T
X
1
(T
)
(t)
(t)
2
Φ -Reg= OT+η∥u − m ∥∗ ;η
t=1
• RVU-Predictive[3.b] relative to learning rate η > 0 and norm ∥ · ∥, if its Φ-regret satisfies
a relation of the form!
TTX
1 X (t)
1
(T
)
(t)
(t)
2
(t−1)
2
∥u − m ∥∗ −∥x − x∥ ;(3.6)+ηΦ -Reg
= OT
η
η t=2
t=1
• Strongly predictive relative to learning rate η > 0 and norm ∥ · ∥, if its Φ-regret satisfies
a relation of the form!
TTX
1 X (t)
1
(T
)
(t)
(t)
2
(t−1)
2
+η
∥u − m ∥∗ −∥x − x∥ .
max{0, Φ -Reg } = OT
η
η t=2
t=1

The following chain of implications follows directly from the definition above:(strongly predictive) =⇒ (RVU-predictive) =⇒ (weakly predictive).

3.3

No-external-regret algorithms for probability simplexes

In this section we review a few no-external-regret algorithms for probability simplexes. Several
more are known in the literature.

3.3.1

Multiplicative weights update (MWU) and its predictive variant (OMWU)

The predictive multiplicative weights update algorithm is arguably the best studied no-external-regret
algorithm with probability simplex, due to its strong properties that will recall in this subsection.
We remark that in the literature, the predictive multiplicative weights update algorithm is often
given under the assumption that the prediction m(t) is set to u(t−1) at all times t, as is standard
within the COLS, and is known commonly under the alternative names optimistic multiplicative
weights update, optimistic randomized weighted majority, and optimistic hedge. In this dissertation we[3.b] The term RVU stands for Regret bounded by Variation of Utilities (RVU) and was introduced by Syrgkanis, Agarwal,
Luo, and Schapire (2015) to describe the functional form (3.6).

42§3.3. NO-EXTERNAL-REGRET ALGORITHMS FOR PROBABILITY SIMPLEXES

present the algorithm in its general form, that is, with no assumptions on m(t) , but still refer to it
as OMWU to avoid confusion due to how widespread the acronym is. At all times t, OMWU produces
as strategy the distribution x(t−1) ∈ ∆d as specified in Algorithm 3.1.
Algorithm 3.1: Predictive multiplicative weights update algorithm (OMWU)
Data: Learning rates η (t) > 0
1u(0) , m(0) ← 0 ∈ ℝd ; x(0) ← d1 1 ∈ ∆dfunction NextStrategy(m(t) ∈ ℝd )[▷ set m(t) = 0 for non-predictive variant]3
w(t) ← u(t−1) − m(t−1) + m(t)4
for k ∈ [[d]] dox(t−1) [k] · exp{η (t) w(t) [k]}5x(t) [k] ← P(t−1) [k ′ ] · exp{η (t) w (t) [k ′ ]}k′ ∈[[d]] x(t)
6return x27function ObserveUtility(u(t) )[▷ No operation until the next call of NextStrategy]

The nonpredictive version of OMWU, called multiplicative weights update (MWU) (or alternatively
randomized weighted majority and hedge), can be obtained from OMWU as the special case in whichm(t) = 0 at all t. We mention the following well-known upper bound for the regret cumulated
by OMWU.
Theorem 3.2 (Syrgkanis, Agarwal, Luo, and Schapire (2015)). After any T iterations, theregret cumulated by the OMWU algorithm (Algorithm 3.1) set up with constant learning rate
η (t) = η > 0, satisfies the RVU-predictive regret bound
Reg(T ) ⩽

T
T
X
log d
1 X (t)+η∥u(t) − m(t) ∥2∞ −∥x − x(t−1) ∥21 .
η
4η
t=1
t=2

Furthermore, each iteration runs in O(d) time.

In particular, Theorem 3.2 immediately implies that as long as the losses and predictions have√√bounded norm, the choice of learning rate η (t) = 1/ T guarantees Reg(T ) = OT (1/ T ).
In nonsequential (that is, normal-form) games, where each player i ∈ [[n]] has a single decision
point with a set Ai of actions, OMWU can be used to learn Nash and coarse correlated equilibria(Section 5.2.2). Specifically, when each player i learns within the COLS using OMWU with the same,(t)
constant learning rate ηi := η as their learning algorithm Ri , the following strong properties are
known to hold. For simplicity we assume without loss of generality that all utilities in the game43

§3. NO-REGRET LEARNING IN GAMES

are in the range [0, 1] (if not, rescaling all utilities to satisfy the condition leaves the equilibria
unchanged).
Theorem 3.3 (OT (T 1/4 ) per-player regret; Syrgkanis, Agarwal, Luo, and Schapire, 2015). For−1/4
all T , if η = T√n−1 , the regret of each player i ∈ [[n]] is bounded as
(T )

Regi

√
⩽ (4 + log |Ai |) n − 1 · T 1/4 .

Theorem 3.4 (Near-optimal per-player regret; Daskalakis, Fishelson, and Golowich, 2021).
1
There exist universal constants C, C ′ > 1 so that, for all T , if η ⩽ Cn log
4 T , the regret of eachplayer i ∈ [[n]] is bounded as
(T )

Regi

⩽

log |Ai |+ C ′ log T.
ηTheorem 3.5 (Optimal regret sum; Syrgkanis, Agarwal, Luo, and Schapire, 2015).
1η ⩽ √8(n−1), at all times T the sum of the players’ regrets satisfies
Xi∈[[n]]

(T )

Regi

⩽

If

n
max log |Ai |.
η i∈[[n]]We remark that while Theorem 3.4 provides a substantially better bound than Theorem 3.3
asymptotically, for moderate values of T , Theorem 3.3 provides a better numerical bound.
When Γ is a two-player zero-sum game, the following also holds when the two players learn
within the COLS using OMWU.
(t)Theorem 3.6 (Last-iterate convergence). There exists a certain schedule of learning rates ηi(t)(t)
such that the players’ strategies (x1 , x2 ) converge to a Nash equilibrium of the game (Hsieh,
Antonakopoulos, and Mertikopoulos, 2021). Furthermore, if Γ has a unique Nash equilibrium(t)(x∗1 , x∗2 ) and each player uses any constant learning rate ηi := η ⩽ 18 , at all times t the(t)(t)
strategy profile (x1 , x2 ) satisfies
(t)

(t)

KL(x∗1 ∥ x1 ) + KL(x∗2 ∥ x2 ) ⩽ C(1 + C ′ )−t ,where the constants C, C ′ only depend on the game, and KL(· ∥ ·) denotes the KL-divergence
between two distributions (C. Wei, C. Lee, M. Zhang, and Luo, 2021).

44§3.3. NO-EXTERNAL-REGRET ALGORITHMS FOR PROBABILITY SIMPLEXES

3.3.2

Regret matching (RM), regret matching+ (RM+ ), and variants

Regret matching (RM, Hart and Mas-Colell, 2000) is a simple no-external-regret algorithm forprobability simplexes that has been widely used for computational game solving due to its
good practical performance and lack of hyperparameters. Regret matching+ (RM+ , Tammelin,
2014; Tammelin, Burch, Johanson, and Bowling, 2015) is a modification of RM which has been
repeatedly observed to perform better in practice, although no theoretical justification for such an
improved performance is known. Both of the algorithm predate the introduction of predictions as
a concept in online optimization, so their formulation—given in Section 4.3.1—explicitly ignoresthe prediction vector supplied to NextStrategy.
Algorithm 3.3: Regret matching+ (RM+ )Algorithm 3.2: Regret matching (RM)1r (0) ← 0 ∈ ℝd , x(0) ← 1/d ∈ ∆d1function NextStrategy( _ )θ (t) ← [r (t−1) ]+4
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
5
else
return x(t) ← any point in ∆dz (0) ← 0 ∈ ℝd , x(0) ← 1/d ∈ ∆dfunction NextStrategy( _ )θ (t) ← [z (t−1) ]+4
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
5
else
return x(t) ← any point in ∆d

2

2

3

3

function ObserveUtility(u(t) )7r (t) ← r (t−1) + u(t) − ⟨u(t) , x(t) ⟩1function ObserveUtility(u(t) )7z (t) ← [z (t−1) + u(t) − ⟨u(t) , x(t) ⟩1]+

6

6

Predictive variants of RM and RM+ will be given in Chapter 4. The predictive variants of RM
and RM+ retain all advantages of the original, nonpredictive algorithm (including the lack ofhyperparameters), while at the same time capitalizing from the benefit of (weak) predictivity,
further increasing the practical performance of these no-external-regret algorithms.
Regret matching and regret matching+ satisfy the following (nonpredictive) regret bound.

Theorem 3.7 (Farina, Kroer, and Sandholm, 2021b). The external regret cumulated up to any
time T by the RM and RM+ algorithm satisfies, for all η > 0, the bound
v
u T
u X
2
Reg(T ) ⩽ t2
u(t) − ⟨u(t) , x(t) ⟩1 2 .
t=1

In particular, Theorem 3.7 shows that whenever the norm of the utility vectors is bounded bya time-independent quantity, the external regret incurred by RM and RM+ is upper bounded as√
Reg(T ) = OT ( T ), a sublinear quantity.
Variants of RM have been proposed in the literature, including discounted regret matching(Discounted RM) and linear regret matching (Brown and Sandholm, 2019). These have been

45

§3. NO-REGRET LEARNING IN GAMES

observed to perform better in practice, though they are less well studied, and no predictive
versions are known. We recall Discounted RM and Linear RM in Algorithms 3.4 and 3.5 respectively.
Algorithm 3.4: Discounted regret matching(Discounted RM)
Data: α := 3/2, β := 0 discounting parameters
1

r

(0)

d

←0∈ℝ , x(0)d← 1/d ∈ ∆function NextStrategy( _ )3
θ (t) ← [r (t−1) ]+4
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
5
else
return x(t) ← any point in ∆d
2

(t)

d

function ObserveUtility(u ∈ ℝ )for k ∈ [[d]] do
8
if r (t−1) [k] > 0 then9r (t) [k] ← r (t−1) [k] · tα /(tα + 1)
10
else
11
r (t) [k] ← r (t−1) [k] · tβ /(tβ + 1)(t)
12r ← r (t) + u(t) − ⟨u(t) , x(t) ⟩1
6
7

Algorithm 3.5: Linear regret matching (Linear RM)1

r (0) ← 0 ∈ ℝd , x(0) ← 1/d ∈ ∆dfunction NextStrategy( _ )θ (t) ← [r (t−1) ]+4
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
5
else
return x(t) ← any point in ∆d23function ObserveUtility(u(t) ∈ ℝd )7r (t) ← r (t−1) + t · u(t) − ⟨u(t) , x(t) ⟩1
6

46§3.3. NO-EXTERNAL-REGRET ALGORITHMS FOR PROBABILITY SIMPLEXES

Part II

Computation of coarse-correlated andNash equilibria

Chapter 4

Composability of learning dynamics
and predictive counterfactual regret
minimization
4.1

Contributions and related work

In this chapter we will show that predictive no-external-regret learning algorithms lend themselves
to a rather natural compositional calculus. In particular, we will show that for many composite
strategy sets obtained via convexity-preserving operations on simpler sets (e.g., Cartesian product,
convex hull, intersection, and more), a predictive no-regret learning algorithm can be obtained by
combining, as black boxes, any predictive no-regret algorithms for the simpler sets. We call thesecomposition rules regret circuits. This approach has parallels with the calculus of convex sets
and functions found in books such as Boyd and Vandenberghe (2004). It likewise is reminiscentof disciplined convex programming (Grant, Boyd, and Ye, 2006), which emphasizes the solving of
convex programs via composition of simple convex functions and sets.

Applications to sequence-form polytopes An especially notable application of our frameworkis the construction of a weakly-predictive no-external-regret algorithm for sequence-form strategy
polytopes (Section 4.3). In that case, our compositional calculus will show how any weaklypredictive no-external-regret algorithm for a single-decision (i.e., nonsequential) problems canbe promoted to a weakly-predictive no-external-regret algorithm for tree-form problems. The
resulting algorithm is a generalization of the popular algorithm known as counterfactual regret
minimization (CFR) (Zinkevich, Bowling, Johanson, and Piccione, 2007). CFR and its newer
variants (Lanctot, Waugh, Zinkevich, and Bowling, 2009; Brown and Sandholm, 2015; Tammelin,50§4.1. CONTRIBUTIONS AND RELATED WORK

Burch, Johanson, and Bowling, 2015; Brown, Kroer, and Sandholm, 2017; Brown and Sandholm,
2017a; Brown and Sandholm, 2019; Brown and Sandholm, 2019), have been a central component
in several recent milestones in solving imperfect-information games. Bowling, Burch, Johanson,
and Tammelin (2015) used CFR+ to near-optimally solve heads-up limit Texas hold’em. Brown
and Sandholm (2017c) used CFR variants, along with other scalability techniques such as real-timeendgame solving (Ganzfried and Sandholm, 2015; Burch, Johanson, and Bowling, 2014; Moravcik,
Schmid, Ha, Hladik, and Gaukrodger, 2016; Brown and Sandholm, 2017b) and automated action
abstraction Brown and Sandholm, 2014, to create Libratus, an AI that beat top human specialist
professionals at the larger game of heads-up no-limit Texas hold’em. Moravčík, Schmid, Burch,
Viliam Lisý, Morrill, Bard, Davis, Waugh, Johanson, and Bowling (2017) also used CFR variants
and endgame solving to beat professional human players at that game. CFR was originallyproposed as an algorithm for approximating Nash equilibria in two-player zero-sum game, witha rather lengthy and complex proof. By using our regret circuit formalism, we will construct analgorithm that specializes into CFR when used in self-play in two-player zero-sum games, but isotherwise a no-external-regret algorithm in general. In other words, we move away from theunderstanding of CFR as a method operating on a two-player game tree, and instead embrace
the one-sided, TFDP-based point of view, whereby CFR is a learning algorithm for a genericplayer’s tree-form decision problem in a game. This point of view makes it apparent that CFR canbe used beyond two-player zero-sum games, for all applications in which no-external-regret isuseful (Section 3.1.2). All this comes with the benefit of a radically simpler proof of correctness,and enables using predictions within the CFR framework in a principled way. As we show in
Section 4.3.2, this leads to some of the fastest learning algorithms known today for equilibriumcomputation in large extensive-form games, when paired with predictive no-regret algorithms
for single-decision problems, such as ref PRM+ .

Other applications throughout the dissertation However, the applications of our compositional
calculus go well beyond constructing algorithms for sequence-form strategy spaces. The
framework laid out in this chapter will lead to practical algorithms when constructing noΦ-regret learning dynamics leading to extensive-form correlated equilibria (Chapter 8), andsocial-welfare-maximizing equilibria (Chapter 10).
Algorithms constructed via the regret circuits compositional calculus tend to perform betterin practice than those constructed using other approaches, for example those based on online
convex optimization techniques (Chapter 5). In fact, methods based on composition have several
positive features that are hard to mirror in other, more monolithic approaches to constructing
no-external-regret algorithms. For one, by treating every no-regret algorithm that appears in thecompositional construction as an independent black box, the compositional approach enables
one to select the best individual algorithm for each of them. Second, our framework is amenable
to pruning or warm-starting techniques in different parts of the composition, and substituting§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
51

one or more parts of the composition with an approximation. Finally, it is immediately clear inwhat cases the no-external-regret algorithms that are being composed can be run in distributedand parallel environments.

4.2

Regret circuits

In this section we begin developing a set of rules for how no-external-regret algorithms canbe constructed for composite convex sets. Specifically, given no-regret algorithms for convex
sets (say, X and Y) we show how to compose these no-regret algorithms for the following
standard convexity-preserving operations performed on the sets: convex hull, Cartesian product,affine transformation. In Section 4.2.5, we will also introduce and give a regret circuit for a
special operation, called scaled extension, which will become important later on in the dissertation,
specifically in Chapter 10.

4.2.1

Pictorial depiction of regret circuits

Regret circuits define the ways in which the inputs (predictions and utility vectors) and outputs(strategies) of no-regret algorithms can be combined to define a composite no-regret algorithm.
To aid intuition and memory, we will pictorially visualize regret circuits as circuits in which
components are no-regret-algorithms, and wires carry utility vectors, predictions, and decisions.
A simple example of what such a representation will look like in this dissertation is given to theright of Algorithm 4.1 on page 52.
• a no-external-regret algorithm is drawn as a box with the name of the algorithm placed at
the center, such as

RX. We will use the convention of calling the no-regret algorithms

that are being combined as RX , RY , et cetera. The subscript of R, say X or Y, corresponds tothe set from which the algorithm draws its strategies.
• Input wires to the no-regret algorithms carry prediction and utility vectors the algorithms
receive at the generic time t. They are dashed and colored dark red, like this:.
• Output wires from no-regret algorithms carry the strategy output at the generic time t. They
are solid lines colored dark blue, like this:.
• The nodedenotes an adder, that is a node that outputs the sum of all its inputs.
is used to denote an operation, other than addition, that manipulates its• The node
input(s) into its (single) output.
• The black dot denotes a connection point between wires: all wires leaving the connection
point carry the same signal (utility, prediction, or strategy) as the only wire entering theconnection point.
• The boundary of the regret circuits is marked by a gray rectangle, such as.

52§4.2. REGRET CIRCUITS

4.2.2

Cartesian product

Let X ⊆ ℝdx and Y ⊆ ℝdy be two sets, and let RX and RY be no-regret algorithms for X and Y
respectively. In this section we show that it is possible to minimize regret on X × Y by simply
minimizing it on X and Y independently—via RX and RY —and then combining the decisions.
More precisely, we can combine RX and RY to form a no-external-regret algorithm for the
Cartesian product X × Y ⊆ ℝdx +dy by means of the following natural procedure:(t)(t)• To output the next strategy, given a prediction[4.a] vector m(t) := (mx , my ) ∈ ℝdx +dy ,(t)(t)
we feed mx and my to X and Y respectively, and receive their next strategies x(t) ∈ X ,y (t) ∈ Y. We then proceed to outputting the strategy (x(t) , y (t) ) ∈ X × Y.
(t)

(t)

(t)

(t)

• Given the utility vector u(t) := (ux , uy ) ∈ ℝdx +dy , we forward ux to RX and uy to RY .
The algorithm we just described is summarized with pseudocode in Algorithm 4.1 and depictedpictorially contextually to its right.
(t)

(t)

(t)

(t)

m(t) := (mx , my )Algorithm 4.1: Regret circuit for X × Y
1

2
3
4

(t)

u(t) := (ux , uy )(t)function NextStrategy(m(t) := (mx , my ) ∈ ℝdx +dy )[▷ Set m(t) = 0 for non-predictive version](t)
x ← RX .NextStrategy(mx )(t)y (t) ← RY .NextStrategy(my )
(t)
(t)
(t)

return (x , y

my

(t)

uy

(t)

RX
(t)

RY

(t)

function ObserveUtility(u(t) := (ux , uy ) ∈ ℝdx +dy )(t)
6
RX .ObserveUtility(ux )
7

(t)

ux

)∈X ×Y

5

(t)

mx

y (t)

x(t)

(t)
RY .ObserveUtility(uy )(x(t) , y (t) )Regret analysis We now turn our attention to the analysis of the regret cumulated by the circuit.
By construction, at all times t the no-regret algorithms RX and RY produce iterates x(t) and y (t) ,(t)(t)
respectively, and observe utility vectors uX and uY , respectively. Hence, the regret accumulated
by RX and RY up to any T relative to any comparators x̂ ∈ X , ŷ ∈ Y is given by(T )RegX (x̂) :=

T
X
(t)
⟨u(t)
x , x̂ − x ⟩,
t=1

(T )
RegY (ŷ) :=

T
X
(t)
⟨u(t)
y , ŷ − y ⟩.

(4.1)

t=1

[4.a] In this dissertation predictions of the next utility vectors are always assumed to be given as input, with theunderstanding that the non-predictive version of the algorithm can be recovered by setting the prediction to be 0, thezero vector.

§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION53

On the other hand, the regret circuit produces strategies (x(t) , y (t) ) ∈ X ×Y and received utility(t)vectors (utX , uY ) at all times t. Hence, its accumulated regret compared to any (x̂, ŷ) ∈ X × Y
satisfies
*
!
!
!+
T
(t)
X
ux
x̂
x(t)
(T )
(T )
(T )
RegX ×Y (x̂, ŷ) :=−= RegX (x̂) + RegY (ŷ),
(4.2)
(t) ,
(t)
ŷ
y
u
y
t=1
where we used (4.1) in the last step. By taking a maximum over all (x̂, ŷ) ∈ X × Y in (4.2) we
further obtain
(T )

(T )

(T )

RegX ×Y = RegX + RegY .

(4.3)

In summary, collecting (4.2) and (4.3) we have just proven the following characterization of
the regret of Algorithm 4.1.
Theorem 4.1. The regret circuit of Algorithm 4.1 provides a no-regret algorithm for the(T )Cartesian product X × Y whose regret RegX ×Y accumulated up to any time T is related to
(T )
(T )
the regrets RegX and RegY accumulated by RX and RY , respectively, according to theequalities
(T )

(T )

(T )

i. RegX ×Y (x̂, ŷ) = RegX (x̂) + RegY (ŷ), for all x̂ ∈ X , ŷ ∈ Y; and
(T )

(T )

(T )

ii. RegX ×Y = RegX + RegY .
(T )

(T )

(T )

So, in particular, RegX ×Y grows sublinearly in T since RegX and RegY are sublinear by
the hypothesis that RX and RY are no-regret algorithms.

Extension to multiple sets We conclude the section with a couple of remarks on how one canadapt the construction we have been describing thus far to Cartesian products of more than two
sets, say the Cartesian product Z := X1 × X2 × · · · × Xn given access to no-regret algorithms RXi
for the individual sets. Two avenues are possible.
• First, one can observe that X1 × X2 × · · · × Xn = (((X1 × X2 ) × X3 ) × . . . ) × Xn . Hence, it ispossible to construct a composite no-regret algorithm for Z by composing Algorithm 4.1
n − 1 times: first, combining RX1 and RX2 to obtain algorithm RX1 ×X2 , then combining thealgorithm we just obtained with RX3 , and so on.
• Alternatively, we point out that the idea behind Algorithm 4.1 and its corresponding analysis
generalize straightforwardly to multiple sets, by defining a regret circuit that is able tooutput strategies on X1 × · · · × Xn by combining the individual strategies returned by theindividual algorithms RX1 , . . . , RXn .

54§4.2. REGRET CIRCUITS

4.2.3

Convex hull

We now show how to combine RX and RY for two sets X ⊆ ℝd and Y ⊆ ℝd to construct a
no-external-regret algorithm for the convex hullnoco{X , Y} := λx + (1 − λ)y : x ∈ X , y ∈ Y, λ ∈ [0, 1] .
The construction is more involved than in the case of Cartesian products we saw in Section 4.2.2,
but some of the ideas carry over. Like in the Cartesian product construction, we will use theno-regret algorithms RX and RY to propose candidate points x(t) , y (t) . However, in this case, wecannot simply return the pair (x(t) , y (t) ). Rather, we will use a third no-regret algorithm R∆2 forthe 2-simplex ∆2 to decide how to “mix” x(t) and y (t) . More precisely, we will do the following.
• To output the next strategy given the prediction vector m(t) ∈ ℝd , we ask RX and RY fortheir next strategies—call them x(t) and y (t) —upon supplying m(t) to both. Then, we will(t)(t)
ask R∆2 for its next strategy λ(t) = (λ1 , λ2 ) ∈ ∆2 given prediction(t)m∆2 :=(t)!
⟨m(t) , x(t) ⟩,⟨m(t) , y (t) ⟩(t)and return the convex combination λ1 x(t) + λ2 y (t) ∈ co{X , Y}.
• When at time t we receive the utility vector u(t) ∈ ℝd , we forward u(t) to RX and RY . Then,
we forward to R∆2 the utility vector(t)u∆2 :=!
⟨u(t) , x(t) ⟩.
⟨u(t) , y (t) ⟩(4.4)

The algorithm we just described is reported in pseudocode in Algorithm 4.2 and pictoriallycontextually to the right.

Regret analysis By construction, the no-regret algorithms RX and RY output strategies x(t) , y (t)and observe utilities u(t) at all time t. Hence, their regret cumulated up to time T relative to any
comparators x̂ ∈ X , ŷ ∈ Y is given by(T )RegX (x̂) :=TX⟨u(t) , x̂ − x(t) ⟩

(4.5)

t=1
(T )

RegY (ŷ) :=TX⟨u(t) , ŷ − y (t) ⟩.
t=1

(4.6)

§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
55

m(t) , u(t)Algorithm 4.2: Regret circuit for co{X , Y}
1function NextStrategy(m(t) ∈ ℝd )[▷ Set m(t) = 0 for non-predictive version]

5x(t) ∈ X ← RX .NextStrategy(m(t) )y (t) ∈ Y ← RY .NextStrategy(m(t) ) (t) (t) ⟨m , x ⟩(t)m∆2 ←⟨m(t) , y (t) ⟩
(t)
(t)
(t)
(t)
λ := (λ1 , λ2 ) ∈ ∆2 ← R∆2 .NextStrategy(m∆2 )6

return λ1 x(t) + λ2 y (t) ∈ co{X , Y}

2
3
4

(t)

RY

RX

y (t)

x(t)
(t)
m∆2

(t)

(t)
u∆2

R∆2

function ObserveUtility(u(t) )RX .ObserveUtility(u(t) )9
RY .ObserveUtility(u(t) ) (t) (t) ⟨u , x ⟩
(t)
:=
2
10
R∆ .ObserveUtility u∆2⟨u(t) , y (t) ⟩
7

λ(t)

8

(t)

(t)

λ1 x(t) + λ2 y (t)
(t)

(t)

Similarly, the no-regret algorithm R∆2 outputs strategies λ(t) = (λ1 , λ2 ) at all times t and(t)observes utility vectors u∆2 , defined in (4.4), at all times t. So, the regret cumulated by R∆2compared to any λ̂ ∈ ∆2 is given by(T )Reg∆2 (λ̂) :=

T
X

(t)

⟨u∆2 , λ̂ − λ(t) ⟩

t=1

=

T
X

!
(t)

(t)

(t)

λ̂1 ⟨u , x ⟩ + λ̂2 ⟨u , y

(t)

⟩ −

T
X

!
(t)
(t)
λ1 ⟨u(t) , x(t) ⟩ − λ2 ⟨u(t) , y (t) ⟩ .

t=1

t=1

The following theorem connects these three quantities, showing that when all of them are
sublinear, Algorithm 4.2 indeed yields a no-regret algorithm.
Theorem 4.2. The regret circuit of Algorithm 4.2 provides a no-regret algorithm for the(T )convex hull co{X , Y} whose regret Regco{X ,Y} up to any time T , is related to the regrets
(T )

(T )

(T )

RegX , RegY , and Reg∆2 accumulated by RX , RY , and R∆2 , respectively, according toi. for all comparators x̂ ∈ X , ŷ ∈ Y, and (λ̂1 , λ̂2 ) ∈ ∆2 ,
(T )

(T )

(T )

(T )

Regco{X ,Y} (λ̂1 x̂ + λ̂2 ŷ) = Reg∆2 (λ̂1 , λ̂2 ) + λ̂1 RegX (x̂) + λ̂2 RegY (ŷ),

(T )
(T )
(T )
(T )
ii. Regco{X ,Y} ⩽ Reg∆2 + max RegX , RegY .
(T )

(T )

(T )

(T )

So, in particular, Regco{X ,Y} grows sublinearly in T since RegX , RegY , and Reg∆2 aresublinear by the hypothesis that RX , RY , and R∆2 are no-regret algorithms.

56§4.2. REGRET CIRCUITSProof. We start by proving the first point. Starting from the definition of regret, we have
(T )Regco{X ,Y} (λ̂1 x̂ + λ̂2 ŷ) :=

T
X
(t)
(t)
⟨u(t) , λ̂1 x̂ + λ̂2 ŷ⟩ − ⟨u(t) , λ1 x(t) + λ2 y (t) ⟩
t=1

T
X

= λ̂1

T
X
⟨u , x̂⟩ + λ̂2⟨u(t) , ŷ⟩ −
(t)

t=1

t=1

T
X

!
(t)
(t)
λ1 ⟨u(t) , x(t) ⟩ + λ2 ⟨u(t) , y (t) ⟩

(4.7)

t=1

for all comparators x̂ ∈ X , ŷ ∈ Y, and (λ̂1 , λ̂2 ) ∈ ∆2 . From (4.5) and (4.6) we can relate the firsttwo sums to the regrets accumulated by RX and RY , respectively, according to
T
T
X
X
(T )
⟨u(t) , x̂⟩ = RegX (x̂) +⟨u(t) , x(t) ⟩,
t=1

(4.8)

t=1

T
T
X
X
(T )
⟨u(t) , ŷ⟩ = RegY (ŷ) +⟨u(t) , y (t) ⟩.
t=1

(4.9)

t=1

Plugging Equations (4.8) and (4.9) into (4.7), we obtain(T )Regco{X ,Y} (λ̂1 x̂ + λ̂2 ŷ) = λ̂1(T )RegX (x̂) +

!
!
T
T
X
X
(T )
(t)
(t)
(t)
(t)
⟨u , x ⟩ + λ̂2 RegY (ŷ) +⟨u , y ⟩
t=1

−

t=1
T
X

!
(t)
(t)
λ1 ⟨u(t) , x(t) ⟩ + λ2 ⟨u(t) , y (t) ⟩

t=1
(T )
(T )
(T )
= λ̂1 RegX (x̂) + λ̂2 RegY (ŷ) + Reg∆2 (λ̂1 , λ̂2 ).

(4.10)

This concludes the proof of the first point. To show the second point, it is enough to note thatfor any (λ̂1 , λ̂2 ) ∈ ∆2 ,
n
o
(T )
(T )
(T )
(T )
λ̂1 RegX (x̂) + λ̂2 RegY (ŷ) ⩽ max RegX (x̂), RegY (ŷ) .

(4.11)

Hence,
Reg(T ) :=

=

⩽

max

Reg(T ) (λ̂1 x̂ + λ̂2 ŷ)

max

n
o
(T )
(T )
(T )
Reg∆2 (λ̂1 , λ̂2 ) + λ̂1 RegX (x̂) + λ̂2 RegY (ŷ)(From (4.10))

max

n
n
oo
(T )
(T )
(T )
Reg∆2 (λ̂1 , λ̂2 ) + max RegX (x̂), RegY (ŷ)(From (4.11))(λ̂1 ,λ̂2 )∈∆2x̂∈X ,ŷ∈Y(λ̂1 ,λ̂2 )∈∆2x̂∈X ,ŷ∈Y(λ̂1 ,λ̂2 )∈∆2x̂∈X ,ŷ∈Y§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
57


(T )
(T )
(T )
= Reg∆2 + max RegX , RegY
as we wanted to prove.

Extension to multiple sets Of course, the construction shown in Algorithm 4.2 can be readilyextended to handle the convex hull co{X1 , . . . , Xn } of n sets as follows.
• At each time instant t, we let m no-regret algorithms RX1 , . . . , RXm (for X1 , . . . , Xm , respec(t)(t)
tively) output their next strategies x1 , . . . , xm . Those strategies are combined according to
the convex combination coefficients λ(t) output by the additional no-regret algorithm R∆m(t) (t)(t) (t)for the m-simplex ∆m to form the convex combination strategy λ1 x1 + · · · + λm xm ∈co{X1 , . . . , Xm }.
• The utility vector u(t) is fed into all the no-external-regret algorithms RXi (i = 1, . . . , m).
(t)Then, the utility vector u∆m , defined as(t)⟨u(t) , x1 ⟩


(t)
..
,
u∆m = 
.


(t)
(t)
⟨u , xm ⟩is input into a no-regret algorithm for the m-simplex ∆m .
With the same argument used earlier, it follows that the regret bound in that case is
(T )

(T )

Regco{X1 ,...,Xm } ⩽ Reg∆m + max{RegTX1 , . . . , RegTXm }.

4.2.4

Affine transformation and Minkowski sum′

Let H : ℝd → ℝd , x 7→ Ax + b be an affine map between two Euclidean spaces, and let X ⊆ ℝd
be a strategy set for which a no-regret algorithm RX is given. In this section, we will show how
RX can be adapted into a no-regret algorithm for the strategy set′

H(X ) := {Ax + b : x ∈ X } ⊆ ℝd ,the image of X under the affine map H. In this case we use this rather natural procedure:• Given a prediction vector m(t) ∈ ℝd , we feed A⊤ m(t) to RX , receive its next strategy
x(t) ∈ X , and output H(x(t) ) = Ax(t) + b.
• Given a utility vector u(t) ∈ ℝd , we feed A⊤ u(t) to RX .
The construction is summarized by the circuit in Figure 4.1.

58

§4.2. REGRET CIRCUITSm(t)A⊤ m(t)u(t)A⊤ u(t)RX

H(x(t) )x(t)Figure 4.1: Regret circuit for the image H(X ) of X under the affine transformation H.

Regret analysis
regretWe now analyze the regret circuit we have just described, seeking to relate the(T )RegH(X ) (ẑ) :=TX⟨u(t) , ẑ − H(x(t) )⟩t=1

accumulated by the circuit relative to a generic comparator ẑ ∈ H(X ), to that accumulated by RX .
By construction, at all times t the no-regret algorithm RX produces iterates x(t) ∈ X and receives
the utility vector A⊤ u(t) . Hence, the regret accumulated by RX up to any time T is given by(T )RegX (x̂) :=T
X⟨A⊤ u(t) , x̂ − x(t) ⟩ =

t=1

=

T
X

T
X
⟨u(t) , Ax̂ − Ax(t) ⟩
t=1
(T )

⟨u(t) , (Ax̂ + b) − (Ax(t) + b)⟩ = RegH(X ) (H(x̂)).

t=1

Since every ŷ ∈ H(X ) can be written as ŷ = H(x̂) for some x̂ ∈ X , we can further conclude
(T )
(T )
that RegX = RegH(X ) . In conclusion, we have shown the following.
Theorem 4.3. The regret circuit depicted in Figure 4.1 provides a no-regret algorithm for the(T )image H(X ) of X under affine transformation H, whose regret RegH(X ) accumulated up to(T )any time T is related to the regret RegX accumulated by RX according to
(T )

(T )

i. RegH(X ) (H(x̂)) = RegX (x̂) for all comparators x̂ ∈ X ; and
(T )

(T )

ii. RegH(X ) = RegX .
(T )

(T )

So, in particular, RegH(X ) grows sublinearly in T since RegX is sublinear by the hypothesisthat RX is a no-regret algorithms.
Application: Minkowski sum As a straightforward application, the above construction canbe employed to construct a no-regret algorithm for the Minkowski sum X + Y := {x + y : x ∈X , y ∈ Y} of two sets. Indeed, note that X + Y = σ(X × Y), where σ : (x, y) 7→ x + y is alinear map. Hence, we can combine the construction for affine transformations presented in this§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
59

section, together with the construction of the Cartesian product (Section 4.2.2, Algorithm 4.1).
The resulting regret circuit is in Figure 4.2.

m

RX

(t)

u(t)

RY

x(t)
x(t) + y (t)y (t)Figure 4.2: Regret circuit for the Minkowski sum X + Y.

4.2.5

Scaled extension

In this section we introduce the last regret circuit that will be used in the rest of the dissertation. It
refers to a particular convexity-preserving operation, scaled extension (defined below), which wasintroduced in (Farina, Ling, Fang, and Sandholm, 2019b) to capture certain correlated strategy
spaces. We will make extensive use of the scaled extension operation in Chapters 9 and 10.
Definition We now introduce the definition of the scaled extension operation.
Definition 4.1. Let X and Y be nonempty sets, and let h : X → ℝ+ be a nonnegative affine
real function. The scaled extension of X with Y via h is defined as the seth

X ◁ Y := {(x, y) : x ∈ X , y ∈ h(x)Y}.

As we show next, the operation above preserves convexity, non-emptiness, and compactnessof the sets.
Lemma 4.1. Let X , Y and h be as in Definition 4.1. If X and Y are nonempty, compact, orhconvex, then X ◁ Y is nonempty, compact, and convex, respectively.

h

Proof. Let Z := X ◁ Y. We break the proof into three parts:• Non-emptiness. Since X and Y are nonempty by hypothesis, let x and y be arbitrary
points in X and Y. The element (x, h(x)y) belongs to Z and therefore Z is nonempty.
• Compactness. We now prove that Z is a compact set via the Heine-Borel theorem, by
proving that it is bounded and closed. First, we argue that Z is bounded. Indeed, note thath is affine and therefore continuous, and since X is compact we conclude by Weierstrass’
theorem that h∗ := maxx∈X h(x) exists and is finite. Hence, any (x, h(x)y) satisfies

60§4.2. REGRET CIRCUITS∥(x, h(x)y)∥1 ⩽ ∥x∥1 + h∗ ∥y∥1 < ∞,since both X and Y are compact and hence bounded, showing that Z is bounded. Now, we
argue that Z is (sequentially) closed. Indeed, let {z i } → z̄ be a convergent sequence suchthat z i ∈ Z for i = 1, 2, . . . ; we will prove that z̄ ∈ Z. By definition of Z, for all i it must
be z i = (xi , h(xi )y i ) for some (xi , y i ) ∈ X × Y. Since {z i } converges, then the sequences{xi } and {h(xi )y i } must also converge. Let x̄ := lim xi ; by closedness of X , it must bex̄ ∈ X . Furthermore, by continuity of h, lim h(xi ) = h(x̄). Now, using the (sequential)
compactness of Y, we can assume without loss of generality that {y i } convergesa ; let Y ∋ȳ := lim y i . By the usual properties of limits, z̄ = lim z i = lim (xi , h(xi )y i ) = (x̄, h(x̄)ȳ),and since x̄ ∈ X , ȳ ∈ Y we have z̄ ∈ Z and Z is sequentially closed.
• Convexity. Let z, z ′ be any two points in Z, and let λ ∈ (0, 1). By definition of Z, there
must exist x, x′ ∈ X and y, y ′ ∈ Y such that z = (x, h(x)y) and z ′ = (x′ , h(x′ )y ′ ). If
h(x) = h(x′ ) = 0, then the affinity of h implies h(λx + (1 − λ)x′ ) = 0 and therefore
λz + (1 − λ)z ′ = λx + (1 − λ)x′ , 0= λx + (1 − λ)x′ , h(λx + (1 − λ)x′ )y ∈ Z.
On the other hand, if at least one of h(x) and h(x′ ) is strictly positive, then the quantitys := λh(x) + (1 − λ)h(x′ ) is also strictly positive and we can write
λz + (1 − λ)z ′ = λx + (1 − λ)x′ , λh(x)y + (1 − λ)h(x′ )y ′



λh(x)
(1 − λ)h(x′ ) ′= λx + (1 − λ)x′ , s
y+
y
s
s



λh(x)
(1 − λ)h(x′ ) ′= λx + (1 − λ)x′ , h(λx+(1−λ)x′ )
y+
y
,
s
s
where the last equality follows from the definition of s together with the affinity of h. The
quantity in the square bracket is a convex combination of the vectors y and y ′ , and it istherefore equal to some y ′′ ∈ Y by convexity of Y. Hence,
λz + (1 − λ)z ′ = λx + (1 − λ)x′ , h(λx + (1 − λ)x′ ) y ′′ ∈ Z,
as we wanted to show.
a Or else, extract a convergent subsequence.

Regret circuit We now show that it is always possible to construct a no-regret-algorithm forh

Z = X ◁ Y, where h(x) = ⟨a, x⟩ + b is a nonnegative affine function, starting from no-regret§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
61′

algorithms RX , RY for X ⊆ ℝd and Y ⊆ ℝd , respectively. In particular, we consider the following
algorithm.
(t)

′

(t)

• To output the next strategy, given prediction vector m(t) := (mx , my ) ∈ ℝd × ℝd , we feed(t)
vector my to RY , and receive its next strategy y (t) ∈ Y. Then, we construct the prediction
vector
(t)
(t)
(t)
m̃(t)
x := mx + ⟨my , y ⟩a,feed it to RX , and receive its next strategy x(t) ∈ X .
(x(t) , h(x(t) )y (t) ) ∈ Z.
(t)

Finally, we return the point

′

(t)

(t)

• When at time t we receive the utility vector u(t) := (ux , uy ) ∈ ℝd × ℝd , we forward uy
to RY , and forward the utility vector
(t)
(t)
(t)
ũ(t)
x := ux + ⟨uy , y ⟩a(4.12)

to RX .
The regret circuit is summarized pictorially in Figure 4.3.

(t)

(t)

mx , ux(t)(t)(mx , my )(t) (t)(ux , uy )

x(t)

RX

⟨y

(t)


x(t)
h(x(t) )y (t)(t), uy ⟩a

(t)

my

(t)
uy

RY

y (t)
h

Figure 4.3: Regret circuit for the scaled extension X ◁ Y.

Regret analysis We now relate the regret accumulated by the construction we just described up
to any time T compared to a generic point (x̂, h(x̂)ŷ) ∈ Z, that is the quantity

Reg

(T )
h

(x̂, h(x̂)ŷ) :=

X ◁Y
(T )

T
X
t=1

(T )

*

x̂
u(t) ,h(x̂)ŷ!
−x(t)h(x(t) )y (t)!+,(4.13)

to the regrets RegX , RegY accumulated by RX and RY respectively. By construction, at all(t)times t the no-regret algorithm RY outputs strategies y (t) ∈ Y and receives utilities uy . Hence,
the regret accumulated by RY up to time T compared to any ŷ ∈ Y is

62

§4.2. REGRET CIRCUITS(T )RegY (ŷ) :=

T
X
(t)
⟨u(t)
y , ŷ − y ⟩.

(4.14)

t=1

The no-regret algorithm RX outputs strategies x(t) ∈ X and receives utilities ũ(t)x defined in (4.12);
hence its regret compared to any x̂ ∈ X is(T )RegX (x̂) :=TX(t)⟨ũ(t)
x , x̂ − x ⟩.

(4.15)

t=1

The connection between the three regret quantities defined above is given in the next theorem.
Theorem 4.4. The regret circuit of Figure 4.3 provides a no-regret algorithm for the scaledhextension X ◁ Y whose regret Reg

(T )
h

X ◁Y

(T )

up to any time T , is related to the regrets RegX and(T )RegY accumulated by RX and RY , respectively, according toi. Reg

(T )
h

X ◁Y

ii. Reg

(T )
h

X ◁Y

(T )

(T )

(x̂, h(x̂)ŷ) = RegY (ŷ) + h(x̂) RegX (x̂) for all x̂ ∈ X , ŷ ∈ Y; and
(T )

(T )

⩽ RegY + h∗ · RegX , where h∗ := maxx∈X h(x).[4.b]So, in particular, Reg

(T )
h

X ◁Y

(T )

(T )

grows sublinearly in T since RegX and RegY are sublinear by

the hypothesis that RX and RY are no-regret algorithms.

Proof. By expanding the definition of ũ(t)x given in (4.12) in the definition (4.15) of regret for
RX , we obtain(T )RegX (x̂) =

T
X

!
(t)
⟨u(t)
x , x̂ − x ⟩

+

t=1

=

T
X

T
X

!
(t)
⟨u(t)
y ,y ⟩⟨a, x̂⟩ −

t=1

!
(t)
⟨u(t)
x , x̂ − x ⟩

+

t=1

T
X

T
X

(t)
(t)
⟨u(t)
y , y ⟩ · ⟨a, x ⟩

t=1

!
(t)
⟨u(t)
y ,y ⟩

⟨a, x̂⟩ + b

t=1

−

T
X


(t)
(t)
⟨u(t)
y , y ⟩ · ⟨a, x ⟩ + b

t=1

=

T
X
t=1

!
(t)
⟨u(t)
x , x̂ − x ⟩

+

T
X

!
(t)
⟨u(t)
y , y ⟩ h(x̂) −

t=1

T
X
(t)
(t)
⟨u(t)
y , y ⟩ · h(x ).
t=1

Using (4.14), we can rewrite the middle sum as[4.b] Since h is affine and X is compact, h∗ exists and is finite by Weierstrass’ theorem.

§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
63

T
T
X
X
(T )
(t)
⟨u(t)
,
y
⟩
=
−
Reg
(ŷ)
+
⟨u(t)
y
y , ŷ⟩,
Y
t=1

t=1

and hence(T )RegX (x̂) =

T
X

!
(t)
⟨u(t)
x , x̂ − x ⟩(T )− RegY (ŷ) +

+

t=1

T
X

!
⟨u(t)
y , ŷ⟩

h(x̂)

t=1

−

T
X

(t)
(t)
⟨u(t)
y , y ⟩ · h(x )

t=1

*

(T )
= −h(x̂) RegY (ŷ) +

T
X

(t)
ux
(t)
uy

t=1
(T )

= −h(x̂) RegY (ŷ) + Reg

(T )

!

x̂
,
h(x̂)ŷ!
−x(t)h(x(t) )y (t)!+(x̂, h(x̂)ŷ),

h

X ◁Y

where we used (4.13) in the last equality. Rearranging yields the result.

4.3

Predictive counterfactual regret minimization paradigm

The set of sequence-form strategies can be decomposed in a bottom-up fashion by using theconvex hulls and Cartesian products operations. We illustrate this intuitively by means of a small
example.
Example 4.1. Consider the small tree-form decision problem of Figure 4.4, which was
originally introduced in Example 2.7. As our construction is bottom-up, we will denote withthe symbol Qv , with v ∈ J ∪ K, the partial sequence-form strategy space corresponding to
the subtree rooted at v.
A
1

2
D
789

B

C

3 4

5 6

Figure 4.4: Sequential decision-making problem used in the example.

We can characterize the set of sequence-form strategies as follows:

64§4.3. PREDICTIVE COUNTERFACTUAL REGRET MINIMIZATION PARADIGM• At the terminal decision nodes j = B, C, D, Qj is a probability simplex. Specifically,
QB = QC = ∆2 and QD = ∆3 .
• At the (only) observation node k, a strategy for the subtree rooted at k must provide
independent strategies for the subtrees rooted at B and C. So,
Qk = QB × QC
is the Cartesian product of the strategy spaces for the subtrees rooted B and C.
• At the decision node A, we need to first pick a probability distribution of play for thetwo actions (sequences 1 and 2). Let’s call the probabilities assigned to those actions as
λ1 and λ2 , respectively. Clearly, (λ1 , λ2 ) ∈ ∆2 . Once the probabilities of sequences 1 and
2 are chosen, strategies for the subtrees rooted at the observation node k and D. Since
sequence-form strategies associate to each sequence σ the product of the probabilities of
all actions on the path form the root of the TFDP to σ, the set of all valid sequence-form
strategies for the subtree rooted in A (that is, the whole TFDP) isno
QA = (λ1 , λ2 , λ1 xk , λ2 xD ) : (λ1 , λ2 ) ∈ ∆2 , xk ∈ Qk , xD ∈ QD
  



0
1


  
 0   1    = co  , 
 .

  





Qk   0 


 0
Q≽D 
The above approach can be used in any TFDP. In particular, we always have that the sequenceform strategy space of a subtree rooted at an observation node is the Cartesian product of thesequence-form strategy spaces of the children subtrees. Furthermore, the sequence-form strategy
space of a subtree rooted at a decision point is the convex hull of the sequence-form strategy
spaces of the children subtrees, augmented with the indicators of the actions. These inductive
rules are summarized in Table 4.1.
By applying the regret circuits described in Sections 4.2.2 and 4.2.3, we can then construct a
no-regret algorithms for any sequence-form strategy space. The resulting no-regret algorithm iscalled CFR, whose pseudocode is given in Algorithms 4.3 and 4.4. In a nutshell, CFR decomposes
the problem of minimizing regret on the whole tree-form decision process into local regret
minimization problems at each of the individual decision nodes j ∈ J . Any no-regret algorithm
Rj for simplex domains can be used to solve the local regret minimization problems. Popular
options are the regret matching (RM) algorithm, and the regret matching+ (RM+ ) algorithm
discussed in Section 3.3.2.
By combining the regret guarantees analyzed for the Cartesian product and convex hull regret§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
65

Algorithm 4.3: Predictive CFR (weakly-predictive no-external-regret algorithm for sequenceform strategy polytope)
Data: {Rj : j ∈ J }: no-regret algorithms for ∆Aj ; one for each decision node j ∈ J .
function NextStrategy(m(t) )[▷ Compute the expected prediction for each subtree rooted at each v ∈ J ∪ K]
2
M (t) ← empty dictionary[▷ Maps keys J ∪ K ∪ {⊥} to real numbers](t)3
M [⊥] ← 0
4
for each node in the tree v ∈ J ∪ K in bottom-up order in the TFDP do
5
if v ∈ J then6
Let j = v
1(t)[▷ At each decision node j ∈ J , we now construct the counterfactual utility vector mj ]
7
8
9

(t)
mj ← 0 ∈ ℝAj

for each action a ∈ Aj do(t)
mj [a] ← m(t) [ja] + M (t) [ρ(j, a)][▷ Query each of the Rj for their strategy local at each decision node]
(t)

(t)

10

bj ∈ ∆Aj ← Rj .NextStrategy(mj )11[▷ Update expected values using the new local strategy]
X (t)
(t)
M (t) [j] ←bj [a] · mj [a]
a∈Aj

13

else
Let k = v

14

M (t) [k] ←12XM (t) [ρ(k, s)]s∈Sk[▷ Construct the sequence-form representation of the strategy that plays according to the
(t)

15
16
17
18
19
20
21
22

distribution bj at each decision node j ∈ J ]x(t) = 0 ∈ ℝΣfor each decision node j ∈ J in top-down order in the TFDP dofor each action a ∈ Aj do
if pj = ∅ then(t)x(t) [ja] ← bj [a]else(t)x(t) [ja] ← x(t) [pj ] · bj [a]return x(t)(The algorithm continues in Algorithm 4.4)

66

§4.3. PREDICTIVE COUNTERFACTUAL REGRET MINIMIZATION PARADIGM

Algorithm 4.4: (Continued) Predictive CFR (weakly-predictive no-external-regret algorithm
for sequence-form strategy polytope)
function ObserveUtility(u(t) ∈ ℝΣ )[▷ Compute the expected utility for each subtree rooted at each v ∈ J ∪ K]
2
V (t) ← empty dictionary;[▷ Maps keys J ∪ K ∪ {⊥} to real numbers]
3
V (t) [⊥] ← 0;
4
for each node in the tree v ∈ J ∪ K in bottom-up order in the TFDP do
5
if v ∈ J then6
Let j = v;


X (t)
7
V (t) [j] ←bj [a] · u(t) [ja] + V (t) [ρ(j, a)] ;
1

a∈Aj

9

else
Let k = v;

10

V (t) [k] ←8XV (t) [ρ(k, s)];

s∈Sk
(t)

11
12
13
14
15

[▷ At each decision node j ∈ J , we now construct the counterfactual utility vector uj ]for each decision node j ∈ J do(t)uj ← 0 ∈ ℝAj ;for each action a ∈ Aj do(t)
uj [a] ← u(t) [ja] + V (t) [ρ(j, a)];(t)

Rj .ObserveUtility(uj );§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
67

Decision node

···
Q1

Qn

Observation point   
 
e2
en 
e1




 


  










X1   0 
 0 co  0 ,X2 , . . . , 0  .. 
 ..   .. 
 . 
 .  . 






0
0
Xn

···
X1 × X2 × · · · × Xn
Q1

Qn

Table 4.1: Bottom-up construction rules for sequence-form strategy spaces. ei denotes the i-th indicator
vector, that is, the vector whose entries are all 0 except for the entry in position i, which is set to 1.

circuits (Theorems 4.1 and 4.2), it is immediate that the regret cumulated by the CFR algorithm
satisfies the following bound.
(T )Proposition 4.1. Let Regj (j ∈ J ) denote the regret cumulated up to time T by each of
the no-regret algorithms Rj . Furthermore, let bj ∈ ∆Aj be arbitrary strategies for the local
decision nodes j, and let x̂ ∈ Q be the sequence-form strategy corresponding to the behavioral
strategy {bj : j ∈ J }. Then, the regret Reg(T ) cumulated by CFR (Algorithms 4.3 and 4.4) up
to time T satisfies the equality
Reg(T ) (x̂) =

X

(T )

x̂[pj ] · Regj (b̂j ).

j∈J

In particular,
Reg(T ) ⩽X(T )max{0, Regj }.

j∈J

4.3.1

Predictive RM (PRM) and predictive RM+ (PRM+ ) algorithms

As mentioned in Section 3.3.2, variants of the RM algorithm such as RM+ and Discounted RM haveemerged as the preferred no-regret-algorithms for use in game theoretic settings, due to theirstrong performance in practice and their lack of hyperparameters. However, those algorithms do
not support prediction, and therefore do not lead to an interesting predictive variant of CFR.
Motivated by the desire to combine the benefits of RM-based methods with those afforded
by predictivity, in this section we introduce the first predictive versions of RM and RM+ . The
resulting algorithms, coined respectively predictive RM (PRM) and predictive RM+ (PRM+ ), are given
in Algorithms 4.5 and 4.6. The algorithms coincide with RM and RM+ given in Algorithms 3.2

68§4.3. PREDICTIVE COUNTERFACTUAL REGRET MINIMIZATION PARADIGM

Algorithm 4.5: Predictive regret matching (PRM)
1
2

3
4
5
6
7

r (0) ← 0 ∈ ℝd , x(0) ← 1/d ∈ ∆dfunction NextStrategy(m(t) )[▷ Set m(t) = 0 for non-predictive version]θ (t) ← [r (t−1) + m(t) − ⟨m(t) , x(t−1) ⟩1]+
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
else
return x(t) ← any point in ∆dfunction ObserveUtility(u(t) )r (t) ← r (t−1) + u(t) − ⟨u(t) , x(t) ⟩1

Algorithm 4.6: Predictive regret matching+ (PRM+ )12

z (0) ← 0 ∈ ℝd , x(0) ← 1/d ∈ ∆dfunction NextStrategy(m(t) )[▷ Set m(t) = 0 for non-predictive version]θ (t) ← [z (t−1) + m(t) − ⟨m(t) , x(t−1) ⟩1]+
if θ (t) ̸= 0 return x(t) ← θ (t) / ∥θ (t) ∥1
else
return x(t) ← any point in ∆d

3
4
5
6
7

function ObserveUtility(u(t) )z (t) ← [z (t−1) + u(t) − ⟨u(t) , x(t) ⟩1]+

and 3.3 when the prediction vector m(t) is set to 0.
Theorem 4.5 (Farina, Kroer, and Sandholm, 2021b). PRM and PRM+ are no-external-regret
algorithms for the domain ∆d . For all x̂ ∈ ∆d and all times T satisfy the weakly-predictive
regret boundTX
2
1
R (x̂) ⩽ + ηu(t) − m(t) − ⟨u(t) − m(t) , x(t) ⟩1
η
2
t=1
T

⩽

4.3.2

T
X
1
2
+ 2η(d + 1)
u(t) − m(t) 2 .
η
t=1

Experimental evaluation

We empirically investigate the instantiation of the predictive CFR algorithm described in Section 4.3
together with the weakly-predictive no-external-regret algorithm for probability simplexes PRM+
described in Section 4.3.1.
In the experiments, we apply two heuristics that usually lead to better practical performance: we average
the sequence-form strategies x(1) , . . . , x(T ) using the formula x̄(t) :=


(t−1)
6t
6t
1 − (t+1)(t+2) x̄
+ (t+1)(t+2)x(t) , and we use the alternating updates scheme. We call thisalgorithm Predictive CFR + (PCFR+ ). We compare PCFR+ to the prior state-of-the-art CFR variants:
CFR+ (Tammelin, 2014), Discounted CFR (DCFR) with its recommended parameters (Brown andSandholm, 2019), and Linear CFR (LCFR) (Brown and Sandholm, 2019).
We conduct the experiments on common parameteric benchmark games. Each game isidentified with an alphabetical mnemonic that uniquely denotes the game and parameters. A fulldescription of the game instances, as well as their dimensions, is available in Appendix A. Results
are shown in Figure 4.5.
The x-axis shows the number of iterations of each algorithm. Every algorithm pays almost
exactly the same cost per iteration, since the predictions require only one additional thresholding§4. COMPOSABILITY OF LEARNING DYNAMICS AND PREDICTIVE COUNTERFACTUAL REGRETMINIMIZATION
69

Avg. pred. err.

Nash gap

G25 — Goofspiel

GL25 — Goof. lim.info.

100

100

10−2

10−2

10−4

10−3
10−7
10−11

0

1000

2000 10−301000

Nash gap
Avg. pred. err.

0

1000

2000

0

1000

2000

10−11
10−16

−5

10
1000

2000

0

1000

B2324 — Battleship

10−5
10−3

2000

REL27 — River

10−2

P26 — Pursuit-evasion

103

100

101

10−4
10−8

10−1
0

1000

2000 10−1

−8

10

0

1000

2000 10−4

10−13

101

2000

0

L22d2 — Leduc poker

10−1
10−3
10−2
0

1000

1000

2000

10−3

10−2

10−7

10−6

10−11

10−10
0

1000

2000 10−3

10−8

10−8

10−6

10−13

10−13

1000

2000

Legend:

0

1000

2000

SM2 — Hard matrix

10

0

2000

K23 — Kuhn poker

2000 10−3

−4

1000

10−14

10−3
1000

0

10−9

10−2

0

Nash gap

2000 10−6

10−4

10−6

0

Avg. pred. err.

D26 — Liar’s dice

0

1000
+

PCFR

2000
+

CFR

LCFR

0

1000

2000

0

1000

2000

DCFR

Figure 4.5: Performance of PCFR+ , CFR+ , DCFR, and LCFR on nine games. In all plots, the x axis is thenumber of iterations of each algorithm. For each game, the top plot shows that the Nash gap on
the y axis (on a log scale), the bottom plot shows and the average prediction error (on a log scale).

70§4.3. PREDICTIVE COUNTERFACTUAL REGRET MINIMIZATION PARADIGM

step in PCFR+ . For each game, the top plot shows on the y-axis the Nash gap, while the bottom
plot shows the accuracy in our predictions of the regret vector, measured as the average ℓ2
norm of the difference between the actual loss u(t) received and its prediction m(t) across all
no-regret-algorithms at all decision points in the game. For all non-predictive algorithms (CFR+ ,LCFR, and DCFR), we let m(t) = 0. For our predictive algorithm, we set m(t) = u(t−1) at all timest ⩾ 2 and m(1) = 0, in accordance with the canonical optimistic learning setup (Section 3.2.1).
Both y-axes are in log scale. On B2324 Battleship and P26 Pursuit-evasion, PCFR+ is fasterthan the other algorithms by 3-6 orders of magnitude already after 500 iterations, and around 10
orders of magnitude after 2000 iterations. On G25 Goofspiel, PCFR+ is also significantly fasterthan the other algorithms, by 0.5-1 order of magnitude. On L22d2 Leduc poker as well as the
REL27 River endgame, the predictions in PCFR+ do not seem to help as much as in other games.
On the River endgame, the performance is essentially the same as that of CFR+ . On L22d2 Leducpoker, it leads to a small speedup over CFR+ . On both of those games, DCFR is fastest. In contrast,
DCFR actually performs worse than CFR+ in the non-poker experiments, though it is sometimes
on par with CFR+ .
Finally, PRM+ converges very rapidly on the SM2 game, a 2-by-2 matrix nonsequential gamethat was identified as a hard instance for RM-based methods (Farina, Kroer, and Sandholm, 2019b).
By using optimism, we seem to be able to sidestep the issue.
We observe that, as expected, the convergence rate of PCFR+ is closely related to how good
the predictions m(t) of u(t) are. On B2324 Battleship and P26 Pursuit-evasion, the predictions
become extremely accurate very rapidly, and PCFR+ converges at an extremely fast rate. On
G25 Goofspiel, the predictions are fairly accurate (the error is of the order 10−5 ) and PCFR+ isstill significantly faster than the other algorithms. On the REL27 River endgame, the average
prediction error is of the order 10−3 , and PCFR+ performs on par with CFR+ , and slower than DCFR.
Overall, the experiments show that PCFR+ tend to be superior to CFR+ and DCFR on non-poker
games thanks to its ability to incorporate predictions, whereas on poker games DCFR is the fastest.

Chapter 5

Notions of distance forsequence-form strategies, and prox
methods
We have seen in Chapter 4 that an effective way of constructing no-external-regret dynamics
for sequence-form strategy polytopes is through regret circuits. While algorithms constructed
through regret circuits are currently the practical state of the art for finding equilibria in large
games, in this chapter we look into techniques that belong to the important literature of (online)
convex optimization, which enjoy strong theoretical properties that are not known to apply toalgorithms constructed through regret circuits. The investigation of these optimization techniques
will naturally expose us to a series of fundamental questions about the metric structure andproperties of sequence-form strategy polytopes. We touch on some of these questions below.
Unlike regret circuits, which guarantee feasibility of the strategies produced by exploiting the
known combinatorial structure of the strategy set, most convex optimization methods maintainfeasibility of the iterates by projecting onto the feasible sets. Typically, much flexibility is allowedin the notion of distance—called the distance-generating function (DGF)—that is used to define theprojection (or, more precisely, proximal) steps. This immediately leads to the following natural
question, which serves as the central theme of this chapter.
What are suitable notions of distance (distance-generating functions) for sequence-form strategies
in imperfect-information extensive-form games, leading to efficient proximal steps?
In principle, Euclidean distance can be used as a notion of distance between sequence-form
strategies. However, its efficacy is limited in practice by the fact that projecting points onto
the polytope of sequence-form strategies according to the Euclidean distance is prohibitively
expensive in practice (Gilpin, Peña, and Sandholm, 2012).

72§5.1. CONTRIBUTIONS AND RELATED WORK

5.1

Contributions and related work

In this chapter we provide several contributions in the direction of better understanding whichnotions of distance best serve sequence-form strategy polytopes. Most importantly, in Section 5.4.2
we provide the first DGF that enables, at the same time, (i) projections that can be computed inlinear-time in the number of sequences, and (ii) polynomial diameter of the set of the sequenceform strategies, a key quantity that affects the runtime of convex optimization algorithms.
This chapter is structured as follows. In Section 5.2.1 we recall the properties that DGFs needto satisfy for a generic convex feasible set, and introduce proximal setups. In Section 5.2.2 werecall four convex optimization algorithms (two of which define no-external-regret algorithms,
and two of which do not), elucidating their dependence on the choice of DGFs. In Section 5.3 weinvestigate projections onto sequence-form strategy polytopes with respect to the Euclidean norm
(and small variations of it, which will be important in Chapter 6). In particular, we will show thatexact projections can be computed inductively on the structure of the TFDP, albeit at a superlinear
cost in the number of sequences. Such a cost is prohibitive in large games. Motivated by the desireof overcoming the superlinear dependence, in Section 5.4, we move away from Euclidean distanceto focus on notions of distance that enable linear-time (in the number of sequences) projections
onto the sequence-form strategy polytope. Specifically, in Section 5.4.1 we introduce the class ofdilated DGFs, a class of DGFs specifically designed for sequence-form strategy polytopes originallyintroduced by Hoda, Gilpin, Peña, and Sandholm (2010). Dilated DGFs enable fast projections,but the diameter they induce currently exhibits an unfavorable, exponential dependence on thedimension of the sequence-form strategy polytope. Finally, in Section 5.4.2 we introduce a new
DGF, which we coin dilatable global entropy DGF. Dilatable global entropy retains the appealing
linear-time guarantees on projections, while overcoming—for the first time—the exponential
diameter in favor of only a polynomial one. It is currently the theoretical state-of-the-art notion of
distance for strategies in imperfect-information extensive-form games.
Prior work The framework of dilated DGFs was introduced by Hoda, Gilpin, Peña, andSandholm (2010). Hoda, Gilpin, Peña, and Sandholm (2010) also introduce a notion of a “nice”
DGF. Their definition is similar to the one used in this chapter, but only states that certain
operations should be “easily computable”. In contrast, we attach a concrete meaning to thatstatement: we take it to mean linear time in the dimension of the domain.
Kroer, Waugh, Kılınç-Karzan, and Sandholm (2020) and Kroer, Farina, and Sandholm (2018b)
note the drawback in the current analysis of dilated DGFs regarding the exponential dependenceof the DGF weights in the depth of the decision space. Farina, Kroer, and Sandholm (2019b)
extend the analysis of the strong-convexity modulus of φ with respect to the ℓ2 norm on Q, but
their weighting scheme is still exponential in the worst case.
The predictive versions of OMD and FTRL presented in this chapter as applications of proximal

73§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

setups can be traced back to the works by Chiang, Yang, C.-J. Lee, Mahdavi, C.-J. Lu, R. Jin, andZhu (2012), A. Rakhlin and Sridharan (2013), S. Rakhlin and Sridharan (2013), and Syrgkanis,
Agarwal, Luo, and Schapire (2015). The mirror prox algorithm was introduced by Nemirovski(2004). The excessive gap technique was introduced by Nesterov (2005a).
Finally, we remark that the question as to what DGF achieves desirable properties has been
explored in the literature for several classes of related decision sets as well, including i) the
simplex case, where both the negative entropy DGF and the Euclidean DGF are known to havegood properties (Held, Wolfe, and Crowder, 1974; Beck and Teboulle, 2003; Condat, 2016), andii) the case of positive semidefinite matrices with a trace constraint, where the matrix entropy
performs well (Aharon Ben-Tal and Nemirovski, 2005).

5.2

Preliminaries

In this section we recall key concepts and applications related to proximal setups.

5.2.1

Distance-generating functions and proximal setups

Let X be a convex and compact set. A distance-generating function for X is defined as follows.
Definition 5.1 (Distance-generating function). A distance-generating function (DGF) φ for acompact and convex set X ⊆ ℝd is a function φ : X → ℝ such that:• it is continuous on X and differentiable in the relative interior of X ;• it is strongly convex in the relative interior of X with respect to some norm ∥ · ∥, that is,there exists a constant µ > 0 such that
∇φ(x) − ∇φ(x′ ), x − x′ ⩾ µ∥x − x′ ∥2∀ x, x′ ∈ relint X .

For twice-differentiable φ, the strong convexity condition is automatically verified aslong asm, ∇2 φ(x)m ⩾ µ∥m∥2 , ∀x ∈ relint X , m ∈ ℝn .

(5.1)

Once a distance-generating function φ for X has been picked, several important tools can be
defined, which collectively form a proximal setup for X :• The Bregman divergence Dφ : X × relint X → ℝ⩾0 associated with φ yields a notion of distance
between points defined as[5.a][5.a] A Bregman divergence need not be symmetric and thus might not be a metric.

74§5.2. PRELIMINARIESDφ (x ∥ x′ ) := φ(x) − φ(x′ ) − ∇φ(x′ ), x − x′∀ x ∈ X , x′ ∈ relint X .

(5.2)• The φ-diameter of X is
Ωφ,X := max
Dφ (x ∥ x′ ) ⩽ max φ(x) − min φ(x)′x,x ∈X

x∈X

(5.3)

x∈X

• Finally, we denote the largest possible value of the ℓ1 norm on X with the symbol
MX := max∥x∥1 .
x∈XWhile not a part of the assumptions on the DGF φ, it is typically assumed that φ allows one toefficiently compute the following two quantities, which come up at every iteration of most convex
optimization algorithms:• the gradient ∇φ(x) of φ at any point x ∈ relint X ;• the gradient of the convex conjugate φ∗ of φ at any point g ∈ ℝd :∇φ∗ (g) = arg max ⟨g, x̂⟩ − φ(x̂) .
x̂∈X

The gradient of the convex conjugate can be intuitively thought of as a linear maximization
problem over X (i.e., the support function of X , which is a non-smooth convex optimization
problem), smoothed by the regularizer φ. For that reason, in this dissertation we shall refer to∇φ∗ (g) either symbolically, or occasionally as the smoothed support function.
Because the above two quantities arise so frequently in optimization methods, it is importantthat the chosen distance-generating function allow for efficient computation of them. In particular,
in this dissertation we are concerned with “nice” DGFs that enable linear-time (in the dimensionφ) exact computation of those two quantities.
Definition 5.2. A distance-generating function φ is said to be “nice” if φ(x), ∇φ(x) and∇φ∗ (g) can be computed exactly in linear time in the dimension of the domain of φ.

Finally, we mention a closely related operation that comes up often in optimization methods:
the proximal operator (or prox operator for short), defined asproxφ (g ∥ x̃) := arg min ⟨g, x⟩ + Dφ (x ∥ x̃) = ∇φ∗ −g + ∇φ(x̃) ∈ X

(5.4)

x∈X

for any x̃ ∈ X and g ∈ ℝd . In light of (5.4), the prox operator can be implemented efficiently
provided that ∇φ and ∇φ∗ can. So, prox operators can be computed exactly in linear time in the§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

75

dimension φ for “nice” DGFs.

5.2.2

Applications

Proximal setups are ubiquitous in convex optimization. In this section, we discuss four importantconvex optimization methods, all of which require a proximal setup.
The first two methods—online mirror descent (OMD) and follow-the-regularized-leader (FTRL)—
are the premier algorithms in online convex optimization. Given any convex and compact set
X , OMD and FTRL define predictive no-external-regret algorithms with rather strong properties.
By having access to good proximal setups (for example, enjoying linear-time projections) for
sequence-form strategy polytopes X = Q, we will be able to efficiently instantiate OMD and FTRL
in imperfect-information extensive-form games.
The other two methods—excessive gap technique and mirror prox—are the two premieraccelerated bilinear saddle-point algorithm. They are offline optimization methods designed
to solve problems of the form maxx∈X miny∈Y x⊤ Ay, which are frequent in games. The most
famous example is perhaps the computation of a Nash equilibrium in a two-player zero-sum
imperfect-information extensive-form game, in which case the sets X and Y are the sequence-form
polytopes of the players, and A is the payoff matrix of the game. This again underlines theimportance of researching proximal setups for sequence-form polytopes with strong properties,
which will be our focus in the next two sections.

5.2.3

Online mirror descent and follow-the-regularized-leader

Follow-the-regularized-leader (FTRL, Shalev-Shwartz and Singer, 2007) and online mirror descent(OMD) are the two best-known (and studied) algorithms in online convex optimization. While
their predictive variants are relatively new and can be traced back to the works by Chiang, Yang,
C.-J. Lee, Mahdavi, C.-J. Lu, R. Jin, and Zhu (2012), A. Rakhlin and Sridharan (2013), S. Rakhlin
and Sridharan (2013), and Syrgkanis, Agarwal, Luo, and Schapire (2015), in line with the rest of
the dissertation we use the name FTRL and OMD to refer to the predictive versions, with the usual
understanding that non-predictive variants of FTRL and OMD algorithms correspond to predictive
FTRL and predictive OMD when the prediction m(t) is set to the 0 vector at all t.
Algorithms 5.1 and 5.2 give pseudocode for FTRL and OMD. In both algorithm, η > 0 is anarbitrary step size parameter, X ⊆ ℝd is a convex and compact set, and φ : X → ℝ⩾0 is a
1-strongly convex distance-generating function with respect to some norm ∥ · ∥. We also recall
that the symbol Dφ (· ∥ ·) used in OMD denotes the Bregman divergence associated with φ, a standard
surrogate notion of distance in convex optimization which was defined in (5.2).
Predictive FTRL and predictive OMD satisfy the following regret bound.

76§5.2. PRELIMINARIES

Algorithm 5.1: Predictive FTRL
1
2

Algorithm 5.2: Predictive OMD

U (0) ← 0 ∈ ℝd1

function NextStrategy(m(t) ∈ ℝd )[▷ Set m(t) = 0 for non-predictive version]

x(t) ← arg min ⟨U (t−1) + m(t) , x̂⟩ +

3

x̂∈X

return x

4

2



1
φ(x̂)
η

(t)

5

function ObserveUtility(u6

U (t) ← U (t−1) + u(t)dfunction NextStrategy(m(t) ∈ ℝd )[▷ Set m(t) = 0 for non-predictive version]

3x(t) ← arg min ⟨m(t), x̂⟩ +
x̂∈X

4
(t)

z (0) ∈ X such that ∇φ(z (0) ) = 0

return x(t)(t)
function ObserveUtility(u∈ ℝd )

1
6
z (t) ← arg min ⟨u(t) , ẑ⟩ + Dφ ẑ
η
ẑ∈X

∈ℝ )




1
Dφ x̂ z (t−1)η

5z (t−1)Proposition 5.1 (Syrgkanis, Agarwal, Luo, and Schapire, 2015). Let Ωφ,X denote the diameterof φ over X , as defined in Equation (5.3). At all times T , the regret cumulated by predictive
FTRL (Algorithm 5.1) and predictive OMD (Algorithm 5.2) compared to any strategy x̂ ∈ X isbounded as
Reg(T ) ⩽TTX
1 X (t)Ωφ,X+η∥ℓt − m(t) ∥2∗ −∥x − x(t−1) ∥2 ,
η
cη
t=1
t=2

where c = 4 for FTRL and c = 8 for OMD, and where ∥ · ∥∗ denotes the dual of the norm ∥ · ∥with respect to which φ is 1-strongly convex.

We remark that the bound is RVU-predictive (see Section 3.2.4), a stronger guarantee comparedto CFR’s weak-predictivity (Chapter 4). In particular, predictive OMD and predictive FTRL guarantee
OT (1/T ) convergence to Nash equilibrium, and OT (T 1/4 ) per-player external regret when usedin self-play, in turn implying OT (T −3/4 ) convergence to a coarse-correlated equilibrium in generalimperfect-information extensive-form games.

5.2.4

Bilinear saddle points: Excessive gap technique and mirror prox

Another important class of convex optimization methods with applications to game solving is
methods for solving bilinear saddle-point problems (BSPPs), whose general form is
min max ⟨x, Ay⟩,

(5.5)

x∈X y∈Y

where X ⊆ ℝdx , Y ⊆ ℝdy are convex and compact sets, and A ∈ ℝdx ×dy . This captures,among other, the problem of computing a Nash equilibrium in a two-player zero-sum imperfectinformation extensive-form game.

77§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

We will now present the EGT and mirror prox algorithms for solving BSPPs. These algorithms
depend on two proximal setups: one for X and one for Y, denoted φx and φy , respectively. Let∥ · ∥x and ∥ · ∥y be the norms associated with the strong convexity of φx and φy in the given
proximal setup. The convergence rate then depends on the following operator norm of the payoff
matrix A:
∥A∥ := max ⟨x, Ay⟩ : ∥x∥x ⩽ 1, ∥y∥y ⩽ 1 .
The magnitude of ∥A∥ is the primary way in which the norm matters: if both φx and φy arepstrongly convex with respect to the ℓ2 norm, then ∥A∥ can be on the order of dx dy , whereas if
both are with respect to the ℓ1 norm, then ∥A∥ is simply equal to its largest entry.
Excessive gap technique (EGT) algorithm The excessive gap technique (EGT) is a first-order method
introduced by Nesterov (2005b), and one of the primary applications is to solve BSPPs such as
Equation (5.5). EGT assumes access to a proximal setup for X and Y, with one-strongly-convex
DGFs φx , φy , and constructs smoothed approximations of the optimization problems faced by
the max and min players. Based on this setup, we formally state the EGT of Nesterov (2005a) in
Algorithm 5.3. EGT alternatingly takes steps focused on decreasing one or the other smoothing
parameter. These steps are called ShrinkX and ShrinkY in Algorithm 5.3.
Algorithm 5.3: Excessive gap technique (EGT) algorithm
function Initialize()
t←0
(0)
(0)
3
µx ← ∥A∥, µy ← ∥A∥
4x̃ ← arg minx̂∈X φx (x̂)(0)
5y (0) ← ∇φ∗y (A⊤ x̃/µy )function ShrinkX()x̄ ← −∇φ∗x (−Ay t−1 /µt−1x )13x̂ ← (1 − τ )xt−1 + τ x̄14ȳ ← ∇φ∗y (A⊤ x̂/µt−1)yfunction ShrinkY()ȳ ← ∇φ∗y (A⊤ xt−1 /µt−1)y21ŷ ← (1 − τ )y t−1 + τ ȳ22x̄ ← −∇φ∗x (−Aŷ/µt−1
x )

1

11

19

2

12

20



6

x(0) ← proxφx

1
(0)

µx

Ay (0) x̃



15


16
17

function Iterate()
8t ← t + 1, τ ← 2/(t + 2)
9
if t is even then ShrinkX()
10else ShrinkY()
7

18x̃ ← proxφxτAȳ(1−τ )µt−1xt−1xt ← (1 − τ )x
+ τ x̃
y t ← (1 − τ )y t−1 + τ ȳ
µtx ← (1 − τ )µt−1
x

x̄





23
24
25
26

ỹ ← proxφy

−τ
A⊤ x̄
(1−τ )µt−1
y
t−1

ȳ



y t ← (1 − τ )y
+ τ ỹ
xt ← (1 − τ )xt−1 + τ x̄
µty ← (1 − τ )µt−1yAlgorithm 5.3 shows how initial points are selected and the alternating steps and stepsizes
are computed. Nesterov (2005a) proves that the EGT algorithm converges at a rate of OT (1/T ).
Theorem 5.1 (Nesterov, 2005a, Theorem 6.3). At every iteration t ⩾ 1 of the EGT algorithm,
the solution (x(t) , y (t) ) satisfies x(t) ∈ X , y (t) ∈ Y, andmax⟨x(t) , Aŷ⟩ − min ⟨x̂, Ay (t) ⟩ ⩽
ŷ∈Y

x̂∈X

p
4∥A∥ Ωφx ,X · Ωφy ,X
.
t+1

78

§5.2. PRELIMINARIES

Mirror prox (MPROX) algorithm The mirror prox (MPROX) algorithm was introduced by Nemirovski(2004). Rather than construct smoothed approximations, mirror prox directly uses the DGFs
to take first-order steps. Hence, the MPROX algorithm is best understood as an algorithm that
operates on the product space X × Y directly. As such, in most analyses of the MPROX algorithm,
a single 1-strongly convex DGF for the product space X × Y is required. To better align with the
setup used for EGT, we will define the DGF for the product space X × Y starting from proximal
setups for both X and Y, with 1-strongly convex DGFs φx , φy with respect to norms ∥ · ∥x and∥ · ∥y , respectively. With this setup, it is immediate to see that the function
φ : X × Y ∋ (x, y) 7→ φx (x) + φy (y)is a DGF for the productqspace X × Y, which is strongly convex with modulus one with respect
to the norm ∥(x, y)∥ := ∥x∥2x + ∥y∥2y . Furthermore, each proximal step taken with respect to dcan be expressed as two independent proximal steps with respect to φx and φy :
proxφ

gx
gy

!

x̃
ỹ

!!

(*

!
!+
gx
x
,
+ Dφ
gy
y

!!)
x̃
= arg minỹ(x,y)∈X ×Y!
arg minx∈X {⟨g x , x⟩ + φx (x) − ⟨∇φx (x̃), x⟩}=arg miny∈Y ⟨g y , y⟩ + φy (y) − ⟨∇φy (ỹ), y⟩!
proxφx (g x ∥ x̃) .
=proxφy g y ỹxy!

Similarly, the d-diameter of the product space X × Y is equal to the sum of diameters of X and Y
in their respective proximal setups. Finally, we note that the function
!
Ay
,
−A⊤ x

F : X × Y ∋ (x, y) 7→

critical in the analysis of MPROX (Ahron Ben-Tal and Nemirovski, 2001), satisfies
x
F
y

!

x′
−F
y′

!
=

q

∥A(y − y ′ )∥2x∗ + ∥A⊤ (x − x′ )∥2y∗

∗

s
⩽
⩽

2

max ⟨x̃, A(y − y ′ )⟩∥x̃∥x ⩽1q

+max ⟨x − x′ , Aỹ⟩∥ỹ∥y ⩽1

∥A∥2 · ∥y − y ′ ∥2y + ∥A∥2 · ∥x − x′ ∥2x = ∥A∥ ·that is, it is ∥A∥-Lipschitz with respect to the norm ∥ · ∥ on X × Y.

2
x − x′y − y′!
,

§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

79

Algorithm 5.4 shows the sequence of steps taken in every iteration of the MPROX algorithm.
Algorithm 5.4: Mirror Prox (MPROX) algorithm
function Initialize()
t←0
(0)
3
z x ← arg minx̂∈X φx (x̂)(0)
4z y ← arg minŷ∈Y φy (ŷ)
12function Iterate()
t←t+1

z tx
7
wtx ← proxφx η t Az t−1ytt ⊤ t−1zt
8wy ← proxφy −η A z x
y
t
t+1
t
t
9
z x ← proxφx η Awy z x

10
z t+1
← proxφy −η t A⊤ wtx z tyy5Note: {η t } is a sequence of stepsize parameters. A well-known
and theoretically-sound choice1for η t is η t := ∥A∥
for all t =0, 1, . . . (see also Theorem 5.2).

6

11
12

Pt

Pt

x(t) ← [ τ =1 η τ ]−1 τ =1 η τ wτxPt
Pty (t) ← [ τ =1 η τ ]−1 τ =1 η τ wτyCompared to EGT, mirror prox has a somewhat simpler structure: it simply takes repeatedextrapolated proximal steps. First, a proximal step in the descent direction is taken for both x
and y. Then, the gradient at those new points is used to take a proximal step starting from theprevious iterate (this is the extrapolation part: a step is taken starting from the previous iterate,
but with the extrapolated gradient). Finally, the average strategy is output.
As we recall in the next theorem, like EGT the MPROX algorithm converges at rate OT (1/T ).

Theorem 5.2 (Ahron Ben-Tal and Nemirovski, 2001, Theorem 5.5.1). Suppose the stepsize in
Algorithm 5.4 is set as ηt = 1/∥A∥. Then we havemax⟨x(t) , Aŷ⟩ − min ⟨x̂, Ay (t) ⟩ ⩽
ŷ∈Y

5.3

x̂∈X

∥A∥(Ωφx ,X + Ωφy ,Y ).
2t

Euclidean distance-generating function

We begin our investigation of distance-generating functions for sequence-form strategy spaces
starting from the prototypical notion: the Euclidean distance-generating function. Fix any
tree-form decision process, and let Q be the corresponding sequence-form strategy polytope. The
Euclidean distance-generating function is the function
φ(x) :=

1
1X∥x∥22 =
x[σ]2 ,
22x ∈ Q.

σ∈ΣIn this case, using the observation that ∇φ(x′ ) = x′ for all x′ ∈ Q, it is easy to see that the Bregman
divergence associated with φ reduces to (half of) the squared Euclidean distance between two
points,80§5.3. EUCLIDEAN DISTANCE-GENERATING FUNCTION

Dφ (x ∥ x′ ) :=1
11
∥x∥22 − ∥x′ ∥22 − ⟨x′ , x − x′ ⟩ = ∥x − x′ ∥22 .
2
2
2

Similarly, the gradient of the convex conjugate of φ in a generic point g ∈ ℝΣ reduces to the usual
Euclidean projection,1∇φ∗ (g) = arg max ⟨g, x̂⟩ − φ(x̂) = arg min −⟨g, x̂⟩ + ∥x̂∥22
2
x̂∈Q
x̂∈Q
1
= arg min ∥g − x̂∥22 .
2x̂∈Q
The computation of the projection arg minx̂∈Q ∥g − x̂∥22 is not known to be possible in linear
time in the number of sequences |Σ|. This means the Euclidean DGF is not known to be “nice”
in the sense Definition 5.2, and in many cases it might provide a less appealing proximal setup
than, for example, the dilatable global entropy DGF which will be introduced later in the chapter.
However, there are good reasons to study the properties of the Euclidean DGF. For one, out of
the DGFs that have been proposed for imperfect-information extensive-form games it is the only
one with Lipschitz-continuous gradients, which can be fundamental in certain contexts (see,e.g., Anagnostides, Panageas, Farina, and Sandholm (2022)). Furthermore, some of the techniques
we will develop when studying algorithms for Euclidean projection will prove important in otherchapters (specifically, Chapter 6).

5.3.1

Exact Euclidean projection algorithm

In this section we give a combinatorial algorithm for computing exactly the Euclidean projectionof a generic point g ∈ ℝΣ . Our algorithm formalizes and extends an idea by Gilpin (2009). In fact,
we give an algorithm that can work with any positive definite diagonal inner product norm, thatis a DGF of the form
φw (x) := x, diag(w)x =2
1 X x[σ]
,
2
w[σ]
σ∈Σ

for some given positive vector w ∈ ℝΣ
>0 . It is immediate to see that the Euclidean DGF corresponds
to the special case φ1 in which w = 1 ∈ ℝΣ
>0 is the vector of all ones. In Chapter 6 we will benefit
from considering projections with respect to DGFs φw in which w ̸= 1.
The gradient of the convex conjugate of φw corresponds to the optimization problem(∇φ∗w (g) = arg min
x∈Q


2 )
1 X x[σ]−⟨g, x⟩ +2w[σ](g ∈ ℝΣ ).

(5.6)

σ∈Σ

We will show how one can solve the projection problem (5.6) inductively, in a bottom-up fashion,§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

81

over the structure of the tree-form decision process. To do so, at every decision node j ∈ J we
define the value functionV≽j (t) : ℝ⩾0 → ℝ,

V≽j (t) :=

min




x̂ ∈ t ·Q≽j 

−

X
σ∈Σ≽j



2
1 X x̂[σ] g[σ] x̂[σ] +.
2w[σ] 

(5.7)

σ∈Σ≽j

We will be particularly interested in the derivatives of V≽j (t), which we will denote as[5.b]λ≽j (t) :=dV≽j (t).
dt

As we will show, the function λ≽j (t) is extremely structured, in that it is a strictly monotonicallyincreasing piecewise-linear (SMPL) function, as defined next.
Definition 5.3 (SMPL and quasi-SMPL function). Given an interval I ⊆ ℝ and a function
f : I → ℝ, we say that f is SMPL if it is strictly monotonically increasing and piecewise-linear
on I.
A quasi-SMPL function is a function f : ℝ → [0, +∞) of the form f (x) = [g(x)]+ whereg(x) : ℝ → ℝ is SMPL and [ · ]+ := max{0, · }.

The piecewise-linear nature of λ≽j (t) lends itself nicely to combinatorial algorithms, in thatthe function can be represented in memory implicitly through the set of breakpoints at which theslope of the function changes. We call this a standard representation for the function, as introduced
next.
Definition 5.4. Given a SMPL or quasi-SMPL function f , a standard representation for it is anexpression of the formf (x) = ᾱ + α0 x +SXαs [x − βs ]+ ,s=1valid for all x in the domain of f , where S ∈ ℕ⩾0 and β1 < · · · < βS . The size of the standard
representation is defined as the natural number S.

We review properties and manipulations of SMPL functions in an appendix to this chapter,
Section 5.A. With the above definitions we are ready to state the following result, which is central
in our analysis.
[5.b] For t = 0 we define λ≽j (t) in the usual way as λ≽j (0) = limt→0+V≽j (t)−V≽j (0)V(t)= limt→0+ ≽jt .
t

82§5.3. EUCLIDEAN DISTANCE-GENERATING FUNCTION

Lemma 5.1. For any decision node j ∈ J , the function λ≽j (t) is SMPL and has a standard
representation of size |Q≽j |, which can computed in polynomial time in |Σ≽j |.

Proof. The result is known when j is a terminal decision node, i.e., Q≽j is a simplex. Fix aparticular j ∈ J , and assume by induction that the result holds for all j ′ ≻ j.
By definition, the value function decomposes recursively asX  x• [a] 2X1−V≽j (t) = ming[ja] x• [a] +2
w[ja]x• ∈t∆Aj 
a∈Aj
a∈Aj


 X
X X
X  xj ′ [σ] 2 +min−g[σ]xj ′ [σ] +w[σ] 
x ′ ∈x• [a]Q≽j ′ a∈Aj j ′ ∈Cja jσ∈Σ≽j ′σ∈Σ≽j ′


2

 X

X X
1 X x• [a]
−
+
= min
g[ja] x• [a] +V≽j ′ (x• [a]) .
(5.8)

2
w[ja]
x• ∈t∆Aj ′a∈Aja∈Aj ′ j ∈Cjaa∈Aj

Consider the KKT conditions for x• in Equation (5.8):−g[ja] +Xx• [a]+
λ≽j ′ (x• [a]) = λ• + µ[a]
2
w[ja]
′

∀a ∈ Aj(Stationarity)

j ∈Cjax• ∈ t · ∆Aj(Primal feasibility)

λ• ∈ ℝ, µ ∈ ℝd⩾0

(Dual feasibility)∀a ∈ Ajµ[a] x• [a] = 0(Compl. slackness)

Solving for x• [a] in the stationarity condition, and using the conditions x• [a] µ[a] = 0 andµ[a] ⩾ 0, it follows that for all a ∈ Aj+x• [a] = w[ja]2 λ• + g[ja] −Xj ′ ∈Cλ≽j ′ (x• [a]) .

(5.9)

ja

Strict monotonicity and piecewise-linearity of x• [a] as a function of λ• . Given the propertiesof SMPL functions, it is immediate to see that x• [a] is unique as a function of λ• . Indeed, notethat (5.9) can be rewritten as§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS



83

+



x• [a] = w[ja]2 λ• − w[ja]2 −g[ja] +X

λ≽j ′ (x• [a]) ,j ′ ∈Cja

which is a fixed-point problem of the form studied in Lemma 5.8 in the appendix for
y = w[ja]2 λ• and function fa defined asfa (x• [a]) = w[ja]2 −g[ja] +X

λ≽j ′ (x• [a]),j ′ ∈Cja

which is clearly SMPL by inductive hypothesis. Hence, the unique solution to the previous
fixed-point equation is given by the quasi-SMPL function
ga : λ• 7→

i+
1 h
−1
(x
[a]
+
f
)
(λ
)
,
•
a
•
w[ja]2a standard representation of which can be computed in time O(|Q≽j |) by combining the resultsof Lemmas 5.3, 5.5 and 5.6 given that a standard representation of each λ≽j ′ (t) (j ′ ∈ Cja ) ofsize |Σ≽j ′ | is available by inductive hypothesis.
Strict monotonicity and piecewise-linearity of λ• as a function of t. At this stage, we knowthat given any value of the dual variable λ• , the unique value of the coordinate x• [a] thatsolves the KKT system can be computed using the quasi-SMPL function ga . In turn, this meansthat we can remove the primal variables x• from the KKT system, leaving us a system in λ•and t only. We now show that the solution λ⋆• of that system is a SMPL function of t ∈ [0, +∞).
Indeed, the value λ⋆• (t) that solves the KKT system has to satisfy the primal feasibility
condition
t=

X
a∈Aj

x• [a] =Xga (λ• ).

a∈Aj

Fix any t > 0. The right-hand side of the equation is a sum of quasi-SMPL functions. Hence,from Lemma 5.4, we have that the right-hand side has a standard representation of size at mostP
P
|Aj | + a∈Aj j ′ ∈Cja |Σ≽j ′ | = |Σ≽j | can be computed in time O(|Σ≽j | log |Aj |). Furthermore,from Lemma 5.7, we have that the λ⋆• that satisfies the equation is unique, and in fact that the
mapping (0, +∞) ∋ t 7→ λ⋆• (t) is SMPL with standard representation of size at most |Σ≽j |.
Relating λ• and λ≽j (t). Since λ⋆• (t) is the coefficient on t in the Lagrangian relaxation of (5.8),
it is a subgradient of V≽j (t), and since there is a unique solution, we get that it is the derivative,

84§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS

that is, λ⋆• (t) = λ≽j (t) for all t ∈ (0, +∞). To conclude the proof by induction, we then needto analyze the case t = 0, which has so far been excluded. When t = 0, the feasible set
tQ≽j is the singleton {0}, and V≽j (0) = 0. Since V≽j (t) is continuous on [0, +∞), and sincelimt→0+ λ≽j (t) = limt→0+ λ⋆• (t) exists since λ⋆• (t) is piecewise-linear, then by the mean value
theorem,λ≽j (0) = lim+ λ⋆• (t),t→0that is, the continuous extension of λ⋆• must be (right) derivative of V≽j (t) in 0. As extending
continuously λ⋆• (t) clearly does not alter its being SMPL nor its standard representation, we
conclude the proof of the inductive case.

5.4

Distance-generating functions with linear-time projections

Most convex optimization methods maintain feasibility by projecting onto the feasible set at each
iteration. Hence, designing notions of distance that enable efficient projections is an important
step in scaling those methods.
In this section, we focus on notions of distance that enable projections in linear time in thedimension of the dimension of the sequence-form strategy polytope. Remarkably, in Section 5.4.2
we introduce the first such notion of distance whose linear-time projection property does notcome at the cost of an exponential diameter.

5.4.1

Dilated distance-generating functions

Dilated distance-generating functions are a general framework for constructing “nice” DGFs
(in the sense of Definition 5.2) for (the relative interior of the) sequence-form polytopes (Hoda,
Gilpin, Peña, and Sandholm, 2010). Specifically, a dilated DGF for a sequence-form polytope
is constructed by taking a weighted sum over suitable local regularizers φ∅ : ℝ⩾0 → ℝ⩾0 andφj : ∆Aj → ℝ⩾0 for all decision nodes j ∈ J , and is of the form
φ : Q ∋ x 7→ α∅ φ∅ (x[∅]) +

X



αj φ□
j x[pj ], (x[ja])a∈Aj ,(5.10)

j∈J

where
A

j
φ□
j : ℝ⩾0 × ℝ⩾0 → ℝ,

φ□j (p, z) :=0p · φ j

if p = 0(5.11) zpotherwise.

Each local function φj : ∆Aj → ℝ is assumed to be continuously differentiable and strongly§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

85

convex modulus one on the relative interior of the probability simplex ∆Aj . By dividing(x[ja])a∈Aj by x[pj ] in (5.11), we renormalize (x[ja])a∈Aj to the simplex, measure the DGF there,
and then scale that value back by x[pj ]. Finally, the weight αj is a flexible weight term that can bechosen to ensure good properties. Hoda, Gilpin, Peña, and Sandholm (2010) showed that if each
local DGF φj is strongly convex, then the dilated DGF φ is also strongly convex (although they
do not give an explicit modulus), and they show that the associated smoothed support function
can easily be computed, provided that the smoothed support function for each φj can easily be
computed. For dilated DGFs, the strongest general result on the strong-convexity modulus comesfrom Farina, Kroer, and Sandholm (2019b), where the authors show that if each local DGF φj isstrongly convex modulus one with respect to the ℓ2 norm, and the weights of the decision nodes
Pare set recursively according to αj = 2 + 2 maxa∈Aj j ′ ∈Cja αj ′ (so, in particular decision nodes
without descendants have a weight of 2), then φ is strongly convex modulus one with respect tothe ℓ2 norm on Q.
The local DGFs must be chosen so that they are compatible with the relative interior of thesimplex. For a given simplex ∆k , these are usually chosen either as:
Pk• the (negative) entropy DGF, log k + i=1 y[i] log y[i], where we let y[i] log y[i] = 0 whenevery[i] = 0; or
Pk• the Euclidean DGF, 12 i=1 (y[i] − 1/k)2 . These are both 1-strongly convex on relint ∆k (forentropy with respect to the ℓ1 norm and for Euclidean with respect to the ℓ2 norm), andtheir associated smoothed support functions can be computed in O(k) time (see, e.g., Ahron
Ben-Tal and Nemirovski (2001) and Condat (2016)).

5.4.1.1“Nice”ness of dilated distance-generating functions

One of the most important properties of dilated DGFs is that they lead to a “nice” DGF as long as
each local convex conjugate gradient ∇φ∗j can be computed in time linear in |Aj |.[5.c] Specifically,
the gradient of a dilated DGF and of its convex conjugate can be computed exactly in closed formby combining the gradients of each φj and their convex conjugates, as shown in Algorithm 5.5.
5.4.1.2

Dilated entropy distance-generating function

As mentioned, the dilated entropy DGF is the instantiation of the general dilated DGF frameworkof Section 5.4.1 with the particular choice of using the (negative) entropy function at each decisionnode. In particular, for any choice of weights α∅ , αj > 0 it is the regularizer of the form[5.d][5.c] In particular, this makes the dilated entropy and dilated Euclidean DGFs “nice” DGFs.
[5.d] In this dissertation, we let 0 log(0) = 0 log(0/0) = 0. Since the dilated entropy DGF is a Legendre function, it isguaranteed that all iterates and prox-steps will remain in the relative interior of the optimization domain at all times, thus
avoiding the non-differentiability issue of the entropy function at the boundary of Q.

86§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS

Algorithm 5.5: Gradient and smoothed support function for general dilated DGFs.
function ConjugateGradient(g ∈ ℝΣ )z ← 0 ∈ ℝΣ
12z[∅] ← 1
13
for j ∈ J in bottom-up order do 14(z[ja])a∈Aj ← ∇φ∗j (g[ja])a∈Ajfunction Gradient(x ∈ relint Q)
g ← 0 ∈ ℝΣ3
for j ∈ J in bottom-up order do

1

10

2

4
5
6

b ∈ ∆Aj ←x[ja]x[pj ]

11a∈Aj

for a ∈ Aj dog[ja] ← g[ja] + αj ∇φj (b)

15



7
8
9

g[pj ] ← g[pj ] + αj φj (b) − ⟨∇φj (b), b⟩

16g[∅] ← g[∅] + α∅ ∇φ∅ (x[∅])return g

17
18
19

for j ∈ J in top-down order dofor a ∈ Aj doz[ja] ← z[pj ] · z[ja]return z ∈ QQ ∋ x 7→ α∅ x[∅] log x[∅] +g[ja]z[ja]

a∈Aj



X

X

g[pj ] ← g[pj ] − φj ((z[ja])a∈Aj ) +X

αj x[pj ] log |Aj | +

j∈J

a∈Aj


x[ja] .
x[ja] logx[pj ](5.12)

We will now briefly review existing results specific to the dilated entropy DGF, for whichstronger results are known than for the general class of dilated DGFs. A central result in thepresent paper is to show that there exist DGFs for sequence-form polytopes which are better thanthe dilated entropy DGF, but that these DGFs can be partially recast in a dilated form, in order toenable efficient computation of the smoothed support function.
First, as a direct consequence of the more general discussion in Section 5.4.1 and Algorithm 5.5,
the dilated entropy DGF is a “nice” DGF (in the precise sense of Definition 5.2) no matterthe choice of weights α. In particular, in the case of the negative entropy functions φj (x) :=P
log |Aj | + a∈Aj x[ja] log x[ja], one has∇φj (x) [a] = 1 + log x[a],ega∇φ∗j (g) [a] = P

a′ ∈Aj e

ga′

for all a ∈ Aj , x ∈ relint ∆Aj , and g ∈ ℝAj . By plugging the above expression in the template ofAlgorithm 5.5 we obtain linear-time exact algorithms to compute ∇φ and ∇φ∗ .
Kroer, Waugh, Kılınç-Karzan, and Sandholm (2020) show that the dilated entropy DGF isstrongly convex modulus 1/MQ with respect to the ℓ1 norm, when the weights α are chosen as in
the following definition.
Definition 5.5 (Kroer et al. dilated entropy DGF). Define the DGF weights βj recursively asβ∅ := 2 + 2

X
j∈C∅

βj ,

βj := 2 + 2 maxa∈AjXj ′ ∈Cja

βj ′∀j ∈ J .

87§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

The resulting instantiation of the dilated entropy DGF is the Kroer et al. dilated entropy DGFκ : x 7→ β∅ x[∅] log x[∅] +Xβj x[pj ] log |Aj | +j∈JXx[ja] loga∈Ajx[ja] .
x[pj ]

Example 5.1. Consider the small tree-form decision process introduced in Example 2.7,
reproduced below. In this case, the weights β in the construction of the Kroer et al.’s dilated
entropy DGF φ are computed as follows: βD = βC = βB = 2; βA = 2+2 max{βB +βC , βD } = 10;β∅ = 2 + 2βA = 22. Correspondingly, the DGF φ is the function
κ : x 7→ 22 x[∅] log x[∅]A

+ 10 x[∅] log(2) + x[A1] log+ 2 x[A1] log(2) + x[B3] log

x[A1]
x[A2]
+ x[A2] log
x[∅]
x[∅]
x[B3]
x[B4]
+ x[B4] log
x[A1]
x[A1]

x[C5]
x[C6]
+ 2 x[A1] log(2) + x[C5] log+ x[C6] log
x[A1]
x[A1]

!
1

2
D

!

789
B

C

3 4

5 6

!

x[D8]
x[D9]
x[D7]
+ x[D8] log+ x[D9] log+ 2 x[A2] log(2) + x[D7] log
x[A2]
x[A2]
x[A2]

!
.

We remark that, on the surface, the strong convexity modulus of M1Q with respect to the ℓ1
norm might appear less appealing than the modulus 1 obtained by using the ℓ2 norm. However,recall that the norm that is used to measure strong convexity affects the value of the operator
norm of A, which is significantly smaller under the ℓ1 − ℓ∞ operator norm (where it is equal tomaxr,c |A[r, c]|) than the ℓ2 − ℓ2 operator norm for strong convexity with respect to the ℓ2 norm.
Unfortunately, due to the exponential nature of the weights defined in Definition 5.5, theonly known bound on the diameter of Q under the Kroer et al. dilated entropy DGF is
exponential.
Proposition 5.2 (Kroer, Waugh, Kılınç-Karzan, and Sandholm, 2020). The diameter of Q
under the Kroer et al. dilated entropy DGF κ satisfies
2Ωκ,Q ⩽ 2DQ +2 MQ
max
log |Aj ′ |.
′j ∈JIn fact, this drawback is actually shared by all known analyses of dilated DGFs, not just dilated
entropy (Farina, Kroer, and Sandholm, 2019b).

88

5.4.2

§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS

Dilatable global entropy distance-generating function

As mentioned at the end of the previous subsection, one drawback of both the general andentropy-specific dilated DGFs developed in the past is that they have an exponential dependence
on the depth of the sequence-form polytope. In particular, note that the factor of 2 in the recursive
definition of the weights means that the factor βj for some root decision node is growing atleast on the order of 2DQ , where DQ is the depth of the tree-form decision process. For somesequence-form polytopes this might be acceptable: if the tree-form decision process is reasonably
balanced, then the number of decision nodes is also exponential in depth. However, for othersequence-form polytopes this is unacceptable: the most extreme case would be a single line ofdecision nodes, where the number of decision nodes is linear in DQ , but the βj at the root is
exponentially large. This exponential dependence on depth also enters the convergence rate of
the optimization methods, since it effectively acts as a scalar on the polytope diameter Ω induced
by DQ . Motivated by the need to soundly resolve that drawback, in this subsection we introduce
the first “nice” DGF (in the sense of Definition 5.2) with guaranteed polynomially-small diameter
for any decision node.
Definition 5.6 (Dilatable global entropy). The dilatable global entropy distance generating function
ψ is the function ψ : Q → ℝ⩾0 defined asψ : Q ∋ x 7→ w∅ x[∅] log(x[∅]) +X X

wja x[ja] log x[ja] +j∈J a∈AjX

γj x[pj ] log |Aj |,j∈J

where each γj ⩾ 1 (j ∈ J ) is defined recursively asγ∅ = 1 +Xγj ,γj := 1 + max Xa∈Aj 
j ′ ∈Cja

j∈C∅

γj ′




∀ j ∈ J,

(5.13)

and each wσ ⩾ 1 (σ ∈ Σ) is defined recursively asw∅ := γ∅ −

X

γj ,

j∈C∅

wja := γj −Xj ′ ∈Cja

γj ′ = 1 + max
′


 X

a ∈Aj j ′ ∈Cja′

γj ′





−

X

γj ′

∀ja ∈ Σ.

j ′ ∈Cja

The weights γj defined in (5.13) are very similar to the ones given for the dilated DGFs in theprevious section, except that the whole expression is smaller by a factor of two. Avoiding thisfactor of two is crucial, because it allows us to avoid the exponential dependence on depth. Here,89§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

it is easy to see that γj is upper bounded by the number of decision nodes in the subtree rooted atj, so γj is at most polynomial in the size of the sequential decision problem. In fact, it is not hard
to show that if j is the sole root decision node, then γj is equal to maxx∈Q ∥x∥1 .
Example 5.2. We continue the example started in Example 5.1, this time looking at the dilatableglobal entropy function. In this case, the weights γ in the construction of the dilatable global
entropy DGF ψ are computed as follows: γD = γC = γB = 1; γA = 1 + max{γB + γC , γD } = 3;γ∅ = 1 + γA = 4. Correspondingly, all weights wja are equal to 1 except for wA2 = γA − γD = 2.
Correspondingly, the DGF φ is the functionAψ(x) = x[∅] log x[∅]1

+ x[A1] log x[A1] + 2x[A2] log x[A2]2D+ x[B3] log x[B3] + x[B4] log x[B4]789+ x[C5] log x[C5] + x[C6] log x[C6]B

C+ x[D7] log x[D7] + x[D8] log x[D8] + x[D9] log x[D9]

3 4

5 6

+ 3x[∅] log(2) + x[A1] log(2) + x[A1] log(2)Small TFDP considered in this+ x[A2] log(2).

example.

5.4.2.1

Dilatability property

The adjective dilatable comes from the key property that the dilatable global entropy is equal
to a specific dilated entropy regularizer ψ̃, on the sequence-from strategy space Q. More precisely,
consider the dilated entropy DGF defined asψ̃ : Q ∋ x 7→ γ∅ x[∅] log x[∅] +XX

γj x[pj ] log |Aj | +

j∈J

a∈Aj


x[ja] x[ja] log.
x[pj ]

Then, we have the following.
Theorem 5.3. The dilatable global entropy DGF and the dilated entropy DGF coincide on
the polytope of sequence-form strategies Q, that is, ψ(x) = ψ̃(x) for all x ∈ Q.

Proof. We start by expanding the definition of ψ̃(x) for x ∈ Q:
ψ̃(x) := γ∅ x[∅] log x[∅] +X X
j∈J a∈Aj

γj x[ja] log

x[ja] X+γj x[pj ] log |Aj |x[pj ]
j∈J

90§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS

=

X X

X X

γj x[ja] log x[ja] −γj x[ja] log x[pj ] +γj x[pj ] log |Aj |,(5.14)j∈J

j∈J a∈Ajj∈J a∈Aj

X

where in the second equality we have used the fact that x[∅] = 1, as well as the properties ofPlogarithms. Given the assumption x ∈ Q, it holds that a∈Aj x[ja] = x[pj ] for all j ∈ J andso we can simplify the middle summation in (5.14) and obtain
ψ̃(x) =X X

γj x[ja] log x[ja] −j∈J a∈AjX

γj x[pj ] log x[pj ] +j∈JXγj x[pj ] log |Aj |.

j∈J

The middle summation multiplies the weights γj by a quantity that depends on the parent
sequence pj of j. It can therefore be rewritten equivalently by summing over nonterminal
sequences and multiplying by the weight of the children decision nodes, as follows.
ψ̃(x) =X X

γj x[ja] log x[ja] −X X X

j∈J a∈Ajj∈J a∈Aj j ′ ∈Cja

X X

X X

γj ′ x[ja] log x[ja] +γj x[ja] log x[ja] −j∈J a∈Aj

=

X X



wja x[ja] log x[ja] +j∈J a∈AjXj∈J a∈AjX

γj x[pj ] log |Aj |

j∈J


=

X

γj ′x[ja] log x[ja] +j ′ ∈CjaX

γj x[pj ] log |Aj |j∈Jγj x[pj ] log |Aj |j∈J= ψ(x),
as we wanted to show.
The equality established in Theorem 5.3 does not hold outside the sequence-form polytope—a
set with no interior. Because of that, it is not surprising that in general ∇ψ(x) ̸= ∇ψ̃(x) even atpoints x in the relative interior of the sequence-form polytope. We will return to the difference
between the gradients of ψ and ψ̃ in Section 5.4.2.5, where we show that the mismatch between∇ψ(x) and ∇ψ̃(x) is orthogonal to the sequence-form polytope when x ∈ Q.
5.4.2.2“Nice”ness of the dilatable global entropy DGF

We now show that our dilatable global entropy regularizer is “nice” in the sense of Definition 5.2,
that is, its gradient and the gradient of its convex conjugate can be computed exactly in linear
time in |Σ|.
• The gradient of ψ can be trivially computed in closed form and linear time in |Σ| startingfrom Definition 5.6 asX∇ψ(x) [σ] = (1 + log x[σ])wσ +γj log |Aj |j∈Cσ∀σ ∈ Σ, x ∈ relint Q.

§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS91• Using the dilatability property, we have that the gradient of the convex conjugate satisfies∇ψ ∗ (g) = arg max{⟨g, x⟩ − ψ(x)} = arg max ⟨g, x⟩ − ψ̃(x) = ∇ψ̃ ∗ (g),x∈Q

(5.15)

x∈Q

where we used the dilatability property (Theorem 5.3) in the second equality. Therefore,
since ψ̃ is a dilated DGF and its smoothed support function can be computed in linear time,
the smoothed support function of ψ can be computed in linear time in |Σ|.
An immediate consequence of the niceness established in the previous bullet point is that proximal
operators of ψ can be computed efficiently, asproxψ (g ∥ c) = ∇ψ ∗ (−g + ∇ψ(c)) = ∇ψ̃ ∗ (−g + ∇ψ(c)),
where the last equality is a special case of (5.15).
5.4.2.3

Strong convexity of the dilatable global entropy DGF

On the other hand, we now show that ψ has the advantage of a better strong convex modulus,
compared to the existing dilated entropy DGFs.
Theorem 5.4. The dilatable global entropy function ψ : Q → ℝ⩾0 is a DGF for the sequenceform polytope Q, 1-strongly convex on relint Q with respect to the ℓ2 norm.

Proof. The function ψ is twice-differentiable on (0, 1)Σ ⊇ relint Q. Using (5.1) we conclude that
ψ is 1-strongly convex, since the Hessian is2∇ ψ(x) = diag



wσ
x[σ]




≽ I,
σ∈Σ

where we used the inequalities 0 ⩽ x[σ] ⩽ 1 and wσ ⩾ 1.

Theorem 5.5. The dilatable global entropy function ψ is strongly convex modulus 1/MQ withrespect to the ℓ1 norm on relint Q.

Proof. Using the second-order definition of strong convexity, we wish to show that the inequality⟨m, ∇2 ψ(x)m⟩ ⩾ M1Q ∥m∥21 holds for any m ∈ ℝΣ . Expanding the Hessian matrix and using
the fact that wσ ⩾ 1 for all σ ∈ Σ gives

92§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS2⟨m, ∇ ψ(x)m⟩ =m, diag

wσ
x[σ]




σ∈Σ

 X
m[σ]2
m ⩾
.
x[σ]

(5.16)σ∈ΣOn the other hand, by expanding the definition of ∥m∥21 and applying the Cauchy-Schwarz
inequality, we have!2∥m∥21 =⩽

!2
X |m[σ]| pp|m[σ]| =
x[σ]
x[σ]
σ∈Σ
σ∈Σ
!
!
!
X
X m[σ]2
X m[σ]2x[σ] =∥x∥1 ⩽
x[σ]
x[σ]
X

σ∈Σ

σ∈Σ

σ∈Σ

X m[σ]2
σ∈Σ

x[σ]

!
MQ .

Substituting (5.16) into the last inequality yields a proof of the desired strong convexity
modulus 1/MQ .
5.4.2.4

Diameter of the dilatable global entropy DGF

The properties above immediately imply that our dilatable global entropy DGF satisfies all therequirements for a prox setup on the polytope of sequence-form strategies Q. Here we complete
the analysis by giving bounds on the diameter induced by ψ.
2
Theorem 5.6. The ψ-diameter Ωψ,Q of Q is at most MQ
maxj ′ ∈J log |Aj ′ |.

Proof. By the definition of the polytope diameter and the fact that we chose our DGFs suchthat minx∈A ψ(x) = 0, we haveΩψ,Q ⩽ max ψ(x) ⩽ max
x∈Q

x∈Q

X

γj x[pj ] log |Aj | ⩽ max max
log |Aj ′ |′x∈Q j ∈Jj∈J

⩽ MQ max max
log |Aj ′ |′x∈Q j ∈J

X

X

γj x[pj ]j∈Jx[pj ]

j∈J

2
⩽ MQ
max
log |Aj ′ |,′j ∈Jwhere the second inequality is by noting that log x[σ] ⩽ 0 since x[σ] ⩽ 1 for all σ ∈ Σ, thefourth inequality is by noting that γj is largest at root decision nodes, where it is at most MQ ,P
and the fifth inequality upper bounds j∈J x[pj ] by MQ .
In particular, Theorem 5.6 shows that the dilatable global entropy DGF improves the polytope
diameter of dilated entropy (Proposition 5.2) by a factor of 2DQ +2 , achieving a polytope diameterwith no exponential dependence on the depth DQ of the sequence-form polytope.

§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

93

Example 5.3. Let Q be the sequence-form polytope corresponding to an optimal stopping
problem: an agent has a sequence of k decision nodes, and at each decision node they caneither choose to stop the game, in which case a payoff is reached, or they can choose tocontinue the game, in which case they continue to the next decision node. For such a problem,
Q has depth k, which is also the size of Q as measured by the ℓ1 norm, i.e. MQ = k. In
this case, the diameter associated to ψ is k 2 log 2, whereas the dilated entropy DGF of Kroer,
Waugh, Kılınç-Karzan, and Sandholm (2020) gives diameter 2k+2 k 2 log 2.

Example 5.3 shows the most extreme type of decision problem in terms of our improvement
over existing results. Another example of a polytope that partially embeds this structure is thedecision spaces of a no-limit poker game. In those games, a player may raise by any amountbetween the minimum raise size and the size of their stack. Because of this, there exists a very
deep decision path where players take turns raising by the minimum amount. However, theaverage depth is much smaller than this.
Summing up our results on the dilatable global entropy, we have shown that it enjoys thesame fast smoothed-support-function computation as the dilated entropy DGF, while having abetter way to achieve strong convexity modulus 1/MQ . In particular, the existing dilated entropy
setup requires the weight parameters β to grow exponentially in the depth of the sequence-form
polytope, whereas we have only a linear growth in those weights. More concretely, this meansthat the largest weights maxj∈J βj in the dilated entropy DGF are larger than the largest weights
maxj∈J γj in the dilatable global entropy DGF by a factor of more than 2DQ . This in turn allowed
us to achieve a better polytope diameter by a factor of 2DQ +2 while retaining the same strong
convexity modulus.
5.4.2.5

Gradient properties of the dilatable global entropy DGF

The previous subsections establish that the dilated global entropy DGF ψ̃ is dilatable (i.e., itcoincides with a dilated entropy DGF, ψ̃, on all of their domain Q) and nice (i.e., gradients,smoothed support functions, and proximal operators can be computed efficiently). Furthermore,
we have seen that ψ̃ has very appealing metric properties, including a diagonal Hessian matrix
that renders establishing strong convexity rather trivial, especially as compared to the analyses ofdilated entropy DGFs in the prior literature (Kroer, Waugh, Kılınç-Karzan, and Sandholm, 2020).
In this subsection we go one step further: by studying the relationship between the gradientsof ψ and ψ̃, we will establish that the two enjoy the same strong convexity parameter. In particular,
we will make use of the following property, which we prove at the end of the section.

94§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONS

Lemma 5.2. At any point x ∈ relint Q, the gradients ∇ψ(x) and ∇ψ̃(x) differ only along
orthogonal directions to the affine hull aff Q of Q, that is,∇ψ(x) − ∇ψ̃(x), y − z = 0∀y, z ∈ Q.

Lemma 5.2 immediately implies that for any two points x, x′ ∈ relint Q∇ψ̃(x) − ∇ψ̃(x′ ), x − x′ = ∇ψ(x) − ∇ψ(x′ ), x − x′ ,that is, the strong convexity properties of ψ established in Section 5.4.2.3 apply to ψ̃ as well. In
particular, this shows that the specific dilated entropy DGE ψ̃ is 1-strongly convex on relint Q
with respect to the ℓ2 norm, and 1/MQ -strongly convex with respect to the ℓ1 norm. Additionally,
the analysis of the diameter of ψ transfers directly to ψ̃ by dilatability (Section 5.4.2.1). Combined,
these results enable us to introduce a 1-strongly convex dilated entropy DGF (and not just any nice
DGF) ψ̃ that for the first time has worst-case polynomially-bounded diameter.
We remark that it would be hard to establish such a result without first introducing our dilated
global regularizer, as analyzing the product ⟨∇ψ̃(x) − ∇ψ̃(x′ ), x − x′ ⟩ for a dilated entropy DGFψ̃ is notoriously involved. Instead, the good metric properties of ψ—especially the diagonal
Hessian matrix—make the analysis of the strong convexity and diameter properties trivial.
In fact, every strong convexity proof for an entropy DGF that we are aware of uses the
Hessian-based condition (Ahron Ben-Tal and Nemirovski, 2001; Kroer, Waugh, Kılınç-Karzan,
and Sandholm, 2020). If we had used this definition directly on the dilated DGF φ, thenthe lower bound from Kroer, Waugh, Kılınç-Karzan, and Sandholm (2020) implies that wewould get an exponential dependence on the depth. This is because the Hessian condition isa sufficient but not necessary condition for strong convexity: it asks for the condition to holdalong directions orthogonal to the affine hull of the sequence-form strategy space, which is notnecessary. Arguably, the key insight for the dilatable global entropy is that it allows us to use theHessian-based sufficient condition for proving strong convexity in a way that avoids paying thecost of maintaining strong convexity along irrelevant orthogonal directions. The fact that it also
greatly simplifies the Hessian-based analysis is an added bonus.
Another consequence of Lemma 5.2 is that the proximal steps induced by ψ and ψ̃ coincide.
Indeed, for all centers y ∈ relint Q and gradients g, one hasarg min ⟨g, y⟩ + Dψ (y ∥ x)y∈Q
= arg min ⟨g, y⟩ + ψ(y) − ∇ψ(x)⊤ (y − x)y∈Q§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS= arg min ⟨g, y⟩ + ψ̃(y) − ∇ψ(x)⊤ (y − x)95

(By dilatability, Theorem 5.3)y∈Q= arg min ⟨g, y⟩ + ψ̃(y) − ∇ψ̃(x)⊤ (y − x)(Lemma 5.2)y∈Q= arg min ⟨g, y⟩ + Dψ̃ (y ∥ x) .
y∈QHence, mirror descent, online mirror descent, follow-the-regularizer-leader, mirror prox, and
EGT (among others) all produce the same iterates when instantiated with ψ̃ or ψ. It is conceivablethat some methods—for examples methods that require taking norms of gradients, or methods
that rely on norms induced by the Hessian matrix of the regularizer (e.g., Abernethy and A.
Rakhlin (2009))—would in general not be equivalent when set up using ψ̃ or ψ.
Proof of Lemma 5.2. We will prove the theorem by structural induction on the structure of
the sequence-form strategy space. Specifically, we seek to prove that at all decision nodesj ∈ J , one has
X X ∂(ψ − ψ̃)(x)y[j ′ a′ ] = γj y[pj ] log x[pj ].
′ a′ ]∂x[j′′(5.17)j ≽j a ∈Aj ′Since y is arbitrary, the equality above will immediately imply that for any j such that pj = ∅X X ∂(ψ − ψ̃)(x)(y[j ′ a′ ] − z[j ′ a′ ]) = γj (y[∅] − z[∅]) log x[∅] = 0,
′ a′ ]∂x[j′′(5.18)j ≽j a ∈Aj ′where the last equality follows from the fact that y[∅] = z[∅] = 1 since y, z are sequence-formFstrategies. Now, since the set of sequences can be partitioned as Σ = {∅} ⊔ j∈J :pj =∅ {j ′ a′ :j ′ ≽ j, a′ ∈ Aj ′ }, and furthermore y[∅] = z[∅] = 1, it is evident that (5.18) considered for theset {j ∈ J : pj = ∅} immediately implies the statement.
In order to prove (5.17) we will use structural induction over the structure of the sequenceform decision problem.
• Base case: terminal decision nodes j. Consider a terminal decision node j, that is one forwhich Cja = {} for all a ∈ Aj . Then, for all a ∈ Aj we obtain from direct inspection that∂ψ(x) = wja (1 + log x[ja]);∂x[ja]
Hence,



∂ ψ̃
x[ja]
(x) = γj 1 + log.
∂x[ja]
x[pj ]

96§5.4. DISTANCE-GENERATING FUNCTIONS WITH LINEAR-TIME PROJECTIONSX ∂(ψ − ψ̃)a∈Aj∂x[ja](x)y[ja] =X wja − γj + (wja − γj ) log x[ja] + γj log x[pj ] y[ja]a∈Aj= γj y[pj ] log x[pj ],where we used the fact that by definition γj = wja = 1, as well as the fact that y is aP
sequence-form strategy and therefore a∈Aj y[ja] = y[pj ]. Since the only j ′ ≽ j is jitself, the proof of the base case is complete.
• Inductive step. Consider a nonterminal j. Again by direct inspection, we obtainX∂ψ
γj ′ log |Aj ′ |; and(x) = wja (1 + log x[ja]) +∂x[ja]j ′ ∈Cja


X
X
∂ ψ̃
x[ja]
(x) = γj 1 + log−
γj ′ +
γj ′ log |Aj ′ |∂x[ja]
x[pj ]j ′ ∈Cjaj ′ ∈Cja
X= wja + γj log x[ja] − γj log x[pj ] +γj ′ log |Aj ′ |,j ′ ∈Cja

where the last equality uses the definition of wja := γj −Pj ′ ∈Aj γj .
′Now, the set of descendant decision nodes {j ′ ∈ J : j ′ ≽ j} can be partitioned as
{j} ⊔

G

G

{j ′′ ∈ J : j ′′ ≽ j ′ }.

a∈Aj j ′ ∈Cja

Correspondingly, using the inductive hypothesis we have
X X ∂(ψ − ψ̃)(x)y[j ′ a′ ]∂x[j ′ a′ ]′′j ≽j a ∈Aj ′X ∂(ψ − ψ̃)X X=(x)y[ja] +γj ′ y[ja] log x[ja]∂x[ja]a∈Aj
a∈Aj j ′ ∈Cja



X 
=
(wja − γj ) log x[ja] + γj log x[pj ] y[ja]
a∈Aj

+

X X

γj ′ y[ja] log x[ja]a∈Aj j ′ ∈Cja


=

X
a∈Aj

wja − γj +Xj ′ ∈Cja

γj ′  log x[ja] + γj log x[pj ]y[ja]97§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS=X

γj y[ja] log x[pj ] = γj y[pj ] log x[pj ],a∈AjP
where we again used the definition of wja := γj − j ′ ∈Aj γj ′ , as well as the fact that y isP
a sequence-form strategy and therefore a∈Aj y[ja] = y[pj ].
This concludes the induction proof.

5.5

Experimental evaluation

In this section we compare the numerical performance of the dilated entropy and dilatable global
entropy DGFs for computing Nash equilibria in two-player zero-sum imperfect-information
extensive-form games.
Our experiments will be shown on nine different games, which span a variety of poker games,
other recreational games, as well as a pursuit-evasion game played on a graph. All games are
standard benchmarks in the computational game theory literature, and a full description of thegames is given in Appendix A. In Table 5.1(a) we summarize the game instances we use, as well
as some of their key dimensions: the number of decision points |J1 |, |J2 | for Player 1 and 2,
respectively, the number of sequences |Σ1 |, |Σ2 |, and the number of terminal nodes (leaves).

Decision Points Sequences|J1 | + |J2 | |Σ1 | + |Σ2 |Game instance
K23 Kuhn poker
L2232 Leduc poker (3 ranks)
L22d2 Leduc poker (13 ranks)
G24 Goofspiel
B2323 Battleship (3 turns)
B2324 Battleship (4 turns)
D26 Liar’s dice
P24 Pursuit-evasion (4 turns)
P25 Pursuit-evasion (6 turns)

12
288
5148
34 95281 027
1 050 723
24 576382
11 888(A) — Game instances and sizes

Leaves
|Z|

26
30
674
1116
12 01498 95642 65813 824327 070552 132
3 236 158 3 487 428
49 142147 4202086
15 89869 029118 514Weights β
Avg
Max
8.857
11.76612.057
6.912
3.294
3.753
15.5568.286
19.42438
686
12 32623 4422894
27 47065 54662
254(B)

Weights γ
Avg Max
2.286
7
2.117
43
2.131 703
1.698 917
1.242
99
1.331 483
2.043 1399
1.943
5
2.508
7
(C)

Table 5.1: (A) Various measures of the size of each of the games that we test algorithms on. (B), (C) The
magnitude of the dilated entropy DGF and dilatable global entropy DGF weights.

Our experiments will show performance on three algorithms. First, we will plot the performance for both the EGT and MPROX algorithms, with stepsizes and smoothing chosen according to
the theoretical values dictated by Theorems 5.1 and 5.2, with one minor variation: for each DGF,
we do not scale by MQ , which is required in order to achieve strong convexity with respect to the98§5.5. EXPERIMENTAL EVALUATIONℓ1 norm. This is done because scaling by this last factor leads to extremely slow performance forall of the DGFs. Second, we will also show results on a tweaked variant of EGT called ‘EGT/AS’,
which implements several heuristics that typically lead to better performance in practice, as seenin Hoda, Gilpin, Peña, and Sandholm, 2010; Kroer, Waugh, Kılınç-Karzan, and Sandholm, 2020;
Kroer, Farina, and Sandholm, 2018b. These heuristic are:
1. µ balancing: At each iteration, we take a step on the player i whose smoothing parameter µi
is larger.
2. Aggressive stepsizing: The original stepsize of EGT at iteration t is τ = 2/(3 + t), which istypically too conservative in practice. Instead, EGT/AS maintains some current value τ ,initially set at τ = 0.5. EGT/AS then repeatedly attempts to take steps with the current τ ,and after every step checks whether the invariant condition of EGT still holds. If not, thenwe undo the step, decrease τ , and repeat the process.
3. Initial µ fitting: The initial EGT values for µx , µy are much too conservative. Instead, At the
beginning of the algorithm we perform a search over initial values for µx = µy . The searchstarts at the candidate value µ = 10−6 and stops as soon as the choice of µx = µy = µ yields
an excessive gap value above 0.1. If the current choice does not, µ is incremented by 20%
and the fitting continues.
For all parameters above, we use the same values as in Kroer, Farina, and Sandholm (2018b), eventhough those values were tuned for the dilated entropy DGF, rather than dilatable global entropy.
In the presentation of the numerical performance, we will generally plot the number of
iterations of the FOM on the x-axis, rather than plot wall-clock time. Since we hold the algorithmic
setup fixed in each plot, apart from the DGF, this gives a fair representation of performance, since
they all use the same set of operations (in particular the same number of gradient computations,which is typically the most expensive operation). For EGT/AS, we will instead plot the number ofgradient computations on the x-axis, since the number of gradient computations can vary foreach DGF, depending on the amount of backtracking incurred.
We will focus on comparing our new dilatable global entropy for the sequence-form polytope (Definition 5.6) to the prior state-of-the-art dilated entropy DGF (Definition 5.5) from Kroer,
Waugh, Kılınç-Karzan, and Sandholm, 2020.
Before we study the numerical performance, we look at the size of the DGF weights β and γ
for each of the games. Table 5.1 column (b) shows the average and maximum size of the dilated
entropy, and Table 5.1 column (c) shows the corresponding values for the dilatable global entropy
DGF. We see that the dilatable global entropy DGF requires vastly less weight, especially in termsof the maximal weights near the root of each decision space.

99§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

5.5.1 MPROX and EGT without aggressive stepsizing
First, we study the theoretically-correct way to use the DGFs. In particular, we instantiate both
EGT and mirror prox with the stepsizes and DGFs as specified in Theorems 5.1 and 5.2, for the
dilatable global entropy and dilated entropy. The results for EGT are shown in Figures 5.1 and 5.2.
Across both algorithms and all nine games, we see that our new dilatable global entropy DGF
performs better, sometimes by over an order of magnitude (e.g., in liar’s dice and pursuit evasion(6 turns)). This is in line with the fact that our new DGF has a better strong convexity modulus,
which allows for a much smaller amount of smoothing, while still guaranteeing correctness. Thisin turns allows the algorithms to safely take larger steps, thereby progressing faster.

Nash gap

K23 – EGT

L2232 – EGT

10−1

L22d2 – EGT
100

100

10−2

10−1
Dilated entropy (ψ)

10−3

10

Global entropy (φ̃)

Dilated entropy (ψ)−1Dilated entropy (ψ)

Global entropy (φ̃)

Global entropy (φ̃)

10−2
100

101

102

103

100

Nash gap

G24 – EGT

102

103

100

B2323 – EGT

101

102

B2324 – EGT

100
Dilated entropy (ψ)

Dilated entropy (ψ)

2.6 × 100

Dilated entropy (ψ)

Global entropy (φ̃)

Global entropy (φ̃)

2.4 × 100

Global entropy (φ̃)

101

102

103

100

D26 – EGT

101

102

103

100

P24 – EGT

100

103

3.6 × 100
3.4 × 100
3.2 × 100
3 × 100
2.8 × 100

100

100

Nash gap

101

101

102

103

P26 – EGT

100

100

10−1
10−1
10

0

Dilated entropy (ψ)

Dilated entropy (ψ)

Global entropy (φ̃)

Global entropy (φ̃)

10

1

10

2

Iteration

10

3

0

10

1

10

2

10

Iteration

Dilated entropy (ψ)

10−1
10

3

Global entropy (φ̃)

10

0

101

102

103

Iteration

Figure 5.1: Performance of the EGT algorithm instantiated with the two entropy DGFs across ninegames. The x-axis shows the number of EGT iterations, and the y-axis shows the distance to Nashequilibrium.

100§5.5. EXPERIMENTAL EVALUATION

K23 – MPROXNash gap

100

L2232 – MPROX

10−1

L22d2 – MPROX

100

100

10−2
Dilated entropy (ψ)
−3

10

100

101

102

10−1

Dilated entropy (ψ)

10−1Global entropy (φ̃)

103

100

G24 – MPROX

101

102

Global entropy (φ̃)

103Nash gap

100

B2323 – MPROX

101

102

103

B2324 – MPROX
3.6 × 100
3.4 × 1000

2 × 103.2 × 100

100

3 × 100

100

6 × 10−1
4 × 10−1
3 × 10−1
10

0

2.8 × 100

Dilated entropy (ψ)

Dilated entropy (ψ)

Global entropy (φ̃)

Global entropy (φ̃)

1

10

2

10

3

10

10

0

D26 – MPROX

10

1

10

2

Dilated entropy (ψ)

2.6 × 1001010Dilated entropy (ψ)

Global entropy (φ̃)

102

103

103

Dilated entropy (ψ)

Global entropy (φ̃)

10
101

102

100

10−1

Dilated entropy (ψ)

101

P26 – MPROX

100

100

Global entropy (φ̃)
0

3

P24 – MPROX

100

Nash gap

Dilated entropy (ψ)

Global entropy (φ̃)

100

101

Iteration

102

Iteration

103

Global entropy (φ̃)

−1

100

101

102

103

Iteration

Figure 5.2: Performance of the MPROX algorithm instantiated with the two entropy DGFs across ninegames.

5.5.2

EGT with aggressive stepsizing

Secondly, we investigate the numerical performance of the two entropy DGFs in the EGT/AS
algorithm in Figure 5.3.

§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS

K23 – EGT/AS
Nash gap

10−1

Dilated entropy (ψ)
Global entropy (φ̃)

10−2
10

L2232 – EGT/AS
100

Dilated entropy (ψ)

100
10−1

10−2

10−2

10

Dilated entropy (ψ)
Global entropy (φ̃)

−1

10−4

10−3
103

102

103

G24 – EGT/AS
100

Nash gap

L22d2 – EGT/AS

Global entropy (φ̃)

−3

101

102

B2323 – EGT/AS

Dilated entropy (ψ)

Dilated entropy (ψ)

Global entropy (φ̃)

Global entropy (φ̃)

10−1

10−1

−2

−2

103

B2324 – EGT/AS
100

Dilated entropy (ψ)
Global entropy (φ̃)

10−1
10

10

10−2
102

103

102

D26 – EGT/AS
Nash gap

100

Dilated entropy (ψ)

103

102

P24 – EGT/AS
100

Global entropy (φ̃)

Dilated entropy (ψ)

P26 – EGT/AS
100

10−1

10−1

10−2

10−2

10−2

10−3
103

Gradient computations

Dilated entropy (ψ)
Global entropy (φ̃)

Global entropy (φ̃)

10−1

102

103

10−3
103

Gradient computations

103

Gradient computations

Figure 5.3: Performance of the EGT/AS algorithm instantiated with the two entropy DGFs, as well as
aggressive stepsizing, µ balancing, and initial µ fitting.

Here we see a smaller performance improvement. For most of the games, we get a small factor
of improvement for the first 100 or so iterations, but then the performance is similar thereafter. For
liar’s Dice there is a persistent improvement to using dilatable global entropy across all iterations.

5.A

Appendix: Properties of SMPL functions

In this appendix we mention several basic results about strictly monotonically increasing
piecewise-linear (SMPL) and quasi-SMPL functions. Proofs of elementary results are omitted.

102§5.A. APPENDIX: PROPERTIES OF SMPL FUNCTIONS

Lemma 5.3. Let f : I → ℝ be SMPL, and consider a standard representation of f ofsize S. Then, for any b ∈ ℝ and k ⩾ 0, a standard representation for the SMPL function
I ∋ x 7→ b + kf (x) can be computed in O(S + 1) time.

Lemma 5.4. The sum f1 + · · · + fn of n SMPL (resp., quasi-SMPL) functions fi : I → ℝ
is a SMPL (resp., quasi-SMPL) function I → ℝ. Furthermore, if each fi admits a standard
representation of size Si , then a standard representation of size at most S1 + · · · + Sn for theirsum can be computed in O(S1 + · · · + Sn + 1) log n) time.

Lemma 5.5. Let f : ℝ → ℝ be SMPL, and consider a standard representation of f of size S.
Then, for any β ∈ ℝ, a standard representation of size at most S for the quasi-SMPL function
I ∋ x 7→ [f (x) − β]+ can be computed in O(S + 1) time.

Lemma 5.6. The inverse f −1 : range(f ) → ℝ of a SMPL function f : I → ℝ is SMPL.
Furthermore, if f admits a standard representation of size S, then a standard representation
for f −1 of size at most S can be computed in O(S + 1) time.

Lemma 5.7. Let f : ℝ → [0, +∞) be quasi-SMPL. The inverse f −1 : (0, +∞) → ℝ of f
restricteda to domain (0, +∞) is SMPL. Furthermore, if f admits a standard representation ofsize S, then a standard representation of size at most S for f −1 can be computed in O(S + 1)
time.
aWe restrict the domain to (0, +∞) because f −1 (0) may be multivalued.

Proof. We have f (x) = [g(x)]+ where g is SMPL. It follows that the function ḡ : I → ℝ definedas ḡ(x) = g(x) for the interval I = {x : g(x) > 0} is SMPL as well. For any x such that f (x) > 0
we have x ∈ I, and thus f −1 = g −1 , and it follows from Lemma 5.6 that f −1 is SMPL.

Lemma 5.8. Let f : [0, +∞) → ℝ be a SMPL function, and consider the function g thatmaps y to the unique solution to the equation x = [y − f (x)]+ . Then, g is quasi-SMPL andsatisfies g(y) = [(x + f )−1 (y)]+ , where (x + f )−1 denotes the inverse of the SMPL function
x 7→ x + f ([x]+ ).

§5. NOTIONS OF DISTANCE FOR SEQUENCE-FORM STRATEGIES, AND PROX METHODS103

Proof. For any y ∈ ℝ, the function hy : x 7→ x − [y − f (x)]+ is clearly SMPL on [0, +∞).
Furthermore, hy (0) ⩽ 0 and hy (+∞) = +∞, implying that hy (x) = 0 has a unique solution.
We now show that g(y) = [(x+f )−1 (y)]+ is that solution, that is, it satisfies g(y) = [y −f (g(y))]+for all y ∈ ℝ. Fix any y ∈ ℝ and letḡ := (x + f )−1 (y)⇐⇒ḡ + f ([ḡ]+ ) = y⇐⇒ḡ = y − f ([ḡ]+ )(5.19)

There are two cases:• If ḡ ⩾ 0, then g(y) = [ḡ]+ = ḡ, and so we haveg(y) = [ḡ]+ = [y − f ([ḡ]+ )]+ = [y − f (g(y))]+ ,as we wanted to show.
• Otherwise, ḡ < 0 and g(y) = 0. From (5.19), the condition ḡ < 0 implies y < f ([ḡ]+ ) =f (0). So, it is indeed the case that0 = g(y) = [y − f (0)]+ = [y − f (g(0))]+ ,as we wanted to show.
Finally, we note that the function (x + f )−1 : ℝ → ℝ is SMPL due to Lemma 5.6, implying thatg(y) is quasi-SMPL.

104§5.A. APPENDIX: PROPERTIES OF SMPL FUNCTIONSChapter 6

Strongly predictive learning
dynamics, and OT (log T /T )convergence in self play
As we have seen in the past chapters, it is possible to devise learning algorithms with guaranteed√
external regret of order OT ( T ) in any imperfect-information extensive-form game, without any
assumption on the sequence of utility vectors observed by the learning algorithm. Furthermore, when
all players employ any such algorithm (with potentially different players opting for different
options), the average distribution of play converges to the set of coarse correlated equilibria.
The combination of the two previous facts is remarkable: equilibrium can be reached at√
the approximation rate of OT (1/ T ) even when playing against adversarial unknown players.
However, this begs a natural question:√
Is a regret rate better than OT ( T ) achievable in less adversarial (more predictable) environments,
including the setting of training agents using self play?
What are the optimal performance guarantees we can obtain when learning agents arecompeting against each other?

6.1

Related work

The fundamental question above was first formulated and addressed by Daskalakis, Deckelbaum,and Kim (2011) within the context of nonsequential, zero-sum games. Since then, there has beena considerable interest in extending their guarantee to more general settings (S. Rakhlin and
Sridharan, 2013; Syrgkanis, Agarwal, Luo, and Schapire, 2015; D. J. Foster, Li, Lykouris, Sridharan,

106§6.2. CONTRIBUTIONS

and Tardos, 2016; Chen and Peng, 2020; Daskalakis and Golowich, 2022; Piliouras, Sim, andSkoulakis, 2021). In particular, Daskalakis, Fishelson, and Golowich (2021) recently establishedthat when all players in a general normal-form game employ the optimistic variant of multiplicative
weights update (MWU) (reviewed in Section 3.3.1), the regret of each player grows nearly-optimallyas OT (log4 T ) after T repetitions of the game, leading to an exponential improvement over theguarantees obtained using traditional techniques within the no-regret framework. However,while normal-form games are a common way to represent strategic interactions in theory, mostsettings of practical significance inevitably involve more complex strategy spaces. For thosesettings, any faithful approximation of the game using the normal form is typically inefficient,requiring an action space that is exponential in the natural parameters of the problem, thereby
limiting the practical implications of those prior results.

6.2

Contributions

As mentioned, a positive answer for the fundamental question expressed above was known
for nonsequential games but not for imperfect-information extensive-form games, which aresignificantly more involved. In this chapter we answer the question in the positive for any
imperfect-information extensive-form games, no matter the number of players. Specifically, in thischapter we will show the first uncoupled no-regret algorithms that, if used in self play, guarantee
cumulating OT (log T ) regret in the worst case, respectively. This method improves over theprior state of the art, which achieved OT (log4 T ) (albeit with better dependence on the size ofthe game) and was based on the significantly different technique of Kernelized multiplicative
weights, described in Chapter 7.
One remarkable property about the algorithm we describe in this section, which we call
log-regularized lifted optimistic FTRL (LRL-OFTRL), is its wide applicability even beyond imperfectinformation extensive-form games. Indeed, like the OMD and FTRL algorithms we discussed in
Chapter 5, LRL-OFTRL can be used to define no-external-regret dynamics that apply to any convex
and compact strategy set X , not just sequence-form strategy spaces. However, LRL-OFTRL differsfrom OMD and FTRL in at least two significant ways. First, LRL-OFTRL defines strongly predictive
dynamics. This property will be fundamental in establishing logarithmic regret bounds in selfplay. Second, LRL-OFTRL does not enable flexibility in the proximal setup used to instantiate
the algorithm, and rather requires that a logarithmic regularizer be used. This inflexibility
should be naturally met with suspicion, as the logarithmic DGF is not “nice” in the sense of
Definition 5.2, and for general convex and compact strategy sets an exact algorithm for projectingwith respect to the logarithmic DGF might not even exist. Luckily, we show that under mild
assumptions—which we show are satisfied in the sequence-form strategy polytope setting—anapproximate projection can be computed with an extremely favorable log log(1/ϵ) dependence on
the desired approximation level ϵ > 0.

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
107

6.3

Setup and algorithm pseudocode

Due to the applicability of LRL-OFTRL to any convex and compact strategy set, we present
LRL-OFTRL in full generality, letting X ⊆ ℝd denote such a strategy set. Without loss of generality,
we will further assume that X ⊆ [0, +∞)d (otherwise, it suffices to first shift the set), and that
there is no index r ∈ [[d]] such that x[r] = 0 for all x ∈ X (if not, dropping the identically-zero
dimension would not alter regret).
The pseudocode for LRL-OFTRL is simple, and is given in Algorithm 6.1.
Algorithm 6.1: Log-Regularized Lifted Optimistic FTRL (LRL-OFTRL)
Data: • X ⊆ ℝd⩾0 convex and compact strategy set• η ∈ ℝ>0 learning rate.
1

Ũ

(1)

← 0 ∈ ℝd+1(t)Σfunction NextStrategy(m ∈ℝ )(t)(t)−⟨m , x ⟩3m̃(t) ←
m(t)
)
( 
 
 (t) 
d
X
(t)
λ
λ
(t)
+ log λ +log y[r]4← arg max η Ũ + m̃ ,yy (t)(λ,y)∈X̃
2[▷ Lifting][▷ FTRL step]

r=1

5

return x(t) :=

y (t)
∈X
λ(t)

[▷ Normalization](t)Σfunction ObserveUtility(u ∈ℝ )(t)(t)−⟨u , x ⟩7ũ(t) ←
u(t)
6

8

Ũ

(t+1)

← Ũ

(t)

[▷ Lifting]+ ũ(t)

It is easy to see that internally LRL-OFTRL builds on the follow-the-regularized-leader (FTRL)
algorithm, which was introduced in Section 5.2.3, but with some important twists.
1. Lifting. First, the FTRL steps are performed not on X , but rather on the lifted strategy set X̃ ,defined asX̃ := (λ, y) : λ ∈ [0, 1], y ∈ λX

⊆ ℝd+1 .

(6.1)(See Figure 6.1 for geometric intuition on the relationship between X and X̃ ). The utility
vectors u(t) ∈ ℝd passed to LRL-OFTRL as feedback will also be lifted to vectors ũ(t) ∈ ℝd+1 ,defined on Line 7; this ensures that ũ(t) is orthogonal to the vector (1, x(t) ).
2. Logarithmic regularization. Second, FTRL is instantiated with the logarithmic DGF for ℝd+1 ,that is, the function

108§6.4. REGRET ANALYSISλ
Lifting
X
1

X̃

Figure 6.1: The lifting operation performed by the LRL-OFTRL algorithm on the strategy set.

ζ(λ, y) := − log λ −dXlog y[r],∀(λ, y) ∈ ℝd+1
>0 .

r=1

We discuss how the optimization problem defined in Line 4 can be solved efficiently in
Section 6.6. Below we point out that Line 4 is indeed well-defined.
Proposition 6.1. For any η ⩾ 0 and at all times t ∈ ℕ⩾0 , the FTRL optimization problem
on Line 4 of Algorithm 6.1 admits a unique optimal solution (λ(t) , y (t) ) ∈ X̃ ∩ ℝd+1
>0 .
3. Normalization. Third, given the iterate (λ(t) , y (t) ) output by the FTRL step at time t, ourno-regret algorithm over X selects the next strategy x(t) := y (t) /λ(t) (Line 5); this is indeed
a valid strategy in X by definition of X̃ in (6.1), as well as the fact that λ(t) > 0 as asserted
in Proposition 6.1.

6.4

Regret analysis

In this section, we study the regret of LRL-OFTRL under the idealized assumption that the
optimization problem on Line 4 (FTRL step) is solved exactly at each time t. In Section 6.6 wewill relax that assumption, and study the regret of LRL-OFTRL when the solution to Line 4 isapproximated using variants of Newton’s method.
Connection between regret on X and on X̃
external regret

Reg

(T )

Our ultimate goal will be that of analyzing the

( T
)
X
(t)
(t)
:= max⟨u , x̂ − x ⟩
x̂∈X

t=1

accumulated by LRL-OFTRL up to any time T ∈ ℕ⩾1 . To do so, it will be particularly useful torelate it to the regret R̃eg

(T )incurred with respect to the lifted utilities ũ(t) by the lifted iterates

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
109

(λ(t) , y (t) ) ∈ X̃ (produced by Line 4) in the lifted space X̃ , that is,

R̃eg

(T )

:=

*
T
X

max
(λ∗ ,y ∗ )∈X̃

t=1

λ∗
ũ ,
y∗

!

(t)

−

λ(t)
y (t)

!+
.

As the following theorem clarifies, there is a strong connection between R̃eg
Theorem 6.1. For any time T ∈ ℕ⩾0 it holds that R̃eg
follows that R̃eg

(T )⩾ 0 and Reg(T ) ⩽ R̃eg

(T )

(T )

(T )

and Reg(T ) .

= max{0, Reg(T ) }. In particular, it

for any T ∈ ℕ⩾0 .

Proof. First, by definition of ũ(t) in Line 7, it follows that for any t,
*

λ(t)
ũ ,
y (t)

!+

(t)

*
=

1
ũ ,
x(t)

!+

(t)

= 0.

As a result, we have that max{0, Reg(T ) } is equal to
*
)
(
T
T
X
X
ũ(t) ,
⟨u(t) , x∗ − x(t) ⟩ = max 0, max
max 0, max
∗
∗
(

x ∈X

x ∈X

t=1

(

t=1

1
x∗

!
−

1
x(t)

!+)

*

!+)
1
= max 0, max
ũ ,
x∗ ∈X
x∗
t=1
*
!+
T
X
λ∗
(t)
ũ ,
= max
y∗
(λ∗ ,y ∗ )∈X̃
t=1
*
!
!+
T
X
λ∗
λ(t)
(T )
(t)
= max
ũ ,
−
= R̃eg ,
∗
(t)
∗
∗
y
y
(λ ,y )∈X̃
t=1
T
X

(t)

as we wanted to show.
(T )The nonnegativity of R̃eg
will be a crucial property in establishing Theorem 6.2. Further,
Theorem 6.1 implies that a guarantee over the lifted space can be automatically translated to aregret bound over the original space X .
Local norms induced by the logarithmic DGF Given any vector (λ, y) ∈ X̃ ∩ ℝd+1
>0 , the Hessian
matrix of ζ at (λ, y) induces a local primal-dual norm pair centered at that point, defined as
a
z

!
(λ,y)

v
u 
2
d 
u a 2 X
z[r]
:= t
+
,
λ
y[r]
r=1

a
z

v
u
d
u
X
2
:= t(aλ)2 +(z[r]y[r])!
∗,(λ,y)r=1

110§6.4. REGRET ANALYSIS

for any (a, z) ∈ ℝd+1 . (It is a well-known fact that ∥ · ∥∗,(λ,y) is the dual norm of ∥ · ∥(λ,y) , and viceversa.) To simplify notation, we will use the streamlined symbols∥ · ∥t := ∥ · ∥(λ(t) ,y(t) )and∥ · ∥∗,t := ∥ · ∥∗,(λ(t) ,y(t) )(6.2)

to denote the local norms centered at the iterate (λ(t) , y (t) ) produced by FTRL at time t (Line 4).
In the next proposition we establish a refined regret bound in terms of this primal-dual norm
pair.
(T )

Proposition 6.2 (Regret of FTRL in local norms). Let R̃eg
be the regret cumulated up to
time T by the internal FTRL algorithm. If ∥u(t) ∥∞ ∥x∥1 ⩽ 1 at all times t ∈ [[T ]], then for any
1time horizon T ∈ ℕ⩾0 and learning rate η ⩽ 50
,
R̃eg

(T )

T
T −1
X
2
1 X
(d+1) log T+ 5ηũ(t) − m̃(t)−⩽η27η t=1
∗,t
t=1

!
!2
λ(t+1)λ(t).
− (t)y (t+1)yt

Proposition 6.2 differs from prior analogous results in that the regularizer is not a barrier over thefeasible set.
Multiplicative stability of the lifted iterates The iterates produced by FTRL satisfy a refinednotion of stability, which we refer to as multiplicative stability.
1
Proposition 6.3. For any time t ∈ ℕ⩾0 and learning rate η ⩽ 50
, if ∥u(t) ∥∞ ∥x∥1 , ∥m(t) ∥∞ ∥x∥1 ⩽1 for all x ∈ X ,!
!
λ(t+1)λ(t)−⩽ 22η.
y (t+1)y (t)tIntuitively, this property ensures that coordinates of successive iterates will have a small multiplicative deviation. We leverage this notion of stability to establish the following crucial
lemma.
1
Lemma 6.1. For any time t ∈ ℕ⩾0 and learning rate η ⩽ 50
, if ∥u(t) ∥∞ ∥x∥1 ⩽ 1,

∥x

(t+1)

(t)

− x ∥1 ⩽ 4∥X ∥1

λ(t+1)y (t+1)

!
−

λ(t)
y (t)

!
.
t

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
111

Establishing strong predictivity Combining this lemma with Proposition 6.2 allows us toobtain an RVU bound for the original space X , with no dependencies on local norms.
Corollary 6.1 (RVU bound in the original (unlifted) space). Fix any time T ∈ ℕ⩾0 , and1
suppose that ∥u(t) ∥∞ ⩽ B for any t ∈ [[T ]]. If η ⩽ 256B∥X
∥1 ,
R̃eg

(T )

⩽ 6B∥X ∥1 +T
−1X
(d + 1) log T+ 16η∥X ∥21∥u(t) − m(t) ∥2∞
η
t=1

−

6.5

T
−1
X
1
∥x(t+1) − x(t) ∥21 .
2512η∥X ∥1 t=1Connecting strong predictivity and logarithmic regret

The variation in one player’s utilities is related to the variation in the joint strategies based on thesmoothness condition of the utility function, connecting the last two terms of the RVU bound. In
particular, for the rest of the chapter, we will work under the following standard assumptions,which are always verified in (finite) normal-form and extensive-form games.
Assumption 6.1. The utility function ui (x1 , . . . , xn ) of any player i ∈ [[n]] satisfies thefollowing properties:
1. (Concavity) ui (xi , x−i ) is concave in xi for x−i = (x1 , . . . , xi−1 , xi+1 , . . . , xn ) ∈ j̸=i Xj ;n
2. (Bounded gradients) for any (x1 , . . . , xn ) ∈ j=1 Xj , ∥∇xi ui (x1 , . . . , xn )∥∞ ⩽ B, forsome parameter B > 0; andn3. (L-smoothness) there exists L > 0 so that for any two joint strategy profiles x, x′ ∈ j=1 Xj ,

×

×

×

∥∇xi ui (x) − ∇xi ui (x′ )∥∞ ⩽ L

X∥xj − x′j ∥1 .

j∈[[n]]

Given the assumption above, and by leveraging the nonnegativity of the regrets in the lifted
space, we establish that the second-order path lengths of the dynamics up to time T are boundedby OT (log T ):

112

§6.6. IMPLEMENTATION AND ITERATION COMPLEXITY

Theorem 6.2. If all players follow LRL-OFTRL with learning rate

11
η ⩽ min,,
256B∥X ∥1 128nL∥X ∥21
where ∥X ∥1 := maxi∈[[n]] ∥Xi ∥1 , then
n T
−1
X
X

(t+1)

∥xi

(t)

− xi ∥21 ⩽ 6144nηB∥X ∥31 + 1024n(d + 1)∥X ∥21 log T.

(6.3)

i=1 t=1

We next leverage Theorem 6.2 to establish Theorem 6.3.
Theorem 6.3. If all players use LRL-OFTRL within the COLS (that is, with m(t) = u(t−1) ; cf.
Section 3.2.1) with learning rate11,
η = min,
256B∥X ∥1 128nL∥X ∥21(T )then for any T ∈ ℕ⩾0 the regret Regi
(T )

Regi

of each player i ∈ [[n]] can be bounded as⩽ 12B∥X ∥1 + 256(d + 1) max nL∥X ∥21 , 2B∥X ∥1 log T.

(6.4)

Furthermore, the algorithm can be adaptive so that if player i is instead facing adversarial√(T )utilities, then Regi = OT ( T ).
For clarity, below we cast (6.4) of Theorem 6.3 in normal-form games with utilities normalized in
the range [−1, 1], in which case we can take B = 1, L = 1 and ∥X ∥1 = 1.
Corollary 6.2 (Normal-form games). Suppose that all players in a normal-form game with1n ⩾ 2 follow LRL-OFTRL within the COLS with learning rate η = 128n
. Then, for any T ∈ ℕ⩾0
and player i ∈ [[n]],
(T )

Regi

6.6

⩽ 12 + 256n(d + 1) log T.

Implementation and iteration complexity

In this section, we discuss the implementation and iteration complexity of LRL-OFTRL. The maindifficulty in the implementation is the computation of the solution to the strictly concave nonsmooth§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
113

constrained optimization problem in Line 4. We start by studying how the guarantees laid out in
Theorem 6.3 are affected when the exact solution to the FTRL problem (Line 5) in Algorithm 6.1 isreplaced with an approximation. Specifically, suppose that at all times t the solution to the FTRL
step (Line 4) in Algorithm 6.1 is only approximately solved within tolerance ϵ(t) , in the sense that
λ(t)
y (t)

!

(t)

−

λ⋆
(t)
y⋆

!
⩽ ϵ(t) ,

(6.5)

(t)
(t)
(λ⋆ ,y ⋆ )where (λ(t) , y (t) ) ∈ ℝd+1
>0 and
(t)

λ⋆
(t)
y⋆

!

( *
:= arg max η Ũ(λ,y)∈X̃(t)

λ+ m̃ ,y!+(t)+ log λ +dX)log y[r] .

r=1

Then, it can be proven directly from the definition of regret that the guarantees given in
PT
Corollary 6.1 deteriorate by an additive factor proportional to the sum of the tolerances t=1 ϵ(t) .
As an immediate corollary, when ϵ(t) := ϵ := 1/T , the conclusion of Theorem 6.3 applies evenwhen the solution to the optimization problem on Line 4 is only approximated up to ϵ tolerance.
Therefore, to complete our construction, it suffices to show that it is indeed possible to efficiently
compute approximate solutions to the FTRL step. In the remainder of this section, we show that
this is indeed the case assuming access to two different types of oracles. It should be stressed that
optimizing over a general convex set introduces several challenges not present under simplex
domains, inevitably leading to an increased per-iteration complexity compared to algorithms
designed specifically for normal-form games—such as OMWU.

6.6.1

Local proximal oracle

First, we will assume access to a proximal oracle in local norm for the set X̃ , that is, access to afunction that is able to compute the solution to the (positive-definite) quadratic optimization
problem1Πw̃ (g̃) := arg min ⟨g̃, x̃⟩ + ∥x̃ − w̃∥2w̃
2x̃ ∈ X̃



(

2 )
d+1
1 X x̃[r]= arg min ⟨g̃, x̃⟩ +−1
2 r=1 w̃[r]x̃ ∈ X̃

(6.6)

d+1
for arbitrary centers w̃ ∈ ℝd+1
. For certain sets X ⊆ ℝd , exact proximal>0 and gradients g̃ ∈ ℝ
oracles with polynomial complexity in the dimension d can be given. In particular, this is thecase for sequence-form strategy polytopes, where Πw̃ (g̃) can be computed using the algorithm
described in Section 5.3.1. Specifically, the following holds.

114§6.6. IMPLEMENTATION AND ITERATION COMPLEXITY

Proposition 6.4. Let X = Q be the polytope of sequence-form strategies for a player in animperfect-information extensive-form game. Then, the local proximal oracle Πw̃ (g̃) definedin (6.6) can be implemented exactly in time polynomial in the number of sequences |Σ| for|Σ|+1
any w̃ ∈ ℝ>0 and g̃ ∈ ℝ|Σ|+1 .
(t)

(t)

When an efficient, exact local proximal oracle is available, the projection (λ⋆ , y ⋆ ) ∈ X̃ canbe computed up to any precision ϵ > 0 using log log(1/ϵ) calls to the oracle, by using the proximal
Newton algorithm (Tran-Dinh, Kyrillidis, and Cevher, 2015). More precisely, Tran-Dinh, Kyrillidis,
and Cevher (2015) studied the following composite minimization problem:min {F (x̃) := f (x̃) + g(x̃)},(6.7)

x̃∈ℝd+1

where f is a (standard) self-concordant and convex function, and g : ℝd+1 → ℝ ∪ {+∞} is aproper, closed and convex function. In our setting, we will let g be defined asg(x̃) :=

0

if x̃ ∈ X̃ ,+∞ otherwise.

Further, for a given time t ∈ ℕ⩾0 , we let
d+1
D (t)
E X
f : x̃ 7→ −η Ũ + m̃(t) , x̃ −log x̃[r]
r=1

and


1
s̃k := arg min f (x̃k ) + (∇f (x̃k ))⊤ (x̃ − x̃k ) + (x̃ − x̃k )⊤ ∇2 f (x̃k )(x̃ − x̃k ) ,
2
x̃∈X̃

(6.8)

for some x̃k ∈ ℝd+1
>0 . The optimization problem (6.8) can be solved directly through the local
proximal oracle. The proximal Newton method given in Algorithm 6.2.
The algorithm proceeds in two phases. In the first phase we perform damped steps of proximal
Newton until we reach the region of quadratic convergence. Afterwards, we perform full steps ofproximal Newton until the desired precision ϵ > 0 has been reached. The following is known.
Theorem 6.4 (Tran-Dinh, Kyrillidis, and Cevher, 2015, Theorem 9). Algorithm 6.2 returns∗x̃K ∈ ℝd+1
>0 such that ∥x̃K − x̃ ∥x̃∗ ⩽ 2ϵ after at most§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
115

Algorithm 6.2: Proximal Newton method (Tran-Dinh, Kyrillidis, and Cevher, 2015)
Data: Initial point x̃0 ; Precision ϵ > 0; Constant σ := 0.2
for k = 1, . . . , K do
Obtain the proximal Newton direction d̃k ← s̃k − x̃k , where s̃k is defined in (6.8)
3
Set λk ← ∥d̃k ∥x̃k
4
if λk > 0.2 then5x̃k+1 ← x̃k + αk d̃k , where αk := (1 + λk )−1[▷ Damped Step]
6
else if λk > ϵ then7x̃k+1 ← x̃k + d̃k[▷ Full Step]
8else
9return x̃k
1

2


K=

 


f (x̃0 ) − f (x̃∗ )0.28+ 1.5 ln ln
+2
0.017
ϵ

iterations, for any ϵ > 0, where x̃∗ = arg minx̃ F (x̃), for the composite function F definedin (6.7).

Corollary 6.3. Given any ϵ > 0, it is possible to compute (λ(t) , y (t) ) ∈ X̃ ∩ ℝd+1
>0 such that (6.5)(t)
holds for ϵ = ϵ using O(log log(1/ϵ)) operations and O(log log(1/ϵ)) calls to the proximal
oracle defined in Equation (6.6).

Proof. The corollary follows from Theorem 6.4 by initializing Algorithm 6.2 at every iterationt ⩾ 2 with x̃0 := x̃(t−1) = (λ(t−1) , y (t−1) ). Then, as long as ϵ(t−1) is sufficiently small,the number of iterations predicted by Theorem 6.4 will be bounded by O(log log(1/ϵ)), as
claimed.

6.6.2

Linear maximization oracle

Moreover, we consider having access to a weaker linear maximization oracle (LMO) for the set X :LX (u) := arg max⟨x, u⟩.

(6.9)

x∈X

Such an oracle is more realistic in many settings (Jaggi, 2013), and it is particularly natural in thecontext of games, where it can be thought of as a best response oracle. We point out that an LMO
for X automatically implies an LMO for X̃ . The following guarantee follows readily by applying
the Frank-Wolfe (projected) Newton method (Liu, Cevher, and Tran-Dinh, 2020, Algorithms 1 and 2).

116§6.7. EXPERIMENTAL EVALUATION

Theorem 6.5 (Frank-Wolfe Newton). Given any ϵ > 0, it is possible to compute (λ(t) , y (t) ) ∈(t)= ϵ using O(poly(1/ϵ)) operations and O(poly(1/ϵ))X̃ ∩ ℝd+1>0 such that (6.5) holds for ϵ
calls to the LMO oracle defined in Equation (6.9).

6.7

Experimental evaluation

Finally, we support the theory developed in this chapter by conducting experiments on four established benchmark games: K23 and K33 , respectively 2-player and 3-player Kuhn poker (Kuhn,
1950); GL23 2-player limited-information Goofspiel (Ross, 1971); and S1a22 , the baseline versionof (2-player) Sheriff (Farina, Ling, Fang, and Sandholm, 2019a). As usual, a full description of
the game instances used is available in Appendix A. Of these games, only 2-player Kuhn poker( K23 ) is a zero-sum game.

The experiments are meant to verify the logarithmic per-player regret bound established in
Theorem 6.3. Our findings are summarized in Figure 6.2, where we remark that the x-axis is on alogarithmic scale. All plots refer to the choice of learning rate η := 0.5, which was selected after avery mild tuning process.

60

K23 — 2-Player Kuhn poker

K33 — 3-Player Kuhn poker

Player 1

Player 1

Player’s external regret

Player’s external regret§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
117

Player 2
40

20

Player 2
100

Player 3

50

0

0
100

101

102
103
Iteration

104

105

100

Player 1
400

102
103
Iteration

104

105

S1a22 — 2-Player Sheriff
Player’s external regret

Player’s external regret

GL23 — 2-Player Goofspiel

101

Player 2

300
200
100

2000

Player 1
Player 2

1500
1000
500
0

0
0

10

10

1

2

3

10
10
Iteration

10

4

10

5

100

101

102
103
Iteration

104

105

Figure 6.2: The regret of the players when they follow our learning dynamics, LRL-OFTRL. The x-axis
indexes the iteration, while the y-axis the regret. The scale on the x-axis is logarithmic. We observethat the regret of each player grows as OT (log T ), verifying Theorem 6.3.

We confirm that each player’s regret indeed appears to have a linear trend in the semilogarithmic plot, which corresponds to a logarithmic rate of growth as a function of the number of
iterations T .

6.A

Appendix: Proof details

6.A.1

Proof of Proposition 6.1

Proposition 6.1 (Restated). For any η ⩾ 0 and at all times t ∈ ℕ⩾0 , the FTRL optimization
problem on Line 4 of Algorithm 6.1 admits a unique optimal solution (λ(t) , y (t) ) ∈ X̃ ∩ ℝd+1
>0 .

118

§6.A. APPENDIX: PROOF DETAILSProof. Uniqueness follows immediately from strict convexity. In the rest of the proof we focus
on the existence part.
We start by showing that there exists a point x̃ ∈ X̃ whose coordinates are all strictly
positive. By hypothesis (see Section 6.3), for every coordinate r ∈ [[d]], there exists a point xr
such that xr [r] > 0. Hence, by convexity of X ⊆ [0, +∞)d and by definition of X̃ , the point◦(1, x ) :=

!
d
1X
1,
xr .
d r=1

is such that (1, x◦ ) ∈ X̃ ∩ ℝd>0 .
Let now M be the ℓ∞ norm of the linear part in the FTRL step (Line 4 of Algorithm 6.1).
Then, a lower bound on the optimal value v ⋆ of objective is obtained by plugging in the point(1, x◦ ) at least⋆v ⩾ −M (1 + ∥X ∥1 ) +dXlog x◦ [r].

(6.10)

r=1

Let now(
m := exp −(2M + d)(1 + ∥X ∥1 ) +dX)log x◦ [r]> 0.

(6.11)

r=1

We will show that any point (λ, y) ∈/ [m, +∞) ∩ X̃ cannot be optimal for the FTRL objective.
Indeed, take a point (λ, y) ∈/ [m, +∞) ∩ X̃ . Then, at least one coordinate of (λ, y) is strictly lessthan m. If λ < m, then the objective value at (λ, y) is at mostM λ + M ∥X ∥1 + log λ +dXlog y[r] ⩽ M (1 + ∥X ∥1 ) + log m +

r=1

d
X

log∥X ∥1

r=1

⩽ M (1 + ∥X ∥1 ) + log m + d∥X ∥1< (M + d)(1 + ∥X ∥1 ) + log m= −M (1 + ∥X ∥1 ) +dXlog x◦ [r](from (6.11))

r=1⩽ v∗ ,(from (6.10))

where the first inequality follows from upper bounding any coordinate of y with ∥X ∥1 , andthe second inequality follows from using the inequality log z ⩽ z, valid for all z ∈ (0, +∞).
Similarly, if y[s] < m for some s ∈ [[d]], then we can upper bound the objective value at (λ, y) as§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
119

M + M ∥X ∥1 + log 1 +dXlog y[r] ⩽ M (1 + ∥X ∥1 ) + log m +

r=1

d
X

log∥X ∥1

r=1

⩽ M (1 + ∥X ∥1 ) + (d − 1)(∥X ∥1 − 1) + log m< (M + d)(1 + ∥X ∥1 ) + log m ⩽ v ∗ .
So, in either case, we see that no optimal point can have any coordinate strictly less than m.
Consequently, the maximizer of the FTRL step lies in the set S := [m, +∞)d+1 ∩ X̃ . Since both[m, +∞)d+1 and X̃ are closed, and since X̃ is bounded by hypothesis, the set S is compact.
Furthermore, note that S is nonempty, as (1, x◦ ) ∈ S, as for any s ∈ [[d]]log m = −(2M + d)(1 + ∥X ∥1 ) +dXlog x◦ [r]r=1⩽ −(2M + d)(1 + ∥X ∥1 ) + log x◦ [s] + (d − 1) log∥X ∥1⩽ −(2M + d)(1 + ∥X ∥1 ) + log x◦ [s] + (d − 1)(∥X ∥1 − 1)
⩽ log x◦ [s],implying that (1, x◦ ) ∈ [m, +∞)d+1 . Since S is compact and nonempty and the objectivefunction is continuous, the optimization problem attains an optimal solution on S by virtue ofWeierstrass’ theorem.

6.A.2

Proof of Proposition 6.2

For notational convenience, we define the log-regularizer ψ : X̃ → ℝ⩾0 as
d+1

ψ(x) := −1X
log x̃[r],η r=1

and its induced Bregman divergenceDψ (x ∥ ã) :=


d+1 
1X
x̃[r]
h
,
η r=1
ã[r]

where h(a) = a − 1 − ln(a).

Moreover, we definex̃(t) = arg max −Ft (x) = arg min Ft (x),
x∈X̃

x∈X̃

E
D (t)
where Ft (x) = − Ũ + m̃(t) , x + ψ(x).

(6.12)

120§6.A. APPENDIX: PROOF DETAILS

We note that Ft is a convex function for each t and x̃

(t)is exactly equal to

λ(t)
y (t)

!
computed by

Algorithm 6.1. Further, we define an auxiliary sequence {ã(t) }t=1,2,... defined as follows.
D (t) E
where Gt (x) = − Ũ , x + ψ(x).

ã(t) = arg max −Gt (x) = arg min Gt (x),x∈X̃(6.13)x∈X̃

Similarly, Gt is a convex function for each t. We also recall the primal and dual norm notation:∥ã∥t =

2
d+1
X
ã[r]
r=1

x̃(t) [r],∥ã∥∗,t =

d+1
X

2
x̃(t) [r]ã[r] .

r=1

Finally, for a (d +√ 1) × (d + 1) positive definite matrix M, we use ∥ã∥M to denote the induced
quadratic norm ã⊤ Mã.
In the proof of Proposition 6.2, we will make use of the following lemmas.
Lemma 6.2. The update rule (6.12) ensures the following for any x̃ ∈ X̃ :
T D
T D
E
E
X
X
x̃ − x̃(t) , ũ(t) ⩽ ψ(x̃) − ψ(x̃(1) ) +ã(t+1) − x̃(t) , ũ(t) − m̃(t)
t=1

t=1

−

T 
X




Dψ x̃(t) ã(t) + Dψ ã(t+1) x̃(t) .


t=1

Proof. First note that for any convex function F : X̃ → ℝ and a minimizer x⋆ , we have for any
x ∈ X̃ :F (x⋆ ) = F (x) − ⟨∇F (x⋆ ), x − x⋆ ⟩ − DF (x ∥ x⋆ ) ⩽ F (x) − DF (x ∥ x⋆ ),where DF is the Bregman Divergence induced by F and the inequality is by the first-order
optimality. Using this fact and the optimality of ã(t) , we have
Gt (ã(t) ) ⩽ Gt (x̃(t) ) − Dψ x̃(t) ã(t)DE= Ft (x̃(t) ) + x̃(t) , m̃(t) − Dψ x̃(t) ã(t)Similarly, using the optimality of x̃(t) , we have
Ft (x̃(t) ) ⩽ Ft (ã(t+1) ) − Dψ ã(t+1) x̃(t)§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
121


E

D
= Gt+1 (ã(t+1) ) + ã(t+1) , ũ(t) − m̃(t) − Dψ ã(t+1) x̃(t)Combining the inequalities and summing over t, we have
G1 (ã(1) ) ⩽ GT +1 (ã(T +1) ) +

T D
X

E
E D
x̃(t) , ũ(t) + ã(t+1) − x̃(t) , ũ(t) − m̃(t)

t=1

+

T 
X





−Dψ x̃(t) ã(t) − Dψ ã(t+1) x̃(t) .

t=1

D
E
(T +1)
Observe that G1 (ã(1) ) = ψ(x̃(1) ) and GT +1 (ã(T +1) ) ⩽ − x̃, Ũ+ ψ(x̃). Rearranging
then proves the lemma.

1
Lemma 6.3. If η ⩽ 50
and ∥u(t) ∥∞ ∥x∥1 , ∥m(t) ∥∞ ∥x∥1 ⩽ 1 for all x ∈ X , then we haveã(t+1) − x̃(t)tx̃(t+1) − x̃(t)t⩽ 5η ũ(t) − m̃(t)⩽ 5η 2ũ(t) − m̃(t)∗,t√⩽ 10 2η ⩽ 15η,∗,t

(6.14)√⩽ 15 2η ⩽ 22η.

(6.15)

Proof. The second part of both inequalities is clear by definitions:ũ(t) − m̃(t)

2
∗,t



= λ(t)

Dd E DE2 X2x(t) , u(t) − x(t−1) , m(t)+y (t) [r] u(t) [r] − m(t) [r]r=1d
4 X (t) 2⩽ 4(λ(t) )2 +y [r] ⩽ 8,
∥X ∥21 r=1

where we use x, u(t) , x, m(t) ⩽ 1 for any x by the hypotheses (using Cauchy-Schwarz),and similarly |u(τ ) [r]|, |m(τ ) [r]| ⩽ ∥X1∥1 for any time τ and any coordinate r by the assumption.
Analogously,
2ũ(t) − m̃(t)

2
∗,t

d 

 D
E D
E2 X2= λ(t) 2 x(t) , u(t) − x(t−1) , u(t−1)+
y (t) [r] 2u(t) [r] − u(t−1) [r]
r=1

122

§6.A. APPENDIX: PROOF DETAILS⩽ 9(λ(t) )2 +d
9 X (t) 2
y [r] ⩽ 18.
∥X ∥21 r=1


To prove the first inequality in Eq. (6.14), let Et =x : x − x̃(t)t⩽ 5η ũ(t) − m̃(t)


.
∗,t

Noticing that ã(t+1) is the minimizer of the convex function Gt+1 , to show ã(t+1) ∈ Et , itsuffices to show that for all x̃ on the boundary of Et , we have Gt+1 (x̃) ⩾ Gt+1 (x̃(t) ). Indeed,using Taylor’s theorem, for any such x̃, there is a point ξ on the line segment between x̃(t) andx̃ such thatDE 1
2x̃ − x̃(t) 2
Gt+1 (x̃) = Gt+1 (x̃(t) ) + ∇Gt+1 (x̃(t) ), x̃ − x̃(t) +2∇ Gt+1 (ξ)
D
E D
E 1
2
= Gt+1 (x̃(t) ) − ũ(t) − m̃(t) , x̃ − x̃(t) + ∇Ft (x̃(t) ), x̃− x̃(t) +x̃ − x̃(t)2∇2 ψ(ξ)E 1
D2⩾ Gt+1 (x̃(t) ) − ũ(t) − m̃(t) , x̃ − x̃(t) +x̃ − x̃(t)2∇2 ψ(ξ)(by the optimality of x̃(t) )⩾ Gt+1 (x̃(t) ) − ũ(t) − m̃(t)⩾ Gt+1 (x̃(t) ) − ũ(t) − m̃(t)

∗,t

∗,t

x̃ − x̃(t)+tx̃ − x̃(t)

+
t

2
5
= Gt+1 (x̃(t) ) + η ũ(t) − m̃(t)
9
∗,t

2
1
x̃ − x̃(t).
2∇2 ψ(ξ)(by Hölder’s inequality)22x̃ − x̃(t)
9η
t

(⋆)

( x̃ − x̃(t)= 5η ũ(t) − m̃(t)
t

)
∗,t

⩾ Gt+1 (x̃(t) ).
1
Here, the inequality (⋆) holds because Lemma 6.4 (together with the condition η ⩽ 50
)
3 (t)
3 (t)
1 (t)
1 (t)
shows 2 x̃ [i] ⩽ x̃[i] ⩽ 2 x̃ [i], which implies 2 x̃ [i] ⩽ ξ[i] ⩽ 2 x̃ [i] as well, and thus
∇2 ψ(ξ) ≽ 49 ∇2 ψ(x̃(t) ). This finishes the proof for Eq. (6.14). The first inequality of Eq. (6.15)
can be proven in the same manner.

A restatement of the second inequality of Lemma 6.3 was given earlier in this chapter, in
Proposition 6.3.
1
Proposition 6.3 (Restated). For any time t ∈ ℕ⩾0 and learning rate η ⩽ 50
, if ∥u(t) ∥∞ ∥x∥1 , ∥m(t) ∥∞ ∥x∥1 ⩽1 for all x ∈ X ,!
!
λ(t+1)λ(t)−⩽ 22η.
y (t+1)y (t)t§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
123

Lemma 6.4. If x̃ satisfies ∥x̃ − x̃(t) ∥t ⩽ 21 , then 12 x̃(t) [i] ⩽ x̃[i] ⩽ 23 x̃(t) [i] for every coordinatei.

Proof. By definition, ∥x̃ − x̃(t) ∥t ⩽ 12 implies for any i,|x̃[i]−x̃(t) [i]|x̃(t) [i]⩽ 12 , and thus 12 x̃(t) [i] ⩽x̃[i] ⩽ 23 x̃(t) [i].

1Lemma 6.5. If η ⩽ 50
, then we have
T −1




2
1 X (t+1)x̃− x̃(t) .
Dψ x̃(t) ã(t) + Dψ ã(t+1) x̃(t)⩾27η t=1
t
t=1

T 
X

Proof. Recall h(a) = a − 1 − ln(a) and Dψ (x ∥ ã) = η1
T X

Pd+1  x̃[i] 
i=1 h ã[i] . We proceed as





Dψ x̃(t) ã(t) + Dψ ã(t+1) x̃(t)

t=1

⩾

T
−1
X





Dψ x̃(t+1) ã(t+1) + Dψ ã(t+1) x̃(t)t=1
T −1 d+1x̃(t+1) [i]1 XXh=η t=1 i=1ã(t+1) [i]!

ã(t+1) [i]+hx̃(t) [i]!!

T −1 d+1

1 X X (x̃(t+1) [i] − ã(t+1) [i])2(ã(t+1) [i] − x̃(t) [i])2⩾+6η t=1 i=1(ã(t+1) [i])2(x̃(t) [i])2!

2(h(y) ⩾ (y−1)
for y ∈ [ 13 , 3])
6!
T −1 d+1
2 X X (x̃(t+1) [i] − ã(t+1) [i])2(ã(t+1) [i] − x̃(t) [i])2⩾+27η t=1 i=1(x̃(t) [i])2(x̃(t) [i])2!
T −1 d+1
T −1
2
1 X X (x̃(t+1) [i] − x̃(t) [i])2
1 X (t+1)⩾x̃− x̃(t) .
=(t)
27η t=1 i=127η t=1t(x̃ [i])2
Here, the third and the fourth inequalities hold because by Lemma 6.3 and Lemma 6.4, we
(t+1)

(t+1)

(t+1)

[i]
have 21 ⩽ ãx̃(t) [i][i] ⩽ 32 and 12 ⩽ x̃x̃(t) [i][i] ⩽ 32 , and thus 13 ⩽ x̃
⩽ 3.
ã(t+1) [i]We are now ready to establish Proposition 6.2.

124

§6.A. APPENDIX: PROOF DETAILS(T )Proposition 6.2 (Restated; Regret of FTRL in local norms). Let R̃eg be the regret cumulated
up to time T by the internal FTRL algorithm. If ∥u(t) ∥∞ ∥x∥1 ⩽ 1 at all times t ∈ [[T ]], then for1
any time horizon T ∈ ℕ⩾0 and learning rate η ⩽ 50
,
R̃eg

(T )

T −1
T
X
2
(d+1) log T
1 X
⩽
+ 5η
−
ũ(t) − m̃(t)η27η t=1
∗,t
t=1

!
!2
λ(t+1)λ(t).
− (t)y (t+1)yt

Proof. For any comparator x̃ ∈ X̃ , define x̃′ = T T−1 · x̃ + T1 · x̃(1) ∈ X̃ , where we recallx̃(1) = arg minx∈X̃ F1 (x) = arg minx∈X̃ ψ(x). Then, we have
T D
T D
T D
E X
E X
E
X
(t)
′
(t)
x̃ − x̃ , ũ =x̃ − x̃ , ũ+x̃′ − x̃(t) , ũ(t)
t=1

t=1

=

t=1

T D
X

T D
E X
E
1
x̃ − x̃(1) , ũ(t) +x̃′ − x̃(t) , ũ(t)
T t=1
t=1

⩽4+

T D
E
X
x̃′ − x̃(t) , ũ(t) ,t=1

where the last inequality follows from Cauchy-Schwarz together with the assumption that∥u(t) ∥∞ ⩽ ∥X1∥1 .
PT
Now, from Lemma 6.2, the last term t=1 ⟨x̃′ − x̃(t) , ũ(t) ⟩ (cumulative regret against x̃′ ) isbounded by
T D
T D
E
E
X
X
ã(t+1) − x̃(t) , ũ(t) − m̃(t)x̃′ − x̃(t) , ũ(t) ⩽ ψ(x̃′ ) − ψ(x̃(1) ) +
t=1

t=1

−

T 
X





Dψ x̃(t) ã(t) + Dψ ã(t+1) x̃(t) .

t=1

For the term ψ(x̃′ ) − ψ(x̃(1) ), a direct calculation using definitions showsd+1

ψ(x̃′ ) − ψ(x̃(1) ) =x̃(1) [i]
d+1
1X
log ′
⩽
log T.
η i=1x̃ [i]
ηFor the other terms, we apply Lemma 6.3 and Lemma 6.5, which completes the proof.

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
125

6.A.3

Proof of Corollary 6.1

Next, we establish an RVU bound in the original (unlifted) space, namely Corollary 6.1. To thisend, we first proceed with the proof of Lemma 6.1, which boils down to the following simple
claim.
Lemma 6.6. Let (λ, y), (λ′ , y ′ ) ∈ X̃ ∩ ℝd+1
>0 be arbitrary points such that
λ′
y′

!
−

λ
y

!
⩽

1
.
2

−

λ
y

(λ,y)

Then,
y y′
− ′
λ
λ

⩽ 4∥X ∥1 ·
1

λ′
y′

!

!
.
(λ,y)

Proof. Let µ be defined asµ := max

By definition,

λ′
λ −1

λ′
y ′ [r]− 1 , max−1
λr∈[[d]] y[r].

(6.16)⩽ µ, which in turn implies that
(1 − µ)λ ⩽ λ′ ⩽ (1 + µ)λ.

(6.17)(1 − µ)y[r] ⩽ y ′ [r] ⩽ (1 + µ)y[r].

(6.18)

Similarly, for any r ∈ [[d]],As a result, combining (6.17) and (6.18) we get that for any r ∈ [[d]],y ′ [r] y[r]
⩽
−
λ′
λ




1+µ
y[r]
y[r]
−1
⩽ 4µ
= 4µx[r],1−µλλsince µ ⩽ 12 . Similarly, by (6.17) and (6.18),
y[r] y ′ [r]
− ′ ⩽
λ
λ
Thus, it follows thaty ′ [r]y[r]λ′ − λ1−1 − µ y[r]
y[r]
⩽ 2µ
= 2µx[r].
1+µλλ⩽ 4µx[r], in turn implying that

126

§6.A. APPENDIX: PROOF DETAILS∥x′ − x∥1 =dXy ′ [r]
r=1

λ′

−

d
X
y[r]
⩽ 4µ
x[r] ⩽ 4µ∥X ∥1 .
λr=1

(6.19)

Moreover, by definition of (6.16),

2

(µ) ⩽

λ′
y′

!
−

λ
y

! 2
.
t

Finally, combining this bound with (6.19) concludes the proof.

1, if ∥u(t) ∥∞ ∥x∥1 ⩽ 1,
Lemma 6.1 (Restated). For any time t ∈ ℕ⩾0 and learning rate η ⩽ 50

∥x

(t+1)

λ(t+1)y (t+1)(t)− x ∥1 ⩽ 4∥X ∥1

!
−

λ(t)
y (t)

!
.
t

1
Proof. Since η ⩽ 50
by assumption, we havex(t+1) − x(t)t

⩽ 22η <

1.
2

Hence, we are in the domain of applicability of Lemma 6.6, which immediately yields thestatement.

Corollary 6.1 (Restated; RVU bound in the original (unlifted) space). Fix any time T ∈ ℕ⩾0 ,1and suppose that ∥u(t) ∥∞ ⩽ B for any t ∈ [[T ]]. If η ⩽ 256B∥X
∥1 ,
R̃eg

(T )

T
−1
X
(d + 1) log T2⩽ 6B∥X ∥1 ++ 16η∥X ∥1∥u(t) − m(t) ∥2∞
η
t=1

−

T
−1
X
1
∥x(t+1) − x(t) ∥21 .
512η∥X ∥21 t=1

Proof. At first, assume that ∥u(t) ∥∞ ⩽ 1/∥X ∥1 . By definition of the induced dual local normin (6.2),d2X∥ũ(t) − m̃(t) ∥2∗,t ⩽ ⟨x(t), u(t) ⟩ − ⟨x(t) , m(t) ⟩ (λ(t) )2 +y[r]2 (u(t) [r] − m(t) [r])2
r=1

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
127

⩽ ⟨x(t) , u(t) − m(t) ⟩2 +dXx[r]2 (u(t) [r] − m(t) [r])2

r=1⩽ 2∥X ∥21 · ∥u(t) − m(t) ∥2∞ ,(6.20)

Combining with Proposition 6.2 and Lemma 6.1, we get that R̃eg

6+

(T )

is upper bounded by

T
T
−1
X
X
(d + 1) log T1+ 16η∥X ∥21∥u(t) − m(t) ∥2∞ −∥x(t+1) − x(t) ∥21 .
2
η
512η∥X∥1
t=1
t=1

Finally, we relax the assumption that ∥u(t) ∥∞ , ∥m(t) ∥∞ ⩽ 1/∥X ∥1 . In that case, one
can reduce to the above analysis by first rescaling all utilities and predictions by the factor
1/(B∥X ∥1 )—which in turn is equivalent to rescaling the learning rate η by 1/(B∥X ∥1 ). We
then need to correct for the fact that the norm of the difference of utilities gets rescaled by a(T )factor 1/(B∥X ∥1 )2 , and that the regret R̃eg
with respect to the original utilities is a factor
B∥X ∥1 larger than the regret measured on the rescaled utilities. Taking these considerations
into account leads to the statement.

6.A.4

Proof of Theorems 6.2 and 6.3

Finally, we are ready to establish Theorem 6.3. To this end, the main ingredient is the bound on
the second-order path lengths predicted by Theorem 6.2, which is recalled below.
Theorem 6.2 (Restated). If all players follow LRL-OFTRL with learning rateη ⩽ min
1
1,,
256B∥X ∥1 128nL∥X ∥21

where ∥X ∥1 := maxi∈[[n]] ∥Xi ∥1 , then
n T
−1
X
X

(t+1)

∥xi

(t)

− xi ∥21 ⩽ 6144nηB∥X ∥31 + 1024n(d + 1)∥X ∥21 log T.

(6.3)

i=1 t=1

Proof. For any player i ∈ [[n]],



(t+1)

∥ui

(t)

− ui ∥∞

2

⩽ L

n
X

2
(t+1)

∥xj

(t)

− xj ∥1  ⩽ L2 n

j=1

n
X
(t+1)
(t)
∥xj
− xj ∥21 ,
j=1

(T )

by Jensen’s inequality. Hence, by Corollary 6.1 the regret Regi

of each player i ∈ [[n]] can be128§6.A. APPENDIX: PROOF DETAILS

upper bounded by

6B∥X ∥1 +

n T
−1
X
X
(d + 1) log T
(t+1)
(t)
+ 16η∥X ∥21 L2 n∥xj
− xj ∥21η
j=1 t=1

−

T
−1
X
1
(t+1)
(t)
∥x
− xi ∥21 ,512η∥X ∥21 t=1 i

Summing over all players i ∈ [[n]], we have that
n
X

(T )

R̃egi

⩽ 6nB∥X ∥1 + n

i=1(d + 1) log Tηn X16η∥X ∥21 L2 n2 −+i=1

⩽ 6nB∥X ∥1 + n

1512η∥X ∥21

 TX
−1

(t+1)

∥xi

(t)

− xi ∥21

t=1

−1
n T
X
X

(d + 1) log T
1
(t+1)
(t)
∥x
− xi ∥21 ,−η
1024η∥X ∥21 i=1 t=1 i1since η ⩽ 256nL∥X. Finally, the theorem follows since∥21
directly from Theorem 6.1.

(T )⩾ 0, which in turn follows
i=1 R̃egi

Pn

Theorem 6.3 (Restated). If all players use LRL-OFTRL within the COLS (that is, withm(t) = u(t−1) ; cf. Section 3.2.1) with learning rate11
η = min,,
256B∥X ∥1 128nL∥X ∥21(T )then for any T ∈ ℕ⩾0 the regret Regi
(T )

Regi

of each player i ∈ [[n]] can be bounded as⩽ 12B∥X ∥1 + 256(d + 1) max nL∥X ∥21 , 2B∥X ∥1 log T.

(6.4)

Furthermore, the algorithm can be adaptive so that if player i is instead facing adversarial√(T )utilities, then Regi = OT ( T ).
Proof. First of all, by Assumption 6.1 we have that for any player i ∈ [[n]],

(t+1)
(t)
∥ui
− ui ∥2∞ ⩽ L

2
n
n
X
X
(t+1)
(t) 
(t+1)
(t)
2
∥xj
− xj ∥1⩽L n∥xj− xj ∥21 .
j=1

j=1

§6. STRONGLY PREDICTIVE LEARNING DYNAMICS, AND OT (log T /T ) CONVERGENCE IN SELF
PLAY
129

Hence, summing over all t,
T
−1
X

(t+1)

∥ui

(t)

− ui ∥2∞ ⩽ L2 n

t=1

T
−1 X
n
X

(t+1)

∥xj

(t)

− xj ∥21

t=1 j=1⩽ 6144n2 L2 ηB∥X ∥31 + 1024n2 L2 (d + 1)∥X ∥21 log T,
1,where the last bound uses Theorem 6.2. As a result, from Corollary 6.1, if η = 128nL∥X
∥2
1

(T )

R̃egi

⩽ 6B∥X ∥1 +T
−1X
(d + 1) log T
(t+1)
(t)
+ 16η∥X ∥21∥ui− ui ∥2∞η
t=1⩽ 12B∥X ∥1 + 256(d + 1)nL∥X ∥21 log T.
(T )

(T )

(T )

Thus, the bound on Regi follows directly since Regi ⩽ R̃egi by Theorem 6.1. The case1
where η = 256B∥X∥1 is analogous.
Next, let us focus on the adversarial bound. Each player can simply check whether there
exists a time t ∈ [[T ]] such that
(t−1)

X

(τ +1)∥ui(τ )− ui ∥2∞ > 6144n2 L2 ηB∥X ∥31 + 1024n2 L2 (d + 1)∥X ∥21 log t.

(6.21)

τ =1

In particular, we know from Theorem 6.2 that when all players follow the prescribed protocol (6.21) will never by satisfied. On the other hand, if there exists time t so that (6.21)
holds, then it suffices to switch to any no-regret learning algorithm tuned to face adversarial
utilities.

130§6.A. APPENDIX: PROOF DETAILSChapter 7

State-of-the-art regret dependence on
game size via kernelization
7.1

Contributions and related work

In this chapter we establish a new technique for constructing no-external-regret dynamics forimperfect-information extensive-form games, which leads to the current state-of-the-art regretbounds in terms of dependence from the game dimensions, while at the same time enjoying
polylogarithmic (albeit with a worse exponent than the LRL-OFTRL algorithm we presented
in Chapter 6) regret in self-play, and polynomial-time iterations. However, in addition to thebenefits above, I believe the technique is remarkable in that (i) it surprisingly shows that along-held popular wisdom in the literature of imperfect-information extensive-form games is
inaccurate, and (ii) reduces the gap between learning in sequential and nonsequential games, byshowing that in some cases the former (which is substantially harder both conceptually and in acomplexity-theoretic sense in general) can be efficiently reduced to the latter.
In order to understand our technique, it is beneficial to introduce the concept of normal-form
equivalent of the tree-form decision problem faced by a player. In short, because the polytope of(sequence-form) strategies is the convex polytope spanned by the deterministic strategies, pickinga (sequence-form) strategy for the game is equivalent to picking a convex combination of vertices.
Such a process is no longer tree-form, but is rather a single decision with as many “actions” asdeterministic strategies for the player. By doing away with the tree-form structure in favor of asingle decision node, learning in the normal-form equivalent of a TFDP can then be achieved byusing any learning algorithm for nonsequential games, opening the way to reaping the benefitsof techniques for nonsequential games in imperfect-information extensive-form games too. Of
course, the catch of the above approach is that it comes at the cost of an exponential blowupof the strategy space, as the flattened (normal-form equivalent) decision problem typically has

132§7.1. CONTRIBUTIONS AND RELATED WORK

exponentially many actions—as such is the number of deterministic strategies in a TFDP. For
this reason, the normal-form representation was viewed as impractical, historically leading thecommunities working on learning in normal-form and imperfect-information extensive-form
games to follow separate tracks, with the latter community often having to catch up with advances(e.g., last-iterate convergence and predictive regret bounds) from the former, larger community.
We contradict popular belief and show that it is possible to work with the normal-form
equivalent of a TFDP efficiently: we provide a kernel-based reduction that allows us to simulate
predictive multiplicative weights update OMWU algorithm—arguably the premier learning algorithm for nonsequential games; cf. Section 3.3.1—on the normal-form representation, using only
linear (in the TFDP size) time per iteration. Our algorithm, Kernelized OMWU (KOMWU), achieves all
the guarantees provided by the various normal-form results mentioned previously, as well as any
future results on OMWU for nonsequential games.
As an unexpected byproduct, KOMWU obtains new state-of-the-art regret bounds among all
online learning algorithms for TFDPs (see also Table 7.1); we improve the dependence on
the maximum ℓ1 norm ∥Q∥1 over the sequence-form polytope Q from ∥Q∥21 to ∥Q∥1 (for thepnon-optimistic version we improve it from ∥Q∥1 to ∥Q∥1 ).
Algorithm+

CFR (regret matching / regret matching ) Chapter 4
CFR (MWU)
Chapter 4
FTRL/ OMD (dilated entropy)
Chapter 5
FTRL/ OMD (dilatable global entropy)
Chapter 5
Kernelized MWU(this chapter)
Predictive CFR
Chapter 4
Predictive FTRL/ OMD (dilated entropy)
Chapter 5
Predictive FTRL/ OMD (dilatable gl. ent.)
Chapter 5
LRL-OFTRL ‡Chapter 6
Kernelized OMWU
(this chapter)

Per-player regret boundLast-iter. conv.†√
1/2
noO(√A ∥Q∥1 T )noO(√log A ∥Q∥1 T 1/2 )noO(√log A 2D/2 ∥Q∥1 T 1/2 )1/2
no)O( log A ∥Q∥
p 1 T 1/2√
noO( log A ∥Q∥1 T )√noOT ( T )√yes∗O( m log(A) 2D ∥Q∥21 T 1/4 )√noO( m log(A) ∥Q∥21 T 1/4 )O(m poly(A) ∥Q∥21 log T )noyes
O(m log(A) ∥Q∥1 log4 (T ))Table 7.1: Properties of various no-regret algorithms for imperfect-information extensive-form games.
All algorithms take linear time to perform an iteration. The first set of rows are for non-predictive
algorithms. The second set of rows are for predictive algorithms. The regret bounds are perplayer and apply to multiplayer general-sum games. They depend on the maximum numberof actions A available at any decision node, the maximum ℓ1 norm ∥Q∥1 = maxq∈Q ∥q∥1 overthe player’s sequence-form strategy polytope Q, the depth D of the decision polytope, and thenumber of players m. Optimistic algorithms have better asymptotic regret, but worse dependence
on the game constants m, A, and ∥Q∥1 . Note that our algorithms achieve better dependence on∥Q∥1 compared to all existing algorithms. † Last-iterate convergence results are for two-player
zero-sum games, and some results rely on the assumption of a unique Nash equilibrium—see
Section 7.5 for details. ‡ We remark that LRL-OFTRL enjoys polynomial-time iterations (ignoring a
log log T dependence), but not linear-time iterations unlike the other methods in the table. ∗ See
C.-W. Lee, Kroer, and Luo (2021).

Due to the connection between regret minimization and convergence to Nash equilibrium,§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

133

this also improves the state-of-the-art bounds for converging to a Nash equilibrium at either a√rate of 1/ T or 1/T by the same factor. Moreover, KOMWU achieves last-iterate convergence, andas such it is the first algorithm to achieve linear-rate last-iterate convergence with a learning
rate that does not become impractically-small as the game grows large (albeit under a restrictive
uniqueness assumption).
More generally, we remark that KOMWU can simulate OMWU in general 0/1-polyhedral sets (ofwhich the decision sets for TFDPs are a special case): a decision set Ω ⊆ ℝd which is convex andpolyhedral, and whose vertices are all contained in {0, 1}d . KOMWU reduces the problem of running
OMWU on the vertices of the polyhedral set to d + 1 evaluations of what we call the 0/1-polyhedral
kernel. Thus, given an efficient algorithm for performing these kernel evaluations, KOMWU enables
one to get all the benefits of running MWU or OMWU on the simplex of vertices, while retaining the
crucial property that each iteration of OMWU can be performed efficiently.
Regret minimization over 0/1 polyhedral sets, of which the sequence-form strategy polytope
is an example, is closely related to online combinatorial optimization problems (Audibert, Bubeck,and Lugosi, 2014), where the decision maker (randomly) selects a 0/1 vertex in each round insteadof a point in the convex hull of the set of vertices, and the regret is measured in expectation.
We review the approaches most closely related to the use of MWU here. One approach similar to
our KOMWU is to perform MWU over vertices (e.g., Cesa-Bianchi and Lugosi (2012)); the remaining
problem is whether there is an efficient way to maintain and sample from the weights. Suchefficient implementations have been shown in many instances such as paths (Takimoto and
Warmuth, 2003), spanning trees (Koo, Globerson, Carreras Pérez, and Collins, 2007), and m-set(Warmuth and Kuzmin, 2008). The work by Takimoto and Warmuth (2003) is the closest to thekernel approach we adopted, where they show how to produce MWU iterates for paths in directed
graphs. The kernelized method described in this chapter can be seen as a significant extension of
their approach to general 0/1 polyhedral games, unifying many of the previous results listed
above. This unification not only results in important applications to imperfect-information
extensive-form games, but also leads to improvement to previously studied problems such as
n-sets.

7.2

A natural reduction: Running OMWU on the vertices of thestrategy set

Since the set of sequence-form strategies Q in a TFDP is a convex polytope, the decision problemof picking a sequence-form strategy x(t) ∈ Q can be equivalently thought of as the decision
problem of picking a convex combination λ(t) of vertices of Q—that is, of deterministic strategiesπ ∈ Π (Definition 2.5).
This intuition is correct: as we show next, a learning algorithm R for Q ⊆ ℝΣ can be§7.2. A NATURAL REDUCTION: RUNNING OMWU ON THE VERTICES OF THE STRATEGY SET

134

Tree-form decision process
(t)

m
u(t)

R

Algorithm 7.1: Vertex OMWU
Data: Learning rates η (t) > 0

x(t) ∈ Q
12m̃(t) = V⊤ m(t)
ũ

(t)

⊤

(t)

=V u

x(t) = Vλ(t)R̃

w(t) ← u(t−1) − m(t−1) + m(t)4for π ∈ Π doλ(t) ∈ ∆Π

Nonsequential decision processfunction NextStrategy(m(t) ∈ ℝΣ )non-predictive variant]

3

5

m̃(t)
ũ(t)

1
u(0) , m(0) ← 0 ∈ ℝΣ ; λ(0) ← |Π|
1 ∈ ∆Π

6[▷ Initialization][▷ Set m(t) = 0 for[▷ Run the OMWU update on λ using A = Π]λ(t−1) [π] · exp{η (t) ⟨w(t) , π⟩}(7.1)
λ(t) [π] := P
(t−1)
[π ′ ] · exp{η (t) ⟨w(t), π ′ ⟩}π ′ ∈Π λ[▷ Compute new convex combination of vertices]X
Q ∋ x(t) :=λ(t) [π] · π

(7.2)

π∈Π

Figure 7.1: Overview of reduction template from Q to∆Π . The matrix V has thevertices Π of Q as columns.

7

return x(t)function ObserveUtility(u(t) ∈ ℝΣ )
9
t←t+1
8

constructed from any learning algorithm R̃ for the set of vertices Π. Indeed, consider the following
conceptual template:• Each distribution λ(t) ∈ ∆Π output by R̃ defines the convex combination of vertices of Q and isused to construct the sequence-form strategy
Q ∋ x(t) :=X

λ(t) [π] π.

π∈Π• Each prediction m(t) ∈ ℝΣ and utility vector u(t) fed into R is used to construct a predictionm̃ and utility vector ũ ∈ ℝΠ according tom̃(t) [π] := ⟨m(t) , π⟩,ũ(t) [π] := ⟨u(t) , π⟩∀π ∈ Π.

By letting V denote the matrix whose columns are the deterministic strategies π ∈ Π, we
can rewrite the above template more concisely by saying that each output λ(t) ∈ ∆Π of R̃ istransformed into the output x(t) := Vλ(t) ∈ Q of R, and each prediction and utility vectorm(t) , u(t) ∈ ℝΣ of R are transformed into a prediction and utility vector m̃(t) := V⊤ m(t) andũ(t) := V⊤ u(t) for R̃, respectively (see also Figure 7.1). The correctness of the above template isgiven by the observation that, since ũ(t) := V⊤ u(t) and x(t) := Vλ(t) by construction, then the§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

135

following equality of regrets holds.
P(T ):= Tt=1 ⟨ũ(t) , λ(t) ⟩ cumulated by R̃ is equal, at all times T , toLemma 7.1. The regret R̃eg
PT
the regret Reg(T ) := t=1 ⟨u(t) , x(t) ⟩.

In this chapter we are particularly interested in the algorithm obtained by using the above
construction for the specific choice of OMWU (Section 3.3.1) as the algorithm R̃. We coin Vertex OMWU
the resulting learning algorithm R. The algorithm is summarized in pseudocode in Algorithm 7.1.

7.3

Kernelized multiplicative weights

Kernelized OMWU (KOMWU) gives a way of efficiently simulating the Vertex OMWU algorithm given in
Algorithm 7.1 and described in Section 7.2.[7.a]We start by defining the sequence-form feature map and associated kernel in Section 7.3.1, andmove on to describe how the kernel enables efficient simulation of Vertex OMWU in Section 7.3.2.

7.3.1

The sequence-form kernel

As the name suggests, at its heart KOMWU uses a kernel trick to simulate the steps of Vertex
OMWU without actually operating on the exponentially large probability simplex of deterministic
strategies. Let σ ∈ π be a shorthand notation for the set of sequences σ ∈ Σ such that π[σ] = 1,
that is, those sequences that are supported by the deterministic strategy π.
Definition 7.1 (Sequence-form feature map). The sequence-form feature map ϕQ : ℝΣ → ℝΠ is
the function such thatϕQ (x)[π] :=

Y

x[σ]

∀ x ∈ ℝΣ , π ∈ Π.

(7.3)

σ∈π

Definition 7.2. The sequence-form kernel KQ is the function KQ : ℝΣ × ℝΣ → ℝ,
KQ (x, y) := ⟨ϕQ (x), ϕQ (y)⟩ =X Yx[σ] y[σ].

(7.4)π∈Π k∈π[7.a] More generally, the idea underlying KOMWU applies to on polyhedral decision sets whose vertices have 0/1 integer
coordinates, and leads to an efficient implementation of the Vertex MWU algorithm for such a decision set as long as a
specific polyhedral kernel function can be evaluated efficiently. The reader interested in the more general result is welcome
to read the section assuming that Q ⊆ ℝd is an arbitrary polytope with (possibly exponentially many) 0/1 integral
vertices Π := {π 1 , . . . , π |Π| } ⊆ {0, 1}d .

136§7.3. KERNELIZED MULTIPLICATIVE WEIGHTS

We remark that the sequence-form feature map and kernel are defined over the entirety ofℝ , and therefore take as input generic points x, y ∈ ℝΣ which might or might not be valid
sequence-form strategies.
Σ

7.3.2

Using the kernel to simulate the Vertex OMWU algorithm

We show that Vertex OMWU can be simulated using d + 1 evaluation of the kernel KQ at every
iteration. The key observation is summarized in the next theorem, which shows that the iterates
λ(t) produced by Vertex OMWU are highly structured, in the sense that they are always proportional
to the feature mapping ϕQ (b(t) ) for some b(t) ∈ ℝΣ .
Theorem 7.1. Consider the Vertex OMWU algorithm (Algorithm 7.2). At all times t ∈ ℕ⩾0 , thevector b(t) ∈ ℝΣ defined as
(
)
t
X
(t)
(τ )
(τ )
b [σ] := exp −η w [σ](7.5)
τ =1

for all k = 1, . . . , d, is such that
λ(t) =ϕQ (b(t) )KQ (b(t) , 1)

.

(7.6)

Proof. By induction.
• At time t = 0, the vector b(0) is b(0) = 1 ∈ ℝΣ . By definition of the feature map (7.3),
ϕQ (1) = 1 ∈ ℝΠ . So,
KQ (b(0) , 1) =X

1 = |Π|π∈Π
1and hence the right-hand side of (7.6) is |Π|1, which matches λ(0) produced by Vertex OMWU,
as we wanted to show.
• Assume the statement holds up to some time t − 1 ⩾ 0. We will show that it holds at time t
as well. Since π has integral 0/1 coordinates, we can write(
exp{−η

(t)

⟨w

(t)

, π⟩} = exp −η

)
(t)

X

w

(t)

[σ]

σ∈π

=

Y
σ∈π

exp{−η (t) w(t) [σ]}.

(7.7)

§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

137

From the inductive hypothesis and (7.3), for all π ∈ Π,
λ

(t−1)

[π] =

ϕQ (b(t−1) )[π]KQ (b(t−1) , 1)

(t−1)
[σ]
σ∈π b
.
(t−1)
KQ (b
, 1)

Q
=

(7.8)

Plugging (7.7) and (7.8) into (7.1), we complete the inductive step
Q

(t)

σ∈π b

λ [π] = P

π∈Π

Q

(t−1)

σ∈π b

[σ] exp{−η (t) w(t) [σ]}(t−1)[σ] exp{−η (t) w(t) [σ]}=ϕQ (b(t) )[π]KQ (b(t) , 1)

for all π ∈ Π, where in the last step we used the fact thatb(t) [σ] = b(t−1) [σ] exp{−η (t) w(t) [σ]}by (7.5).
The structure of λ(t) uncovered by Theorem 7.1 can be leveraged to compute the iterate x(t)produced by Vertex OMWU, i.e., the convex combination of the vertices (7.2), using d + 1 evaluationsof the kernel KQ . We do so by extending an idea of Takimoto and Warmuth (2003, Equation (5.2)).
Theorem 7.2. Let b(t) be as in Theorem 7.1. For each σ ∈ Σ := {σ1 , . . . , σ|Σ| }, let ēσ ∈ ℝΣ bedefined as the indicator vector0 if σ ′ = σēσ [σ ′ ] := 𝟙σ̸=σ′ :=(7.9)
1 if σ ′ ̸= σ.
Then, at all t ⩾ 1, the iterate x(t) ∈ Q produced by Vertex OMWU can be written as
x

(t)

=

1−

KQ (b(t) , ēσ1 )KQ (b(t) , 1),...,1 −KQ (b(t) , ēσ|Σ| )KQ (b(t) , 1)!
.

(7.10)

Proof. The proof crucially relies on the observation that for all σ ∈ Σ, the feature map ϕQ (ēσ )satisfies
ϕQ (ēσ )[π] =Y
σ ′ ∈πēσ [σ ′ ] =Y

𝟙σ′ ̸=σ = 𝟙σ∈π
/ ,∀ π ∈ Π.

σ ′ ∈πUsing the fact that ϕQ (1) = 1, we conclude thatϕQ (1)[π] − ϕQ (ēσ )[π] = 𝟙σ∈π ,∀ σ ∈ Σ.

(7.11)

138§7.4. EFFICIENT EVALUATION OF THE SEQUENCE-FORM KERNEL

Therefore, for all σ ∈ Σ, we obtain(7.2) Xx(t) [σ] =λ(t) [π] · π[σ] =π∈ΠXλ(t) [π] · 𝟙σ∈π =

π∈Π
(t)

=

(t)

KQ (b , 1)
KQ (b(t) , ēσ )KQ (b(t) , 1)

λ(t) [π] · (ϕQ (1)[π] − ϕQ (ēσ )[π])

π∈Π
(t)

⟨ϕQ (b ), ϕQ (1)⟩ − ⟨ϕQ (b ), ϕQ (ēσ )⟩

=1−

X

=

KQ (b(t) , 1) − KQ (b(t) , ēσ )KQ (b(t) , 1),

where the second equality follows from the integrality of π ∈ Π := Q ∩ {0, 1}Σ , the thirdfrom (7.11), the fourth from Theorem 7.1, and the fifth from the definition of KQ (7.4).
Combined, Theorems 7.1 and 7.2 suggest that by keeping track of the vectors b(t) instead ofλ(t) , updating them using Theorem 7.1 and reconstructing the iterates x(t) using Theorem 7.2,
Vertex OMWU can be simulated efficiently. We call the resulting algorithm, given in Algorithm 7.2,
Kernelized OMWU (KOMWU). Similarly, we call Kernelized MWU the non-optimistic version of KOMWU
obtained as the special case in which m(t) = 0 at all t. In light of the preceding discussion, we
have the following.
Theorem 7.3. Kernelized OMWU produces the same iterates x(t) as Vertex OMWU when it receives
the same sequence of predictions m(t) and losses u(t) ∈ ℝΣ . Furthermore, each iteration of
KOMWU runs in time proportional to the time required to compute the |Σ| + 1 kernel evaluations{KQ (b(t) , 1)} ∪ {KQ (b(t) , ēσ ) : σ ∈ Σ}.

7.4

Efficient evaluation of the sequence-form kernel

To complete our construction of KOMWU, we now show that the sequence-form kernel can beevaluated efficiently. The central result of this section, Theorem 7.5, shows that KOMWU can be
implemented with linear-time iterations in the number of sequences Σ.

7.4.1

Worst-case linear complexity for a single evaluation

We start by verifying that the sequence-form kernel can be evaluated in linear time for any pair ofpoints x, y ∈ ℝΣ . To do so, we introduce a partial kernel function Kj : ℝΣ × ℝΣ → ℝ for every
decision node j ∈ J ,Kj : ℝΣ × ℝΣ → ℝ,

Kj (x, y) :=X Yπ∈Π≽j σ∈πx[σ] y[σ].

(7.12)

139§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

Algorithm 7.2: Kernelized OMWU (KOMWU)
Data: Learning rates η (t) > 0
1u(0) , m(0) , s(0) ← 0 ∈ ℝΣ[▷ Initialization]function NextStrategy(m(t) ∈ ℝΣ )[▷ Compute b(t) according to Theorem 7.1]
3
w(t) ← u(t−1) − m(t−1) + m(t)4s(t) ← s(t−1) + η (t) w(t)5for σ ∈ Σ do
6b(t) [σ] ← exp{−s(t) [σ]}[▷ Set m(t) = 0 for non-predictive variant]

2

7
8
9
10
11

[▷ s(t) =P (τ )
ηw(τ ) ][▷ See (7.5)][▷ Produce iterate x(t) according to Theorem 7.2]
x(t) ← 0 ∈ ℝΣα ← KQ (b(t) , 1)for σ ∈ Σ do
KQ (b(t) , ēσ )x(t) [σ] ← 1 −αreturn x(t)[▷ KQ is defined in (7.4)][▷ See (7.10)]

function ObserveUtility(u(t) ∈ ℝΣ )
13
t←t+1
12

Theorem 7.4. For any vectors x, y ∈ ℝΣ , the two following recursive relationships hold:
KQ (x, y) = x[∅] y[∅]Y

Kj (x, y),(7.13)

j∈C∅

and, for all decision nodes j ∈ J ,Kj (x, y) =Xa∈Ajx[ja] y[ja]Y

Kj ′ (x, y).

(7.14)j ′ ∈Cja

In particular, Equations (7.13) and (7.14) give a recursive algorithm to evaluate the polyhedral
kernel KQ associated with the sequence-form strategy space of any player i in an imperfectinformation extensive-form game in linear time in the number of sequences |Σ|.

7.4.2

Batched computation and amortized complexity

Theorem 7.4 shows that the kernel KQ can be evaluated in linear time (in the number of sequences
Σ) at any pair of points (x, y) ∈ ℝΣ × ℝΣ . So, the KOMWU algorithm (Algorithm 7.2) can be trivially140§7.4. EFFICIENT EVALUATION OF THE SEQUENCE-FORM KERNEL

implemented in quadratic O(|Σ|2 ) time per iteration by directly evaluating the |Σ| + 1 kernel
evaluations {KQ (b(t) , 1)} ∪ {KQ (b(t) , ēσ ) : σ ∈ Σ} needed at each iteration, where ēσ ∈ ℝΣ ,defined in (7.9) for the general case, is the vector whose components are ēσ [σ ′ ] := 𝟙σ̸=σ′ for allσ, σ ′ ∈ Σ.
We will now refine that result by showing an implementation of KOMWU with linear-time(i.e., O(|Σ|)) per-iteration complexity, by exploiting the structure of the particular set of kernel
evaluations needed at every iteration. In particular, we rely on the following observation.
Proposition 7.1.

∗
For any player i ∈ [[m]], vector x ∈ ℝΣ
>0 , and sequence ja ∈ Σ ,Q
x[ja] j ′ ∈Cja Kj ′ (x, 1)
1 − KQ (x, ēja )/KQ (x, 1).
=
1 − KQ (x, ēpj )/KQ (x, 1)
Kj (x, 1)

We defer the proof of Proposition 7.1 as an appendix at the end of the chapter, and first discuss
how the result informs an efficient batched computation of the kernel evaluations needed in KOMWU.
Specifically, in light of Proposition 7.1, we do the following to compute {KQ (b(t) , ēσ ) : σ ∈ Σ} incumulative O(|Σ|) time.
1. First, we compute the values Kj (b(t) , 1) for all j ∈ J in cumulative O(|Σ|) time by using (7.14).
2. Then, we compute the ratio KQ (b(t) , ē∅ )/KQ (b(t) , 1) by evaluating the two kernel separately
using Theorem 7.4, spending O(|Σ|) time.
3. At this point, we repeatedly use Proposition 7.1 in a top-down fashion along the tree-form
decision process of player i to compute the ratio KQ (b(t) , ēja )/KQ (b(t) , 1) for each sequenceja ∈ Σ∗ given the value of the parent ratio KQ (b(t) , ēpj )/KQ (b(t) , 1) and the partial kernel
evaluations {Kj (b(t) , 1) : j ∈ J } from Step 1. For each ja ∈ Σ∗ , Proposition 7.1 gives aformula whose runtime is linear in the number of children decision nodes |Cja | at that sequence.
Therefore, the cumulative runtime required to compute all ratios KQ (b(t) , ēja )/KQ (b(t) , 1) isO(|Σ|).
4. Finally, by multiplying the ratios computed in Step 3 by the value of KQ (b(t) , 1) computed in
Step 2, we can easily recover each KQ (b(t) , ēσ ) for every σ ∈ Σ∗ .
Hence, we have just proved the following result.
Theorem 7.5. For each player i in a perfect-recall extensive-form game, the Kernelized OMWU
algorithm can be implemented exactly, with a per-iteration complexity linear in the numberof sequences |Σ| of that player.

§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

7.5

141

Regret bound of Kernelized OMWU

If the players in an imperfect-information extensive-form game run KOMWU, then we can combine
Theorem 7.3 with standard OMWU regret bounds given in Section 3.3.1 to immediately get thefollowing.
Theorem 7.6. In an imperfect-information extensive-form game, after any T iterations, KOMWU
satisfiesp√
1. A player i using KOMWU with η (t) := η = 8 log(A)∥Q∥1 / T is guaranteed to incurpregret at most Reg(T ) = O( ∥Q∥1 log(A)T ).
−1/4

T
2. For all T , if η (t) := η = √, the regret of each player i ∈ [[m]] is bounded as
√ m−1
(T )
Reg
⩽ (4 + ∥Q∥ log A) m − 1 · T 1/4 .

3. There exist C, C ′ > 0 such that if all m players learn using KOMWU with constant learning
rate η (t) := η ⩽ 1/(Cm log4 T ), then each player is guaranteed to incur regret at mostlog(A)∥Q∥1+ C ′ log T .
η
√
4. If all m player learn using KOMWU with η (t) := η ⩽ 1/ 8(m − 1), then the sum of regretsPmmis at most i=1 Reg(T ) = O(maxmi=1 {∥Q∥1 log Ai } η ).
5. For two-player zero-sum imperfect-information extensive-form games, if both players
learn using KOMWU, then there exists a schedule of learning-rates η (t) such that theiterates converge to a Nash equilibrium. Furthermore, if the NFG representation of
the imperfect-information extensive-form game has a unique Nash equilibrium andboth players use learning rates η (t) = η ⩽ 1/8, then the iterates converge to a Nashequilibrium at a linear rate C(1 + C ′ )−t , where C, C ′ are constants that depend on the
game.

Prior to our result, the strongest regret bound for methods that take linear time per iteration was
based on instantiating, e.g., follow the regularized leader (FTRL) or its optimistic variant with thedilatable global entropy regularizer discussed in Chapter 5 (Section 5.4.2). For FTRL this yields apregret bound of the form O( log(A) ∥Q∥21 T ). For optimistic FTRL this yields a regret bound of√
the form O(log(A) ∥Q∥21 mT 1/4 ), when every player in an m-player game uses that algorithm
and appropriate learning rates.
Our algorithm improves the state-of-the-art rate in two ways. First, we improve the dependence
on game constants by almost a square root factor, because our dependence on ∥Q∥1 is smaller
by a square root, compared to prior results. Secondly, in the multiplayer general-sum setting,every other method achieves regret that is on the order of T 1/4 , whereas our method achieves

142§7.6. EXPERIMENTAL EVALUATION

regret on the order of log4 (T ). In the context of two-player zero-sum imperfect-information
extensive-form games, the bound on the sum of regrets in Theorem 7.6 guarantees convergence
to a Nash equilibrium at a rate of O(maxi ∥Q∥1 log Ai /T ). This similarly improves the prior stateof the art.
C.-W. Lee, Kroer, and Luo (2021) showed the first last-iterate results for imperfect-information
extensive-form games using algorithms that require linear time per iteration. In particular, theyshow that the dilated entropy DGF combined with optimistic online mirror descent leads tolast-iterate convergence at a linear rate. However, their result requires learning rates η ⩽ 1/(8|Σ|).
This learning rate is impractically small in practice. In contrast, our last-iterate linear-rate resultfor KOMWU allows learning rates of size 1/8. That said, our result is not directly comparable to theirs.
The existence of a unique Nash equilibrium in the imperfect-information extensive-form gamerepresentation is a necessary condition for uniqueness in the NFG representation. However, it ispossible that the NFG has additional equilibria even when the imperfect-information extensiveform game does not. C. Wei, C. Lee, M. Zhang, and Luo (2021) conjecture that linear-rate
convergence holds even without the assumption of a unique Nash equilibrium. If this conjecture
turns out to be true for NFGs, then Theorem 7.3 would immediately imply that KOMWU also haslast-iterate linear-rate convergence without the uniqueness assumption.

7.6

Experimental evaluation

We complement the theoretical results provided in this chapter with a numerical investigation of
KOMWU in Kuhn and Leduc poker (Kuhn, 1950; Southey, Bowling, Larson, Piccione, Burch, Billings,
and Rayner, 2005), standard benchmark games from the extensive-form games literature. As
usual, a full description of the game instances we use is available in Appendix A.
Specifically, we investigate whether the strong theoretical guarantees enjoyed by KOMWU, which
crucially rely on predictivity, are shared by non-predictive algorithm such as (vanilla) CFR and
CFR instantiated with RM+ at all decision points. Additionally, we investigate how KOMWU compareswith DOMWU (C.-W. Lee, Kroer, and Luo, 2021).
Results are shown in Figure 7.2 and Figure 7.3.
For algorithms that require learning rates (KOMWU and DOMWU), we consider the four different
choices of constants η (t) := η ∈ {0.1, 1, 5, 10}. We remark that the payoff ranges of these games
are not [0, 1] (i.e., the games have not been normalized). The payoff range of Kuhn poker is 6 forthe 3-player variant and 8 for the 4-player variant. The payoff range of Leduc poker is 21 for the
3-player variant and 28 for the 4-player variant. So, a learning rate value of η = 0.1 corresponds toa significantly smaller learning rate in the normalized game where the payoffs have been shifted
and rescaled to lie within [0, 1].
In all games, we observe that the maximum per-player regret cumulated by KOMWU plateaus
and remains constants, unlike the CFR variants. This behavior is consistent with the near-optimal

143§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

Max. individual regret

K3b — 3-player Kuhn poker

K45 — 4-player Kuhn poker

102

102

101

101

100

100

Max. individual regret

L3133 — 3-player Leduc poker
10

L4133 — 4-player Leduc poker

3

103
102
102
101
101
100
100

101

102
Iteration
CFR

103

CFR (RM+ )

104

KOMWU

100

101

η = 0.1

102
Iteration
1

Figure 7.2: Experimental comparison of KOMWU with CFR.

5

103

10

104

144

§7.A. APPENDIX: PROOF DETAILS

Max. individual regret

K3b — 3-player Kuhn poker

K45 — 4-player Kuhn poker

103

103

102

102

101

101

100

100

L3133 — 3-player Leduc poker

L4133 — 4-player Leduc poker
104

Max. individual regret

3

10

103

102

102
1

10

101
100
100

101

102
Iteration

103

KOMWU

DOMWU

104

100

η = 0.1

101

η=1

102
Iteration
η=5

103

104

η = 10Figure 7.3: Experimental comparison of KOMWU with DOMWU for different choices of learning rates.

per-player regret guarantees of KOMWU (Theorem 7.6). In the 3-player variant of Leduc poker, weobserve that the largest learning rate we use, η = 10, leads to divergent behavior of the learning
dynamics.

7.A

Appendix: Proof details

7.A.1

Proof of Theorem 7.4

Proof. In the proof of this result, we will make use of the following additional notation. Given
any x ∈ ℝΣ and a j ∈ J , we let x(j) ∈ ℝΣ≽j denote the subvector obtained from x by only
considering sequences σ ∈ Σ≽j , that is, the vector whose entries are defined as x(j) [σ] = x[σ]for all σ ∈ Σ≽j .

145§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION• Proof of (7.13). Direct inspection of the definitions of Π and Π≽j (given in Section 2.2.2),
together with the observation that the {Σ≽j : j ∈ C∅ } form a partition of Σ∗ , reveals that(Σπ ∈ {0, 1} :

Π=

)

1

π[∅] = 1

2

π (j) ∈ Π≽j(7.15)∀ j ∈ C∅The observation above can be summarized informally into the statement that “Π isequal, up to permutation of indices, to the Cartesian product j∈C∅ Π≽j ”. The idea for the
proof is then to use that Cartesian product structure in the definition of 0/1-polyhedral
kernel (7.4), as follows

×

KQ (x, y) =X Yx[σ] y[σ]π∈Π σ∈π


=

X


Y

x[∅] y[∅]j ′ ∈Cπ∈ΠY

x[σ] y[σ]

∅ σ∈π (j ′ )




X

=

Y

x[∅] y[∅]j ′ ∈Cπ (j) ∈Π≽j ∀ j∈C∅Y

∅ σ∈π (j ′ )X= x[∅] y[∅]= x[∅] y[∅]


Y

Y


j ′ ∈Cπ (j) ∈Π≽j ∀ j∈C∅

Y

X

x[σ] y[σ]

Y

x[σ] y[σ]

∅ σ∈π (j ′ )x[σ] y[σ]j∈C∅ π (j) ∈Π≽j σ∈π (j)= x[∅] y[∅]Y

Kj (x, y),j∈C∅where the second equality follows from the fact that {∅} ∪ {Σ≽j : j ∈ C∅ } form a partitionof Σ, the third equality follows from (7.15), the fifth equality from the fact that eachπ j ∈ Π≽j can be chosen independently, and the last equality from the definition of partial
kernel function (7.12).
• Proof of (7.14). Similarly to what we did for (7.13), we start by giving an inductive
characterization of Π≽j as a function of the children Π≽j ′ for j ′ ∈ ∪a∈Aj Cja . Specifically,
direct inspection of the definitions of Π≽j , together with the observation that the{Σ≽j ′ : j ′ ∈ ∪a∈Aj Cja } form a partition of Σ≽j , reveals that(Π≽j =π ∈ {0, 1}

Σ≽j

:

1

P

2

π (j ′ ) ∈ π[ja] · Π≽j ′)a∈Aj π[ja] = 1∀ a ∈ Aj , j ′ ∈ Cja.

(7.16)

146§7.A. APPENDIX: PROOF DETAILS

From constraint 1 together with the fact that π[ja] ∈ {0, 1} for all a ∈ Aj , we conclude
that exactly one a∗ ∈ Aj is such that π[ja∗ ] = 1, while π[ja] = 0 for all other a ∈ Aj , a ̸= a∗ .
So, we can rewrite (7.16) as




[ 
Π≽j =
π ∈ {0, 1}Σ≽j :
a∗ ∈Aj 

π[ja∗ ] = 1
2 π[ja] = 0
3 π (j ′ ) ∈ Π≽j ′
4 π (j ′ ) = 0
1∀ a ∈ Aj , a ̸= a∗∀ j ′ ∈ Cja∗
∀ j ′ ∈ ∪a∈Aj ,a̸=a∗ Cja






,





(7.17)

where the union is clearly disjoint. The above equality can be summarized informally
into the statement that “Π≽j is equal, up to permutation of indices, to a disjoint union overactions a∗ ∈ Aj of Cartesian products j∈C ∗ Π≽j ”. We can then use the same set ofja
manipulations we already used in the proof of (7.13) to obtain×

Kj (x, y) =X Yx[σ] y[σ]π∈Π≽j σ∈π


=

X


Y

x[ja∗ ] y[ja∗ ]j ′ ∈Cπ∈Π≽jYx[σ] y[σ]

ja∗ σ∈π (j ′ )




=

X

X

a∗ ∈A

x[ja∗ ] y[ja∗ ]Y
j ′ ∈Cj π j ′ ∈Π≽j ′∀ j ′ ∈Cja∗Yx[σ] y[σ]

ja∗ σ∈π (j ′ )


=

X


Y

x[ja∗ ] y[ja∗ ]a∗ ∈Aj=x[ja] y[ja]

a∈Aj

Y

x[σ] y[σ]

j ′ ∈Cja∗ π (j ′ ) ∈Π≽j ′ σ∈π (j ′ )


X

X


Y

Kj ′ (x, y),j ′ ∈Cja

where the second equality follows from the fact that the {Σ≽j ′ : j ′ ∈ ∪a∈Aj Cja } form apartition of Σ≽j , third equality follows from (7.17), the fourth equality from the fact thateach π j ′ ∈ Π≽j ′ can be picked independently, and the last equality from the definition ofpartial kernel function (7.12) as well as renaming a∗ into a.

7.A.2

Proof of Proposition 7.1

We give a self-contained proof of the result stated in Proposition 7.1, repeated below for
convenience.

§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

Proposition 7.1 (Restated).

147∗
For any player i ∈ [[m]], vector x ∈ ℝΣ
>0 , and sequence ja ∈ Σ ,Q
x[ja] j ′ ∈Cja Kj ′ (x, 1)
1 − KQ (x, ēja )/KQ (x, 1)=.
1 − KQ (x, ēpj )/KQ (x, 1)
Kj (x, 1)

Proof. Note that since x > 0, clearly KQ (x, 1), Kj (x, 1) > 0. Furthermore, from (7.11) we havethat for all σ ∈ Σ
KQ (x, 1) − KQ (x, ēσ ) = ⟨ϕQ (1) − ϕQ (ēσ ), ϕQ (x)⟩ =XY

x[σ ′ ] > 0.

(7.18)π∈Π σ ′ ∈π
π[σ]=1

The above inequality immediately implies that 0 < KQ (x, ēpj )/KQ (x, 1) < 1 and therefore all
denominators in the statement are nonzero, making the statement well-formed.
In light of (7.18), we further haveQx[ja] j ′ ∈Cja Kj ′ (x, 1)
1 − KQ (x, ēja )/KQ (x, 1)=
1 − KQ (x, ēpj )/KQ (x, 1)
Kj (x, 1)⇐⇒⇐⇒Q
x[ja] j ′ ∈Cja Kj ′ (x, 1)
KQ (x, 1) − KQ (x, ēja )=KQ (x, 1) − KQ (x, ēpj )Kj (x, 1)QP
Q
x[ja] j ′ ∈Cja Kj ′ (x, 1)π∈Π,π[ja]=1
σ∈π x[σ]PQ=.
Kj (x, 1)π∈Π,π[pj ]=1σ∈π x[σ](7.19)

We now prove (7.19). Let
A := {π ∈ Π : π[ja] = 1},

B := {π ∈ Π : π[pj ] = 1}

be the domains of the summations. From the definition of Π (specifically, constraints 2 in thedefinition of Q, of which Π is a subset by definition; see Definition 2.5), it is clear that A ⊆ B.
Furthermore, it is straightforward to check, using the definitions of Π≽j , Π, and B, that
π (j) ∈ Π≽j∀π ∈ B(7.20)

We now introduce the function ((· ∥ ·)) : B × Π≽j → B defined as follows. Given any π ∈ Band π ′ ∈ Π≽j , ((π ∥ π ′ )) is the vector obtained from π by replacing all sequences at or below
decision node j with what is prescribed by π ′ ; formally,

148§7.A. APPENDIX: PROOF DETAILSπ ′ [σ] if σ ∈ Σ≽j((π ∥ π ′ ))[σ] :=π[σ] otherwise.

∀ π ∈ B, π ′ ∈ Π≽j(7.21)

It is immediate to check that ((π ∥ π ′ )) is indeed an element of B. We now introduce thefollowing result.
Lemma 7.2. There exists a set P ⊆ B such that every π ′′ ∈ B can be uniquely written as
π ′′ = ((π ∥ π ′ )) for some π ∈ P and π ′ ∈ Π≽j . Vice versa, given any π ∈ P and π ′ ∈ Π≽j ,then ((π ∥ π ′ )) ∈ B.
Proof. The second part of the statement is straightforward. We now prove the first part.
Fix any π ∗ ∈ Π≽j and let P := {((π ∥ π ∗ )) : π ∈ B}. It is straightforward to verify thatfor any π ′′ ∈ B, the choices π := ((π ′′ ∥ π ∗ )) ∈ P and π ′ := π (j) ∈ Π≽j satisfy the equality((π ∥ π ′ )) = π ′′ . So, every π ′′ ∈ B can be expressed in at least one way as π ′′ = ((π ∥ π ′ )) forsome π ∈ P and π ′ ∈ Π≽j . We now show that the choice above is in fact the unique choice.
First, it is clear from the definition of ((· ∥ ·)) that π ′ must satisfy π ′ = π ′′(j) , and so it isuniquely determined. Suppose now that there exist π, π̃ ∈ P such that ((π ∥ π ′ )) = ((π̃ ∥ π ′ )).
Then, π and π̃ must coincide on all σ ∈ Σ \ Σ≽j . However, since all elements of P are of theform ((b ∥ π ∗ )) for some b ∈ B, then π and π̃ must also coincide on all σ ∈ Σ≽j . So, π and π̃
coincide on all coordinates σ ∈ Σ, and the statement follows.
Lemma 7.2 exposes a convenient combinatorial structure of the set B. In particular, it
enables us to rewrite the denominator on the left-hand side of (7.19) as follows
X Y

x[σ] =

X

X

Y

x[σ]

π ′ ∈P π ′′ ∈Π≽j σ∈((π ′ ∥ π ′′ ))π∈B σ∈π


=

X

X 



′′

π ′ ∈P π ∈Π≽j

=

X


=

X

Y

σ∈((π ′ ∥ π ′′ ))
σ∈Σ≽j

X

π ′ ∈P π ′′ ∈Π≽j




x[σ]Yσ∈((π ′ ∥ π ′′ ))σ̸∈Σ≽j



!
Y
 Y

x[σ] 
x[σ]


σ∈π ′′σ∈π ′σ̸∈Σ≽jYXx[σ]


π ′′ ∈Π≽j σ∈π ′′Y

π ′ ∈P σ∈π ′σ̸∈Σ≽j


x[σ]




x[σ]


§7. STATE-OF-THE-ART REGRET DEPENDENCE ON GAME SIZE VIA KERNELIZATION

149




X
= Kj (x, 1) · Y

π ′ ∈P σ∈π ′σ̸∈Σ≽jx[σ]
,(7.22)

where we used (7.21) in the third equality.
We can use a similar technique to express the numerator of the left-hand side of (7.19). Let
Πi,ja := {π ∈ Π≽j : π[ja] = 1}.
Using the constraints that define Π and the definition of A, it follows immediately that for any
π ∈ A, π (j) ∈ Πi,ja . Furthermore, a direct consequence of Lemma 7.2 is the following:
Corollary 7.1. The same set P ⊆ B introduced in Lemma 7.2 is such that every π ′′ ∈ A
can be uniquely written as π ′′ = ((π ∥ π ′ )) for some π ∈ P and π ′ ∈ Πi,ja .
Using Corollary 7.1 and following the same steps that led to (7.22), we express the numeratorof the left-hand side of (7.19) as
X Yx[σ] =

X

X

Y

x[σ]

π ′ ∈P π ′′ ∈Πi,ja σ∈((π ′ ∥ π ′′ ))π∈A σ∈π=XX

π ′ ∈P π ′′ ∈Πi,ja








Y

σ∈((π ′ ∥ π ′′ ))
σ∈Σ≽j





x[σ]Yσ∈((π ′ ∥ π ′′ ))σ̸∈Σ≽j



x[σ]


!
=

X

X

Y

π ′ ∈P π ′′ ∈Πi,ja


=

X

σ∈π ′′ Y
x[σ] 
x[σ]


σ∈π ′
σ̸∈Σ≽jYXx[σ]


π ′′ ∈Πi,ja σ∈π ′′
Y

π ′ ∈P σ∈π ′σ̸∈Σ≽jx[σ]
.

(7.23)The statement then follows immediately if we can prove thatXYπ∈Πi,ja σ∈πx[σ] = x[ja]Y

Kj ′ (x, 1).

j ′ ∈Cja

To do so, we use the same approach as in the proof of Theorem 7.4. In fact, we can directly use
the inductive characterization of Π≽j obtained in (7.17) to write

150§7.A. APPENDIX: PROOF DETAILS

Πi,ja =
π[ja] = 12 π[ja′ ] = 0
3 π (j ′ ) ∈ Π≽j ′
4 π (j ′ ) = 01

π ∈ {0, 1}Σ≽j :






∀ a′ ∈ Aj , a′ ̸= a
∀ j ′ ∈ Cja∀ j ′ ∈ ∪a′ ∈Aj ,a′ ̸=a Cja′






,





which fundamentally uncovers the Cartesian-product structure of Πi,ja . Using the same techniqueas Theorem 7.4, we then have


Y Y
X Y
X
x[ja]
x[σ] =x[σ]
π∈Πi,ja σ∈ππ (j ′ ) ∈Π≽j ′ ∀ j ′ ∈Cjaj ′ ∈Cja σ∈π (j ′ )= x[ja]Y
j ′ ∈CYj ′ ∈Cand the statement is proven.

Yja π (j ′ ) ∈Π≽j ′ σ∈π (j ′ )= x[ja]

X

ja

Kj ′ (x, 1),x[σ]Part III

Computation of extensive-form
correlated and team equilibria

Chapter 8

Uncoupled learning ofextensive-form correlated
equilibrium
In Chapter 4 we have seen how one can construct simple, uncoupled no-external-regret dynamics
for sequence-form strategy polytopes in any imperfect-information extensive-form game. Due tothe connection between external regret and coarse correlated equilibria discussed in Chapter 3,
those dynamics can be used to compute normal-form coarse-correlated equilibria in generalmultiplayer games, as well as Nash equilibria in two-player zero-sum games.
However, a different notion of equilibrium, extensive-form correlated equilibrium (thecounterpart of correlated equilibrium in normal-form games), cannot be obtained throughno-external-regret dynamics, begging the following question.
Do uncoupled, polynomial-time learning dynamics leading to the set of extensive-form correlated
equilibrium exist in any imperfect-information extensive-form game?
In this chapter, we settle it for the positive.

8.1

Contributions and related work

In this chapter we show that it is possible to construct simple, uncoupled learning dynamics
that minimize a stronger notion of regret—that is, that guarantee a stronger notion of hindsightrationality—than external regret. In particular, we will define and study an instance of Φ-regret
based on the notion of trigger deviation functions. As we will show, the minimization of Φ-regret

154§8.1. CONTRIBUTIONS AND RELATED WORKwith respect to the set of trigger deviation functions leads to extensive-form correlated equilibrium (EFCE), the imperfect-information extensive-form game counterpart of what correlated
equilibrium is for single-decision simultaneous-action games (von Stengel and Forges, 2008).

Related work The notion of trigger deviation functions which we introduce in this chapter
builds on prior work on extensive-form transformations by G. J. Gordon, A. Greenwald, and Marks(2008). However, trigger deviation functions are simpler than the extensive-form transformations.
Specifically, extensive-form transformations allow one to specify more than one trigger sequence(together with different continuation strategies, one for each specified trigger sequence), whereas
our notion of trigger deviation functions only contemplates a single trigger sequence. Consequently, the set of all trigger deviation functions is significantly smaller, and simpler, than theset of all extensive-form transformations. The simpler structural properties of the set of trigger
deviation functions, explored in Section 8.3.2, will enable us to construct an efficient no-regret
algorithm for the convex hull of the set of all canonical trigger deviation matrices.
We also mention relevant literature concurrent and subsequent to the conference version of
the paper on which this chapter is based (Celli, Marchesi, Farina, and Gatti, 2020). First, we
acknowledge the thesis work by H. Zhang (2022) on computing certain refinements of EFCE
via polynomial-time uncoupled learning dynamics, though their procedure could require up
to exponential memory. In a later chapter of the thesis, the author notes that some of the
dynamics introduced in the thesis can be modified to guarantee polynomial memory usage whenconvergence to the set of (unrefined) EFCE is sought. That work was conducted independently
and concurrently with ours. In a recent paper, Morrill, D’Orazio, Sarfati, Lanctot, J. Wright, A.
Greenwald, and Bowling (2020) conduct a study of different forms of correlation in extensive-form
games, defining a taxonomy of solution concepts. Each of their solution concepts is attainedby a particular set of no-regret learning dynamics, which is obtained by instantiating the phiregret minimization framework (A. Greenwald and Jafari, 2003; Stoltz and Lugosi, 2007; G. J.
Gordon, A. Greenwald, and Marks, 2008) with a suitably-defined deviation function. As part oftheir analysis, Morrill, D’Orazio, Sarfati, Lanctot, J. Wright, A. Greenwald, and Bowling (2020)
investigate some properties of the well-established CFR regret minimization algorithm (Zinkevich,
Bowling, Johanson, and Piccione, 2007) applied to n-player general-sum extensive-form games,
establishing that it is hindsight-rational with respect to a specific set of deviation functions, which
the authors coin blind counterfactual deviations. In subsequent recent work, Morrill, D’Orazio,
Lanctot, J. R. Wright, Bowling, and A. R. Greenwald (2021) extend their prior work (Morrill,
D’Orazio, Sarfati, Lanctot, J. Wright, A. Greenwald, and Bowling, 2020) by identifying a generalclass of deviations—called behavioral deviations—that induce equilibria that can be found throughuncoupled no-regret learning dynamics. Behavioral deviations are defined as those specifying anaction transformation independently at each decision node of the game. As the authors note, thedeviation functions involved in the definition of EFCE do not fall under that category. A particular§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

155

class of behavioral deviation functions—called causal partial sequence deviations—induces solution
concepts that are (subsets of) EFCEs. Thus, their result begets an alternative set of no-regret
learning dynamics that converge to EFCE, based on a different set of deviation functions thanthose we use in this dissertation.

Organization This chapter is organized as follows. In Section 8.2 we introduce the notion of
trigger agents and trigger deviation functions, and formally establish the important connection
between such notions and convergence to the set of EFCEs via no-regret dynamics. In Section 8.3
we show how regret with respect to the set of all trigger deviation functions can be kept sublinear.
We do so by first identifying a suitable characterization of the set of all trigger deviation functions.
Such a characterization exhibits a strong combinatorial structure that naturally lends itself tobeing addressed via the regret decomposition techniques we saw in Chapter 4. Pseudocode forthe resulting learning dynamics, as well as a theoretical analysis, is given in Section 8.3.3.

8.2

Extensive-form correlated equilibrium and its relation with
Φ-regret

Extensive-form correlated equilibrium (EFCE) has been proposed by von Stengel and Forges(2008) as the natural counterpart to (normal-form) correlated equilibrium in sequential games. In
an EFCE, before the beginning of the game an external mediator draws a recommended action
for each of the possible decision nodes that players may encounter in the game, according tosome probability distribution known to the player. These recommendations are not immediately
revealed to each player. Instead, the mediator incrementally reveals relevant action recommendations as players reach new decision nodes. At any decision node, the acting player is free todeviate from the recommended action, but doing so comes at the cost of future recommendations,which are no longer issued if the player deviates. In an EFCE, the recommended behavior is
incentive-compatible for each player, that is, no player is strictly better off ever deviating from anyof the mediator’s recommended actions.
We make the definition of EFCE formal in the next subsection.

8.2.1

Trigger agents and trigger deviation functions

Multiple equivalent definitions of EFCE can be given; we will follow the equivalent formulation
given by Farina, Ling, Fang, and Sandholm (2019a) based on the concept of trigger agents introducedby G. J. Gordon, A. Greenwald, and Marks (2008) and Dudik and G. J. Gordon (2009). Fix any
n-player imperfect-information extensive-form game, and let i ∈ [[n]] be a player. Furthermore,let σ̂ = (j, a) ∈ Σ∗i be a non-empty sequence for Player i, and let π̂ ∈ Πi, ≽j be a strategy for the

156§8.2. EXTENSIVE-FORM CORRELATED EQUILIBRIUM AND ITS RELATION WITH Φ-REGRET

subtree rooted at decision node j of the tree-form decision process for Player i. The (σ̂, π̂)-trigger
agent is the agent that plays the game as Player i according to the following rules.
• If the trigger agent has never been recommended to play action a at decision node j, thetrigger agent will follow whatever recommendation is issued by the mediator.
• When the trigger agent reaches decision node j and is recommended to play action a, we
say that the trigger agent “gets triggered” by the trigger sequence σ̂ = (j, a). This means
that, from that point on, the trigger agent will disregard the recommendations and playaccording to the continuation strategy π̂ from decision node j onward (that is, at j and all its
descendant decision nodes).
More formally, a (σ̂, π̂)-trigger agent for Player i is a function that maps a recommended strategy
to realized behavior, according to a trigger deviation function, which we now formally define.
Definition 8.1 (Trigger deviation function). Let i ∈ [[n]], σ̂ = (j, a) ∈ Σ∗i , and π̂ ∈ Πi, ≽j . We
call “trigger deviation function corresponding to trigger σ̂ and continuation strategy π̂”, any linear
function ϕσ̂→π̂ : ℝΣi → ℝΣi whose effect on sequence-form strategies is as follows:• all strategies π ∈ Πi that do not prescribe the sequence σ̂ are left unmodified. In
symbols,
ϕσ̂→π̂ (π) = π∀ π ∈ Πi : π[σ̂] = 0;

(8.1)

• all strategies π ∈ Πi that prescribe sequence σ̂ = (j, a) are modified so that the behavior
at j and all of its descendants is replaced with the behavior prescribed by the continuation
strategy π̂. In symbols,π[σ] if σ ̸≽ j
ϕσ̂→π̂ (π)[σ] =π̂[σ] if σ ≽ j∀ σ ∈ Σi , π ∈ Πi : π[σ̂] = 1.

(8.2)

At this stage, it is technically unclear whether a linear function that satisfies Definition 8.1 exists
for all valid choices of σ̂ and π̂. We show that this is indeed the case in Section 8.3.1, by exhibiting
such a linear function via an explicit transformation matrix. Until then, we will take for granted
the well-posedness of the definition of trigger deviations, and focus on their relationship with
EFCE.
Having defined trigger deviation functions, we are now ready to formally define the conceptof extensive-form correlated equilibrium (EFCE) as a distribution of recommended behavior, suchthat no player has incentive to unilaterally deviate from the recommendation according to any
trigger deviation function. Formally, we have the following.

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

157

Definition 8.2 (ϵ-EFCE). Let Γ be an n-player imperfect information game, where eachplayer i ∈ [[n]] has multilinear expected utility ui : Q1 × · · · × Q[[n]] → ℝ. A distributionµ ∈ ∆Π1 ×···×Π[[n]] is an ϵ-EFCE if for all i ∈ [[n]], σ̂ = (j, a) ∈ Σ∗i , and π̂ ∈ Πi, ≽j ,X
π 1 ∈ Π1

X

···



µ[π 1 , . . . , π n ] ui (ϕσ̂→π̂ (π i ), π −i ) − ui (π i , π −i ) ⩽ ϵ.

π n ∈ Πn

In particular, an EFCE is defined as a 0-EFCE.

8.2.2

Convergence to the set of EFCEs via no-Φ-regret dynamics

We now show that the set of EFCEs can be approached by no-Φ-regret dynamics, where Φ is theset of all trigger deviation functions (Definition 8.1). This result can be thought of as the extensionof celebrated connection between correlated equilibrium and internal regret (D. Foster and Vohra,
1997) to imperfect-information extensive-form games.
Theorem 8.1. Consider an n-player repeated imperfect-information extensive-form game,(T )and denote with Regi the Φ-regret of a generic player i, with respect to the set of all trigger
deviation functions (Definition 8.1) for that player, that is,(T )Regi :=

( T 
)




X
(t)
(t)
(t)
(t)
max ∗
ui ϕσ̂→π̂ (xi ), x−i − ui xi , x−i.

σ̂=(j,a)∈Σi
π̂∈Πi, ≽j

t=1

(t)

Furthermore, for all i ∈ [[i]] and time t, let µi(t)expectation is xi , that is, be such that
(t)

xi =

X

∈ ∆Πi be a distribution of strategies whose(t)µi [π i ] π i .

π i ∈Πi

Then, at all times T the average product distribution of playTµ̄(T ) :=1 X (t)µ ⊗ · · · ⊗ µ(t)n ,T t=1 1is an ϵ(T ) -EFCE, where(T )ϵ(T ) :=

maxi∈[[n]] Regi
T

.

(8.3)

158

§8.2. EXTENSIVE-FORM CORRELATED EQUILIBRIUM AND ITS RELATION WITH Φ-REGRET

Proof. Fix any player i ∈ [[n]], trigger sequence σ̂ = (j, a) ∈ Σ∗i , and continuation strategyπ̂ ∈ Πi, ≽j . From the multilinearity of the expected utility function ui as a function ofsequence-form strategies, we obtain that



(t)

(t)



!



ui ϕσ̂→π̂ (xi ), x−i = ui ϕσ̂→π̂

X

(t)

(t)

X

µi [π i ] π i ,µ−i [π −i ] π −i 

π i ∈Πi

π −i ∈Π−i

(t)

X


= ui Xµi [π i ]ϕσ̂→π̂ (π i ),π i ∈Πi=···π 1 ∈ Π1(8.4)

π −i ∈Π−i




X

(t)

µ−i [π −i ] π −i XYπ n ∈ Πn




u
ϕ
(π
),
π
,
µ(t)
[π
]
r
i
σ̂→
π̂
i
−i
r

r∈[[n]]

where (8.4) follows from the linearity of ϕσ̂→π̂ . Similarly, we have

(t)

(t)

ui (xi , x−i ) =X
π 1 ∈ Π1

···

X


Y


π n ∈ Πn




µ(t)
[π
]
u
π
,
π
.
r
i
i
−i
r

r∈[[n]]

Hence, using the definition of Φ-regret for Player i, we can write
(T )

Regi
T

T  



1X
(t)
(t)
(t)
(t)
ui ϕσ̂→π̂ (xi ), x−i − ui xi , x−i
T t=1


 
T



X
Y
X
X
1
(t)


µr [π r ]ui ϕσ̂→π̂ (π i ), π −i − ui π i , π −i
=
···
T t=1
π 1 ∈ Π1
π n ∈ Πnr∈[[n]]
 



X
X
=
···
µ̄[π 1 , . . . , π n ] ui ϕσ̂→π̂ (π i ), π −i − ui π i , π −i .

⩾π 1 ∈ Π1

π n ∈ ΠnRearranging, taking a maximum over players i ∈ [[n]], trigger sequences σ̂ = (j, a) ∈ Σ∗i , andcontinuation strategies π̂ ∈ Πi, ≽j , and applying Definition 8.2 yields the statement.

Theorem 8.1 relates an inherently global object—EFCE, a correlated distribution of play—to
an inherently local, per-player goal—hindsight rationality for a learning agent with respect to a
specific set of deviation functions. In light of this connection, in the rest of the chapter we focus on
how any individual player can ensure their Φ-regret with respect to the set of trigger deviations
grows sublinear in time. As a result, we will assume that a particular player and correspondingtree-form decision process has been isolated, and we will omit (and imply) all player subscripts
from the corresponding TFDP quantities.

159§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

Definition 8.3. We refer to a no-Φ-regret algorithm for the set Φ of all trigger deviation
functions as a no-trigger-regret algorithm.

The rest of the chapter will be concerned with constructing simple, efficient no-trigger-regret
dynamics.

8.3

Construction of no-trigger-regret dynamics

For the rest of the chapter we assume that an arbitrary TFDP has been fixed, and omit any player
indices.
In this section, we give an explicit construction of a no-trigger-regret algorithm.[8.a] To that
end, in Section 8.3.1 we start by giving an explicit characterization of the set of all trigger deviation
functions, showing that it admits a concise combinatorial description in terms of canonical trigger
deviation matrices. Then, in Section 8.3.2 we study the combinatorial structure of the convex
hull of canonical trigger deviation matrices, showing that it can be described using simple
convexity-preserving operations already studied in Chapter 4.

8.3.1

Canonical trigger deviation matrices

In this subsection, we show that for any trigger sequence σ̂ = (j, a) ∈ Σ∗ and continuation
strategy π̂ ∈ Π≽j , a particular trigger deviation function (Definition 8.1) can be written explicitly
in matrix form, using a heavily structured object which we coin canonical trigger deviation matrix.
Such a structural characterization will form the basis for constructing simple, efficient, uncoupled
no-regret learning dynamics that converge to the set of EFCEs in any imperfect-information
extensive-form game.
ΣDefinition 8.4 (Canonical trigger deviation matrix). Let σ̂ = (j, a) ∈ Σ∗ and y ∈ ℝ⩾0≽j . We
denote with Tσ̂→y ∈ ℝΣ×Σ
⩾0 the matrix whose entries are defined asTσ̂→y [σr , σc ] :=



1


if σc ̸≽ σ̂ and σr = σcy[σr ] if σc = σ̂ and σr ≽ j





0

∀ σr , σc ∈ Σ.

(8.5)

otherwise

The matrix Tσ̂→y is called the “canonical trigger deviation matrix corresponding to trigger σ̂ andcontinuation strategy π̂”. Furthermore, the set of all canonical trigger deviation matrices is[8.a] In fact, this was the first efficient construction proposed in the literature.

160§8.3. CONSTRUCTION OF NO-TRIGGER-REGRET DYNAMICS

denoted with the symbolnoΨ := Tσ̂→π̂ : σ̂ = (j, a) ∈ Σ∗ , π̂ ∈ Π≽j .
Canonical trigger deviation matrices fully characterize the set of trigger deviation functions,as we show next.
Lemma 8.1. For any σ̂ = (j, a) ∈ Σ∗ and π̂ ∈ Π≽j , the linear function
ϕσ̂→π̂ : x 7→ Tσ̂→π̂ x,where Tσ̂→π̂ is as defined in Definition 8.4, is a trigger deviation function corresponding totrigger σ̂ and continuation strategy π̂, in the sense of Definition 8.1.

Proof. The proof just amounts to a simple application of several definitions. Let π ∈ Π bean arbitrary sequence-form strategy. By expanding the matrix-vector multiplication Tσ̂→π̂ πusing the definition (8.5), we obtain that for all σ ∈ Σ(Tσ̂→π̂ π)[σ] = π[σ]𝟙σ̸≽σ̂ + π̂[σ]π[σ̂]𝟙σ≽j .

(8.6)

There are only two possibilities:• If π[σ̂] = 0, then (8.6) simplifies toπ[σ] if σ ̸≽ σ̂
(Tσ̂→π̂ π)[σ] =0otherwise.
Since by case hypothesis the probability of the sequence of actions from the root of thegame tree down to σ̂ is zero, then necessarily the probability of any longer sequenceof actions σ ≽ σ̂ must be zero as well, that is π[σ] = 0 for all σ ≽ σ̂. So, Tσ̂→π̂ π = πand (8.1) holds.
• Conversely, assume π[σ̂] = 1. This means that at decision node j ∈ J action a is selected(with probability 1), and therefore π[σ] = 0 for all σ = (j, a′ ) : a′ ∈ Aj , a′ ̸= a. This meansthat π[σ]𝟙σ̸≽σ̂ = π[σ]𝟙σ̸≽j for all σ ∈ Σ. Substituting that equality into (8.6) gives (8.2), as
we wanted to show.
As the combinatorial structure of canonical trigger deviation functions will be the crucialcomponent in the construction of our simple no-regret dynamics converging to the set of EFCEs,
we now pause to provide a few examples. Specifically, in the following example we show three161§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

canonical trigger deviation matrices, and illustrate how they modify sequence-form strategies ina simple extensive-form game.
Example 8.1. We build on the small extensive-form game and example sequence-form
strategies defined in Example 2.7, which are reproduced below for convenience.
π 135


1.0
 
A1  1.0 
 
A2  0.0  
B3  1.0 
 
B4  0.0  
C5  1.0 
 
C6  0.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅

A
1

2
D
789

B

C

3 4

5 6

π 136




π 145


1.0
 
A1  1.0 
 
A2  0.0  
B3  1.0 
 
B4  0.0  
C5  0.0 
 
C6  1.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅



π 27


1.0
 
A1  1.0 
 
A2  0.0  
B3  0.0  
B4  1.0 
 
C5  1.0 
 
C6  0.0 
 
D7  0.0  
D8  0.0 
D9 0.0
∅



π 28


1.0
 
A1  0.0  
A2  1.0  
B3  0.0  
B4  0.0  
C5  0.0 
 
C6  0.0 
 
D7  1.0 
 
D8  0.0 
D9 0.0
∅




1.0
 
A1  0.0  
A2  1.0  
B3  0.0  
B4  0.0  
C5  0.0 
 
C6  0.0 
 
D7  0.0  
D8  1.0 
D9 0.0∅We consider the following three examples of canonical trigger deviation matrices:

Trigger sequence: A1
Continuation: A2, D7

Trigger sequence: A2
Continuation: A1, B3, C5

Trigger sequence: B3
Continuation: B4

∅
1 0 0 0 0 0 0 0 0 0

A1 
0 0 0 0 0 0 0 0 0 0 
A2  0 1 1 0 0 0 0 0 0 0 
B3  0 0 0 0 0 0 0 0 0 0 
B4  0 0 0 0 0 0 0 0 0 0 
C5  0 0 0 0 0 0 0 0 0 0 

C6  0 0 0 0 0 0 0 0 0 0 

D7 
0
1
0
0
0
0
0
1
0
0



D8 0 0 0 0 0 0 0 0 1 0 
D9 0 0 0 0 0 0 0 0 0 1∅

∅
1 0 0 0 0 0 0 0 0 0

A1 
0 1 1 0 0 0 0 0 0 0 
A2  0 0 0 0 0 0 0 0 0 0 
B3  0 0 1 1 0 0 0 0 0 0 
B4  0 0 0 0 1 0 0 0 0 0 
C5  0 0 1 0 0 1 0 0 0 0 

C6  0 0 0 0 0 0 1 0 0 0 

D7 
0
0
0
0
0
0
0
0
0
0



D8 0 0 0 0 0 0 0 0 0 0 
D9 0 0 0 0 0 0 0 0 0 0
∅

A1
A2
B3
B4
C5
C6
D7
D8
D9

Tc

A1
A2
B3
B4
C5
C6
D7
D8
D9

Tb

A1
A2
B3
B4
C5
C6
D7
D8
D9

Ta

∅

1 0 0 0 0 0 0 0 0 0

A1 
0 1 0 0 0 0 0 0 0 0 
A2  0 0 1 0 0 0 0 0 0 0 
B3  0 0 0 0 0 0 0 0 0 0 
B4  0 0 0 1 1 0 0 0 0 0 
C5  0 0 0 0 0 1 0 0 0 0 

C6  0 0 0 0 0 0 1 0 0 0 

D7 
0
0
0
0
0
0
0
1
0
0



D8 0 0 0 0 0 0 0 0 1 0 
D9 0 0 0 0 0 0 0 0 0 1∅

Figure 8.1: Canonical trigger deviation matrices. Entries shaded with dark gray represent theentries of the matrix defined in the second case of Equation (8.5). Given trigger sequenceσ̂ ∈ Σ∗ , all indices (σr , σc ) such that σr , σc ≽ σ̂ are shaded with light gray.

First example (TA1→π̂A1 ) First, let us consider the trigger deviation matrix corresponding totrigger sequence σ̂ = (A, 1), and the continuation strategy π̂ A1 that plays action 2 at decisionnode A, and subsequently action 7 at decision node D. The matrix TA1→π̂A1 is reported

162§8.3. CONSTRUCTION OF NO-TRIGGER-REGRET DYNAMICS

in Figure 8.1 (Left). In order to illustrate the effect of this linear mapping on sequence-form
strategies, we provide some examples using the sequence-form strategy vectors definedin Figure 2.5. First, we observe that any sequence-form strategy choosing action A1 withprobability 1 triggers a deviation which follows the continuation strategy π̂. The deviation
for those sequence-form strategies results in a final sequence-form strategy equal to π 27 . For
example, using some of the-sequence form strategies of Figure 2.5, it can be easily verified (byworking out the matrix-vector product) that:
TA1→π̂A1 π 135 = TA1→π̂A1 π 136 = TA1→π̂A1 π 145 = π 27 .
On the other hand, sequence-form strategies that do not select sequence 1 are left unmodified
by the linear mapping. For instance,
TA1→π̂A1 π 28 = π 28 and TA1→π̂A1 π 27 = π 27 .
Second example (TA2→π̂A2 ) Second, we examine the trigger deviation matrix TA2→π̂A2 fortrigger sequence σ̂ = (A, 2), where the continuation strategy π̂ A2 is defined so that Player 1
plays action 1 at decision node A, sequence 3 at decision node B, and action 5 at decision node
C. The corresponding matrix TA2→π̂A2 is reported in Figure 8.1 (Middle). As in the previous
case, all sequence-form strategy vectors which put probability 1 on action 2 at decision node
A are modified so that be strategy at A and its descendants B, C, D matches the continuation
strategy. For example, we have that
TA2→π̂A2 π 27 = TA2→π̂A2 π 28 = π 135 .
Furthermore, sequence-form strategies which do not put probability 1 on sequence A2 areleft unchanged. So, for example,
TA2→π̂A2 π 136 = π 136 and TA2→π̂A2 π 145 = π 145 .
Third example (TB3→π̂B3 ) As a final example, Figure 8.1 (Right) shows the deviation matrix
TB3→π̂B3 corresponding to trigger sequence σ̂ = (B, 3) and continuation strategy π̂ B3 selecting
action 4 at decision node B. Here, we have that
TB3→π̂B3 π 135 = TB3→π̂B3 π 145 and TB3→π̂B3 π 145 = π 145 .

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

8.3.2

163

Structural decomposition of canonical trigger deviation matrices

The characterization of trigger deviation functions through canonical trigger deviation matrices(Lemma 8.1) enables us to simplify the task of constructing no-trigger-regret dynamics to ensuring
sublinear regret compared to canonical trigger deviation matrices. Therefore, for the rest of
the chapter our objective will be to construct a no-Ψ-regret algorithm for any sequence-form
polytope Q. We will achieve that by actually considering a slightly more complicated problem:
constructing a no-(co Ψ)-regret algorithm, that is, tackling the convex hull of Ψ. It is clear, sinceco Ψ ⊇ Ψ, that any no-(co Ψ)-regret algorithm is automatically a no-Ψ-regret algorithm as well.
From the general theory developed in Chapter 4, a no-(co Ψ)-regret algorithm can be constructed from any no-external-regret algorithm R̃ for the set co Ψ. In this section we show
how the strong combinatorial structure of co Ψ enables use of regret circuits to construct a
no-external-regret algorithm for it.
The starting point of our approach is the observation that, because the convex hull operation
is associative, the setnoco Ψ = co Tσ̂→π̂ : σ̂ = (j, a) ∈ Σ∗ , π̂ ∈ Π≽jcan be evaluated in two stages:
1. First, for each sequence σ̂ = (j, a) ∈ Σ∗ one can define the setnoΛσ̂ := co Tσ̂→π̂ : π̂ ∈ Πj ;2. Then, one can take the convex hull of all Λσ̂ , that is,noco Ψ = co Λσ̂ : σ̂ ∈ Σ∗ .

(8.7)

Our construction of R̃ will follow a similar structure. First, for each σ̂ ∈ Σ∗ we will construct
a no-regret algorithm Rσ̂ for the set of deviations Λσ̂ . Then, we will combine all the no-regret
algorithms Rσ̂ into a composite no-regret algorithm R̃ for the set co Ψ.
No-external-regret algorithm for the inner sets Λσ̂ For any σ̂ = (j, a) ∈ Σ∗ , a no-regret
algorithm for the set Λσ̂ can be constructed starting from any no-regret algorithm for the set Q≽jby using one of the composition rules (regret circuits) seen in Chapter 4. The crucial insight liesin the observation that the mappingHσ̂ : ℝΣ≽j ∋ y 7→ Tσ̂→y164§8.3. CONSTRUCTION OF NO-TRIGGER-REGRET DYNAMICS

is affine, since it is clear from Definition 8.4 that the entries in Tσ̂→y =: Hσ̂ (y) are either constants
or linear combinations of entries in y. By using the fact that affine images and convex hulls
commute, we can therefore writenoΛσ̂ := co Tσ̂→π̂ : π̂ ∈ Π≽j = co Hσ̂ (Π≽j ) = Hσ̂ (co Π≽j ) = Hσ̂ (Q≽j ).
So, we have just proved the following characterization of the set Λσ̂ .
Lemma 8.2. For all sequences σ̂ = (j, a) ∈ Σ∗ , Λσ̂ is the image of Q≽j under the affine
mapping Hσ̂ , that is,noΛσ̂ = Tσ̂→xσ̂ : xσ̂ ∈ Q≽j .
Let Aσ̂ ∈ ℝ(Σ×Σ)×Σ≽j , B ∈ ℝΣ×Σ be such that the affine map Hσ̂ has representationHσ̂ (y) = Aσ̂ y + b∀ y ∈ ℝΣ≽j .

By leveraging the regret circuit for affine transformations we developed in Section 4.2.4, Lemma 8.2
immediately implies that a no-external-regret algorithm Rσ̂ for Λσ̂ can be constructed from any
no-external-regret R̃Q, σ̂ for Q≽j (for example, CFR, §4.3), as done in Algorithm 8.1.
Algorithm 8.1: No-external-regret algorithm Rσ̂ for set Λσ̂ := {Tσ̂→xσ̂ : xσ̂ ∈ Q≽j }Data: • σ̂ = (j, a) ∈ Σ∗ trigger sequence• R̃Q, σ̂ no-external-regret algorithm for set Q≽j (e.g., CFR, Section 4.3)
function NextElement(M(t) ∈ ℝΣ×Σ )(t)2xσ̂ ← R̃Q, σ̂ .NextElement(A⊤σ̂ M(t) )
1

3

(t)

return Tσ̂→x(t) , represented in memory implicitly through the vector xσ̂σ̂function ObserveUtility(U(t) ∈ ℝΣ×Σ )5
R̃Q, σ̂ .ObserveUtility(A⊤σ̂ U(t) )4Algorithm 8.1 can be instantiated with any no-regret algorithm R̃Q, σ̂ for the set of sequenceform strategies Q≽j . The following proposition formalizes the cumulative regret guarantee whenR̃Q, σ̂ is set to the CFR algorithm (Section 4.3), which so far has arguably been the most widelyused no-regret algorithm for sequence-form strategy spaces.
Proposition 8.1. Let σ̂ = (j, a) ∈ Σ∗ be any trigger sequence. Consider the no-regret algorithmRσ̂ (Algorithm 8.1), where R̃Q, σ̂ is set to be the CFR no-regret algorithm (Section 4.3) with RM

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

165

as the choice of local no-regret algorithm for each decision node. Then, upon observing any
sequence of linear utility matrices U(1) , . . . , U(T ) ∈ ℝΣ×Σ , the external regret cumulated by
the elements T(1) := Tσ̂→x1σ̂ , . . . , T(T ) := Tσ̂→xT output by Rσ̂ satisfiesσ̂(T )Regσ̂ :=

T
X
√
(t)
∗
(t)
max
⟨U
,
T
−
T
⟩
⩽
D
|Σ
|
T,
≽j
∗

T ∈Λσ̂

t=1

where D is the range of U(1) , . . . , U(T ) , i.e., any constant such that maxT∈Λσ̂ ⟨U(t) , T⟩ ⩽ D
for all t = 1, . . . , T . Furthermore, the NextElement and the ObserveUtility operations run inO(|Σ≽j |) time.

Proof. From Theorem 4.3, the regret cumulated by Rσ̂ upon observing linear utility matrices
U(1) , . . . , U(t) equals the regret cumulated by the CFR algorithm upon observing linear utility
functions A⊤σ̂ U(t) . Furthermore, the range of A⊤σ̂ U(t) satisfies the inequality
max ⟨A⊤σ̂ U(t) , x⟩ = max ⟨U(t) , Aσ̂ x⟩ ⩽ D,

x∈Q≽j

x∈Q≽j

since Aσ̂ x ∈ Λσ̂ .
So, applying the regret bound of the CFR algorithm,



Xq

(T )
Regσ̂ ⩽ D
j ′ ≽j√|Aj ′ | T ⩽ D

X


√
√
|Aj ′ | T = D|Σ≽j | T ,j ′ ≽jcompleting the proof of the regret bound.
We now look at the complexity analysis. First, we observe that Hσ̂ (y) maps each entry of y
to exactly one entry of the resulting canonical trigger deviation matrix Tσ̂→y . Hence, Aσ̂ has
exactly |Σ≽j | nonzero entries, and therefore the products A⊤σ̂ M(t) , A⊤σ̂ U(t) require O(|Σ≽j |)operations each. Since CFR’s NextElement and ObserveUtility operations both run in linear
time in |Σ≽j |, we conclude that the NextElement of Algorithm 8.1 required a total of O(|Σ≽j |)operation.

No-external-regret algorithm for the convex hull of the Λσ̂ We have seen in Section 4.2.3
that a no-regret algorithm for a composite set of the form co{X1 , . . . , Xm } can be constructed by
combining any individual no-regret algorithms for X1 , . . . , Xm through the convex hull regret
circuit.
Hence, we apply the construction described in Algorithm 4.2 to obtain our no-regret algorithm
R for the set co Ψ = co{Λσ̂ : σ̂ ∈ Σ} starting from the no-regret algorithms Rσ̂ (Algorithm 8.1),

166§8.3. CONSTRUCTION OF NO-TRIGGER-REGRET DYNAMICS∗

one for each sequence σ̂ ∈ Σ∗ , as well as any no-regret algorithm R∆ for the simplex ∆Σ .
Pseudocode is given in Algorithm 8.2.
Algorithm 8.2: No-regret algorithm R̃ for the set co Ψ = co{Λσ̂ : σ̂ ∈ Σ}
Data: • Rσ̂ , one per σ̂ ∈ Σ∗ : no-regret algorithm for Λσ̂ , defined in Algorithm 8.1∗• R∆ no-regret algorithm for ∆Σ (e.g., regret matching (Hart and Mas-Colell, 2000))function NextElement(M(t) ∈ ℝΣ×Σ )2for σ̂ ∈ Σ∗ do
3
Tσ̂→x(t) ← Rσ̂ .NextElement()
σ̂


(t)
4
m∆ ← ⟨M(t) , Tσ̂→x(t) ⟩1[▷ See Algorithm 8.1]

σ̂∈Σ∗
(t)
λ ← R̃∆ .NextElement(m∆ )X(t)returnλ(t) [σ̂] Tσ̂→x(t) , represented in memory as list {(λ(t) [σ̂], xσ̂ )}σ̂∈Σ∗
σ̂
σ̂∈Σ∗
σ̂

5
6

(t)

Σ∗

function ObserveUtility(U(t) ∈ ℝΣ×Σ )8for σ̂ ∈ Σ∗ do
9Rσ̂ .ObserveUtility(U(t) )


(t)
10
u∆ ← ⟨U(t) , Tσ̂→x(t) ⟩
7

σ̂

11

[▷ See Algorithm 8.1]

σ̂∈Σ∗

(t)
R̃∆ .ObserveUtility(u∆ )Theorem 8.2. Consider the no-regret algorithm R̃ (Algorithm 8.2), where R∆ is set to theregret matching algorithm, and Rσ̂ is instantiated as described in Proposition 8.1. Uponobserving any sequence of linear utility matrices U(1) , . . . , U(T ) ∈ ℝΣ×Σ , the regret cumulatedby the transformations T(1) , . . . , T(T ) ∈ co Ψ output by R̃ satisfies(T )Regco Ψ := max∗

T ∈co ΨT
X√
⟨U(t) , T∗ − T(t) ⟩ ⩽ 2D |Σ| T ,t=1

where D is any constant such that maxT∈co Ψ ⟨U(t) , T⟩ ⩽ D for all t = 1, . . . , T . Furthermore,
the NextElement and the ObserveUtility operations run in O(|Σ|2 ) time.

Proof. At all times t, ⟨M(t) , Tσ̂→x(t) ⟩, ⟨U(t) , Tσ̂→x(t) ⟩ ⩽ D is upper bounded by D. Hence,σ̂σ̂from the known regret bound of the regret matching algorithm (Hart and Mas-Colell, 2000;
Zinkevich, Bowling, Johanson, and Piccione, 2007), the regret cumulated by R∆ after T
iterations is upper bounded as§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM(T )Reg∆ ⩽ D

p

167

√
√
|Σ∗ | T ⩽ D|Σ| T .

On the other hand, the regret bound in Proposition 8.1 shows that, for all σ̂ = (j, a) ∈ Σ∗ ,
√
(T )
(T )
the regret Regσ̂ cumulated by Rσ̂ is upper bounded as Regσ̂ ⩽ D|Σ≽j | T . Applying
Theorem 4.2 together with the fact that |Σ≽j | ⩽ |Σ| for all j ∈ J , yields the regret bound in thestatement.
Furthermore, the regret matching algorithm produces elements in O(|Σ∗ |) time, while each
iteration of the loop over Σ∗ requires O(|Σ≽j |) time from Proposition 8.1.

8.3.3

Complete algorithm and analysis

Having established the existence of an efficient no-external-regret algorithm for the set of canonicaltrigger deviation matrices co Ψ (Algorithm 8.2, R̃), we can now invoke the reduction from Φ-regret
to external regret seen in Section 3.2.3 to establish no-trigger-regret dynamics.
Remark 8.1. To complete the construction, it is important to verify that for each canonicaltrigger deviation function T(t) output by R̃, a fixed point strategy x(t) = T(t) x(t) exists andcan be computed.
Existence is straightforward: as x 7→ T(t) x is a continuous function from Q to itself, a
fixed point must exist by Brouwer’s fixed-point theorem.
Furthermore, since Q is a polytope, we remark that a fixed point of each matrix T(t) can
be computed in polynomial time using linear programming. A more efficient algorithm is
proposed in an appendix to this chapter, Section 8.A.

The final algorithm is presented pictorially in Figure 8.2 and in pseudocode in Algorithm 8.3.
Algorithm 8.3: No-trigger-regret algorithm for sequence-form strategy polytope Q
Data: R̃ no-regret algorithm for Ψ, defined in Algorithm 8.2
x(0) ← 0 ∈ ℝΣ(t)
2 function NextElement(m∈ ℝΣ )
X (t)
(t)
3
T =
λσ̂ Tσ̂→x(t) ∈ co Ψ ← R̃.NextElement(m(t) ⊗ x(t−1) )
1

σ̂∈Σ∗
4
5

[▷ Algorithm 8.2]σ̂x(t) ∈ Q ← FixedPoint(T(t) )return x(t)function ObserveUtility(u(t) ∈ ℝΣ )7
R̃.ObserveUtility(u(t) ⊗ x(t) )[▷ See Remark 8.1]

6[▷ See Algorithm 8.2]

From the guarantees of the different pieces combined, we obtain the following.

168§8.4. FINAL REMARKS

No-trigger-regret algorithm for Q (Algorithm 8.3)
R̃ (Algorithm 8.2)
R̃1 (Algorithm 8.1)

R̃Q, 1u(t) ∈ ℝΣ

U (t)T1→x(t)
1

T(t)

R∆

Fixed
point

x(t) ∈ Q

R̃m (Algorithm 8.1)

R̃Q, mTm→x(t)m

Figure 8.2: Pictorial depiction of our no-(co Ψ)-reget algorithm for the set of sequence-form strategies
Q. For notational convenience we let Σ∗ := {1, . . . , m}.

Theorem 8.3. Algorithm R, defined in Algorithm 8.3, is a no-(co Ψ)-regret algorithm forthe set of sequence-form strategies Q, whose cumulative co Ψ-regret upon observing any
sequence of linear utility functions u(1) , . . . , u(T ) ∈ ℝΣ satisfies√
Reg(T ) := max⟨u(t) , T∗ (x(t) ) − x(t) ⟩ ⩽ 2D|Σ| T ,∗T ∈co Ψwhere D is any constant such that maxx∈Q ⟨u(t) , x⟩ ⩽ D for all t = 1, . . . , T . Furthermore,
ObserveUtility requires time O(|Σ|2 ), and NextElement requires O(|Σ|2 ) operations, plus thetime required to compute a fixed point of T(t) (see Remark 8.1). In particular, if the algorithm
described in the appendix (Section 8.A) is used to compute each fixed point, NextElement
P
requires O(|Σ|2 + j∈J FP(|Aj |)) time at all t, where FP(m) is the time required to find a
fixed point of an m × m stochastic matrix.

8.4

Final remarks

We conclude by pointing out that the material in this chapter can be extended in several directions,though for space and organizational reasons we decided not to develop these extensions in detailin this dissertation.
For one, the whole machinery presented in this chapter applies to extensive-form coarse correlated
equilibria (EFCCE) (Farina, Bianchi, and Sandholm, 2020), an intermediate solution concept thatsits between EFCE and coarse-correlated equilibria. Since an EFCE is always an EFCCE, thealgorithm we presented in this chapter can be used—without modifications—to find an EFCCE§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

169

as well. However, as shown by Anagnostides, Farina, Kroer, Celli, and Sandholm (2022), when
an EFCCE is sought it is possible to slightly simplify the construction of the algorithm to obtainslightly faster per-iteration complexity.
Another direction is obtaining learning dynamics that can recover EFCE at the near-optimalrate ÕT (1/T ), similarly to what was done in Chapter 6. This is possible, but is significantly harder
due to the presence of the fixed point computation in EFCE (Anagnostides, Farina, Kroer, Celli,and Sandholm, 2022). We refer the reader interested in this direction to the work by Anagnostides,
Daskalakis, Farina, Fishelson, Golowich, and Sandholm (2022) and Anagnostides, Farina, andSandholm (2023).

8.A Appendix: Inductive computation of fixed points of trigger
deviation matrices
As mentioned in Section 8.3.3, a fixed point of any transformation T ∈ co Ψ can be computed in avariety of ways, including linear programming. In this appendix, we show that the combinatorial
structure of co Ψ enables us to find a fixed point of any T ∈ co Ψ as the result of the solutionof a sequence of fixed point computations for smaller stochastic matrices. This structural
understanding will enable us to conclude that any transformation T ∈ co Ψ admits a fixed point
x = Tx ∈ Q that can be computed in time quadratic in the number of sequences Σ.
As a key step in our algorithm, we will use the following well-known result about stationary
distributions of stochastic matrices.
Fact 8.1. Any stochastic matrix A ∈ 𝕊d admits a fixed point Ax = x ∈ ∆d . Furthermore, sucha fixed point can be computed in polynomial time in d.

Several specialized algorithms are known for computing fixed points of stochastic matrices(see, e.g., Paige, Styan, and Wachter (1975) for a comparison of eight different methods). Since
the particular choice of method is irrelevant, in this dissertation we will make the following
assumption.
Assumption 8.1. Given any m ∈ ℕ⩾1 , we assume access to an oracle for computing a fixed
point of any m × m stochastic matrix A. Furthermore, we assume that the oracle requires atmost FP(m) time in the worst case to compute any such fixed point.
Our algorithm for computing a fixed point of T ∈ co Ψ requires that the transformation T be
expressed as a convex combination of elements from the sets {Λσ̂ }σ̂∈Σ∗ , that is, an expression of

170§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICES

the formT=

X

λσ̂ Tσ̂→xσ̂ ,

σ̂∈Σ∗

X

with

λσ̂ = 1, λσ̂ ⩾ 0, xσ̂ ∈ Q≽j∀ σ̂ = (j, a) ∈ Σ∗ ,

(8.8)

σ̂∈Σ∗

in accordance with the characterization of co Ψ established by (8.7) and Lemma 8.2. Note that
our no-regret algorithm R̃ for the set co Ψ (Algorithm 8.2) already outputs transformations T
expressed in the form above.
Our algorithm operates incrementally, constructing a fixed point sequence-form strategy x forT decision node by decision node, in a top down fashion. To formalize this notion of top-down
construction, we will make use of the two following definitions.
Definition 8.5. Let J ⊆ J be a subset of decision nodes. We say that J is a trunk of J if, forevery j ∈ J, all predecessors of j (that is, all j ′ ∈ J such that j ′ ≺ j) are also in J.

Example 8.2. Consider again the small tree-form decision process of Example 2.7, reproduced
below.
A
1

2
D
789

B

C

3 4

5 6

In this case, the sets { }, {A}, {A, B}, {A, C}, {A, D}, {A, B, D}, {A, C, D}, {A, B, C, D} exhaust
all the possible trunks.
Conversely, sets {B} and {B, D} are not trunks, because A ≺ B and yet A is not in the sets.
Similarly, {C} and {C, D} are not trunks, since A ≺ C and yet A is not in the sets.

Definition 8.6. Let J ⊆ J be a trunk of J (Definition 8.5), and T ∈ co Ψ. We say that avector x ∈ ℝΣ
⩾0 is a J-partial fixed point of T if it satisfies the sequence-form constraints at allj ∈ J, that is,x[∅] = 1,x[pj ] =Xa∈Aj

and furthermore

x[ja] ∀ j ∈ J,

(8.9)

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM(Tx)[∅] = x[∅] = 1,(Tx)[ja] = x[ja] ∀ j ∈ J, a ∈ Aj .

171

(8.10)

It follows from Definition 8.6 that a J -partial fixed point of T is a vector x ∈ Q such thatx = Tx. The following simple lemma establishes a { }-partial fixed point for any transformation
T ∈ co Ψ.

P
Lemma 8.3. Let T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ be any transformation in co Ψ, expressed as in (8.8).
Then, the vector x0 ∈ ℝΣ
⩾0 , whose entries are all zeros except for x0 [∅] = 1, is a { }-partial
fixed point of T.

Proof. Condition (8.9) is straightforward. So, we focus on (8.10). Fix any σ̂ = (j, a) ∈ Σ∗ . The
definition of Tσ̂→x̂ , given in (8.5), implies thatTσ̂→x̂σ̂ [σr , ∅] =
1

if σr = ∅0otherwise∀ σr ∈ Σ.

Consequently, Tσ̂→x̂σ̂ (x0 ) = Tσ̂→x̂σ̂ x0 = x0 (from expanding the matrix-vector multipliP
cation). So, T(x0 ) = σ̂∈Σ∗ λσ̂ Tσ̂→x̂σ̂ (x0 ) = x0 and in particular T(x0 )[∅] = x0 [∅] = 1.
So, (8.10) holds, as we wanted to show.

The key result that powers our algorithm to compute a fixed point of any T ∈ co Ψ is thata J-partial fixed point can be cheaply promoted to be a (J ∪ {j ∗ })-partial fixed point, wherej ∗ ∈ J \ J is any decision node whose predecessors are all in J. Algorithm 8.4 below gives animplementation of such a promotion: Extend(T, J, j ∗ , x) starts with a J-partial fixed point x ofT, and modifies all entries x[j ∗ a], a ∈ Aj∗ , so that x becomes a (J ∪ {j ∗ })-partial fixed point.
Therefore, at a conceptual level, one can repeatedly invoke Extend, growing the trunk J one
decision node at a time until J = J , starting from the { }-partial fixed point x0 described in
Lemma 8.3.

§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICES

172

Algorithm 8.4: Extend(T, J, j ∗ , x)PInput
: • T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ ∈ co Ψ, represented as list {(λσ̂ , xσ̂ )}σ̂∈Σ∗• J ⊆ J trunk• j ∗ ∈ J decision node not in J such that its immediate predecessor is in J
• x ∈ ℝΣ
⩾0 J-partial fixed point of T∗
Output : • x′ ∈ ℝΣ⩾0 (J ∪ {j })-partial fixed point of T1

σp ← pj ∗2Let r ∈ ℝ⩾0j∗ be the vector whose entries are defined, for all a ∈ Aj∗ , as
X X
r[a] :=
λj ′ a′ xj ′ a′ [j ∗ a] x[j ′ a′ ]A

j ′ ≼σp a′ ∈Aj ′3

Let W ∈ x[σp ] · 𝕊Aj∗ be the matrix whose entries are defined, for all ar , ac ∈ Aj∗ , as




X
W[ar , ac ] := r[ar ] + λj ∗ ac xj ∗ ac [j ∗ ar ] + 1 −λσ̂  𝟙ar =ac  x[σp ]σ̂∈Σ∗ , σ̂≼j ∗ ac

if x[σp ] = 0 thenA5
w ← 0 ∈ ℝ⩾0j∗
6 else7b ∈ ∆Aj∗ ← fixed point of stochastic matrix x[σ1 p ] W
4w ← x[σp ] b

8
′

x ←x
10 for a ∈ Aj∗ do
11x′ [j ∗ a] ← w[j ∗ a]′
12 return x
9Before giving a proof of correctness and an analysis of the complexity of Extend, we illustrate
an application of the algorithm in the simple extensive-form game of Example 2.5.
Example 8.3.
Consider the simple extensive-form game of Example 2.5, and recall the three deviation
matrices TA1→π̂A1 , TA2→π̂A2 , TB3→π̂B3 considered in Example 8.1. We will illustrate two
applications of Extend, with respect to the transformation
T :=

1
1
1
TA1→π̂A1 + TA2→π̂A2 + TB3→π̂B3 ∈ co Ψ.
2
3
6• In the first application, consider the trunk J = { }, decision node j ∗ = A, and the{ }-partial fixed point described in Lemma 8.3, that is, the vector x whose components
are all 0 except for the entry corresponding to the empty sequence ∅, which is set to 1.

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

173

In this case, σp = pj ∗ (Line 1 of Algorithm 8.4) is the empty sequence. Since no decisionnode j ′ can possibly satisfy j ′ ≼ σp , the vector r defined on Line 2 is the zero vector.
Consequently, the matrix W defined on Line 3 is

W=

1

2

1/2

1/3

1/2

2/3

!

1
2

which is a stochastic matrix. A fixed point for W is given by the vector b := (2/5, 3/5) ∈∆{2,3} . So, the vector x′ returned by Extend is given byx′ [∅] = 1,

x′ [A1] =2,
5x′ [A2] =3
5and zero entries everywhere else. Direct computation reveals that x′ is indeed a{A}-partial fixed point of T.
• In the second application of Extend, we start from the {A}-partial fixed point that wecomputed in the previous bullet point, and extend it to a {A, D}-partial fixed point.
Here, j ∗ = D, and so σp = A2). The only j ′ ≼ σp is A, and so the vector r defined on
Line 2 isr[7] =

1,
5

r[8] = 0,r[9] = 0.

Consequently, the matrix W defined on Line 3 is
7

8

9

3/5

1/5

1/5


W= 0
0

2/5


0 8
2/5
9



0



7

1
W = 5/3 W
As expected, W ∈ 3/5 𝕊{D7,D8,D9} = x[A2] 𝕊{D7,D8,D9} . A fixed point for x[A2]′is given by the vector b := (1, 0, 0). So, the vector x returned by Extend is given byx′ [∅] = 1, x′ [A1] =2
3
3, x′ [A2] = , x′ [D7] = , x′ [D8] = 0, x′ [D9] = 0,555

and zero entries everywhere else. Once again, direct computation reveals that x′ isindeed a {A, D}-partial fixed point of T.

174§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICESP
Proposition 8.2. Let T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ be a linear transformation in co Ψ expressed as∗in (8.8), x ∈ ℝΣ
⩾0 be a J-partial fixed point of T, and j ∈ J be a decision node not in J such thatits immediate predecessor is in J. Then, Extend(T, J, j ∗ , x), given in Algorithm 8.4, computesa (J ∪ {j ∗ })-partial fixed point of T in time upper bounded by O(|Σ| |Aj∗ | + FP(|Aj∗ |)).

The proof of Proposition 8.2 is deferred until the end of the section. Proposition 8.2 immediately
implies that a fixed point for T ∈ co Ψ can be computed by repeatedly invoking Extend to grow thetrunk J one decision node at a time, until J = J , starting from the { }-partial fixed point x0 ∈ ℝΣ⩾0
introduced in Lemma 8.3. This leads to Algorithm 8.5, whose correctness and computational
complexity is a straightforward corollary of Proposition 8.2.
Algorithm 8.5: FixedPoint(T)P
Input: T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ ∈ co Ψ transformation
Output : x ∈ Q such that x = Txx ← 0 ∈ ℝΣ , x[∅] ← 1
J ← {}[8.b]
3 for j ∈ J in top-down orderdo4x ← Extend(T, J, j, x)
5J ← J ∪ {j}6 return x
1

2[▷ See Algorithm 8.4]

P
Corollary 8.1. Let T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ be a transformation in co Ψ expressed as in (8.8).
Then, Algorithm 8.5 computes a fixed point Q ∋ x = Tx in time upper bounded asP
O(|Σ|2 + j∈J FP(|Aj |)).

Proof of Proposition 8.2
In order to prove correctness of Extend in Proposition 8.2, we will find useful the following
technical lemma.
P
Lemma 8.4. Let T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ be any linear transformation in co Ψ expressed asin (8.8). Then, for all σ ∈ Σ,

[8.b] That is, according to a pre-order tree traversal: if j ≺ j ′ , then j appears before j ′ in the iteration order.

§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

175




X

(Tx)[σ] = 1 −λσ̂ x[σ] +∗X X

λj ′ a′ xj ′ a′ [σ] x[j ′ a′ ].

j ′ ≼σ a′ ∈Aj ′σ̂∈Σ , σ̂≼σ

Proof. Fix any trigger sequence σ̂ = j ′ a′ ∈ Σ∗ . By expanding the matrix-vector multiplication
between Tσ̂→xσ̂ (Definition 8.4) and x, we have that for all σ ∈ Σ,Tσ̂→xσ̂ (x)[σ] = x[σ]𝟙σ̸≽σ̂ + xσ̂ [σ]x[σ̂]𝟙σ≽j ′ .

(8.11)

Therefore, for all σ ∈ Σ,(Tx)[σ] =Xσ̂∈Σ∗λσ̂ (x[σ]𝟙σ̸≽σ̂ + xσ̂ [σ]x[σ̂]𝟙σ≽j ′ )σ̂=j ′ a′ ∈Σ∗




=

X

λσ̂ Tσ̂→xσ̂ (x)[σ] =Xλσ̂ x[σ] +σ̂∈Σ∗ , σ̸≽σ̂= 1 −X X

λj ′ a′ xj ′ a′ [σ] x[j ′ a′ ]j ′ ≼σ a′ ∈Aj ′Xσ̂∈Σ∗ , σ̂≼σ

λσ̂ x[σ] +X X

λj ′ a′ xj ′ a′ [σ] x[j ′ a′ ],j ′ ≼σ a′ ∈Aj ′as we wanted to show.
We are now ready to prove Proposition 8.2, which is restated below for the reader’s convenience.
P
Proposition 8.2 (Restated). Let T = σ̂∈Σ∗ λσ̂ Tσ̂→xσ̂ be a linear transformation in co Ψ∗
expressed as in (8.8), x ∈ ℝΣ
⩾0 be a J-partial fixed point of T, and j ∈ J be a decisionnode not in J such that its immediate predecessor is in J. Then, Extend(T, J, j ∗ , x), given
in Algorithm 8.4, computes a (J ∪ {j ∗ })-partial fixed point of T in time upper bounded byO(|Σ| |Aj∗ | + FP(|Aj∗ |)).
Proof. We break the proof into four parts. In the first part, we analyze the sum of the entries ofvector r defined in Line 2 of Algorithm 8.4. In the second part, we prove that x[σ1 p ] W ∈ 𝕊Aj∗ ,as stated in Line 3. In the third part, we show that the output x′ of the algorithm is indeed a(J ∪ {j ∗ })-partial fixed point of T. Finally, in the fourth part we analyze the computational
complexity of the algorithm.
Part 1: Sum of the entries of r In this first part of the proof, we study the sum of the entries ofthe vector r defined on Line 2 in Algorithm 8.4. By hypothesis, the immediate predecessor of j ∗is in J. So, because x is a J-partial fixed point, the sequence σp := pj ∗ satisfies (Tx)[σp ] = x[σp ].

176§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICES

Hence, expanding the term (Tx)[σp ] using Lemma 8.4, we conclude that



X

1 −

λσ̂ x[σp ] +σ̂∈Σ∗ , σ̂≼σpXXλj ′ a′ xj ′ a′ [σp ] x[j ′ a′ ] = x[σp ].

j ′ ≼σp a′ ∈Aj ′By rearranging terms, we have



X

X

λσ̂ x[σp ] =∗X

λj ′ a′ xj ′ a′ [σp ] x[j ′ a′ ].

(8.12)j ′ ≼σp a′ ∈Aj ′σ̂∈Σ , σ̂≼σp

On the other hand, since xj ′ a′ ∈ Q≽j ′ for all j ′ ≼ σp , a′ ∈ Aj ′ , the sequence-form(probability-mass-conservation) constraints (2.2) imply thatxj ′ a′ [σp ] =X

xj ′ a′ [j ∗ a].

∀ j ′ ≼ σp , a′ ∈ Aj ′ .

a∈Aj∗Hence, plugging the previous equality into (8.12), we obtain



X



λσ̂ x[σp ] =σ̂∈Σ∗ , σ̂≼σp

X

X

X

λj ′ a′ xj ′ a′ [j ∗ a] x[j ′ a′ ]j ′ ≼σp a′ ∈Aj ′ a∈Aj∗




=

X

X


a∈Aj∗

=

X

X

j ′ ≼σλj ′ a′ xj ′ a′ [j ∗ a] x[j ′ a′ ]

′p a ∈Aj ′

r[a],

a∈Aj∗

where the last equality follows from recognizing the definition of r on Line 2 of Algorithm 8.4.
So, in conclusion,XX
r[a] = λσ̂ x[σp ].
(8.13)a∈Aj∗σ̂∈Σ∗ , σ̂≼σpPart 2: W belongs to x[σp ] · 𝕊Aj∗ In this second part of the proof, we will prove that allcolumns of the nonnegative matrix W, defined on Line 3 of Algorithm 8.4, sum to the samevalue x[σp ]. Fix any ac ∈ Aj∗ . Using the definition of W, the sum of the entries in the columnof W corresponding to action ac is177§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

X

W[ar , ac ] =ar ∈Aj∗Xr[ar ] + λj ∗ ac xj ∗ ac [j ∗ ar ]x[σp ]ar ∈Aj∗X+ 1 −λσ̂  𝟙ar =ac x[σp ]σ̂∈Σ∗ , σ̂≼j ∗ ac




X

=




X

λσ̂ x[σp ] + x[σp ] λj ∗ ac σ̂∈Σ∗ , σ̂≼σpxj ∗ ac [j ∗ ar ]

ar ∈Aj∗X+ 1 −λσ̂  x[σp ]σ̂∈Σ∗ , σ̂≼j ∗ ac




X

=




X

λσ̂ x[σp ] + x[σp ] λj ∗ ac + 1 −σ̂∈Σ∗ , σ̂≼σpλσ̂  x[σp ],σ̂∈Σ∗ , σ̂≼j ∗ ac

P
where we used (8.13) in the second equality, and the fact that ar ∈Aj∗ xj ∗ ac [j ∗ ar ] = 1 since
xj ∗ ac ∈ Q≽j ∗ (Definition 2.4), in the third. Using the fact that the set of all predecessors ofsequence (j ∗ , ac ) is the union between all predecessors of the parent sequence σp and {(j ∗ , ac )}itself, after rearranging terms we can writeXar ∈Aj∗W[ar , ac ] = 


X




X

λσ̂ x[σp ] + x[σp ] λj ∗ ac + 1 −σ̂∈Σ∗ , σ̂≼σ

λσ̂  x[σp ]σ̂∈Σ∗ , σ̂≼j ∗ ac

p




X

= x[σp ] 1 + λ(j ∗ ,ac ) +λσ̂ −σ̂∈Σ∗ , σ̂≼σ

p

X

λσ̂ 

σ̂∈Σ∗ , σ̂≼j ∗ ac

= x[σp ].
So, all columns of the nonnegative matrix W sum to x[σp ], and therefore W ∈ x[σp ] · 𝕊Aj∗ .
Part 3: x′ is a (J ∪ {j ∗ })-partial fixed point of T We start by arguing that x′ satisfies thesequence-form constraints (8.9) for all j ∈ J ∪{j ∗ }. The crucial observation is that Algorithm 8.4
only modifies the indices corresponding to sequences (j ∗ , a) for a ∈ Aj∗ and keeps all otherentries unmodified. In particular,
x′ [ja] = x[ja]∀ j ∈ J, a ∈ Aj .

Furthermore, because J is a trunk, the above equation implies thatx′ [pj ] = x[pj ]∀ j ∈ J.

(8.14)

178§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICES

Hence, using the hypothesis that x is a J-partial fixed point of T at the beginning of the call,
we immediately conclude that the constraints (8.9) corresponding to j ∈ J hold for vector x′ ,and the only condition that remains to be verified is thatXx′ [σp ] =x′ [(j ∗ , a)].

(8.15)a∈Aj∗

If x[σp ] = 0, then all entries x′ [(j ∗ , a)] are set to 0 (Line 5) and so (8.15) is trivially satisfied.
On the other hand, if x[σp ] ̸= 0, then x′ [j ∗ a] is set to the value x[σp ] b[a] (Line 8), and since b
belongs to the simplex ∆Aj∗ , (8.15) holds in this case too. So, x′ satisfies (8.9) for all j ∈ J ∪ {j ∗ }as we wanted to show.
We now turn our attention to conditions (8.10). From Lemma 8.4 it follows that (Tx)[σ]only depends on the values of x[j ′ a′ ] for j ′ ≼ σ, a′ ∈ Aj ′ . So, from (8.14) it follows that
T(x′ )[ja] = x[ja] = x′ [ja]∀ j ∈ J, a ∈ Aj ,and the only condition that remains to be verified is that
T(x′ )[j ∗ a∗ ] = x′ [j ∗ a∗ ]∀ a∗ ∈ Aj∗ .

(8.16)

Fix any a∗ ∈ Aj∗ . We break the analysis into two cases.
• If x[σp ] = 0, then w = 0 (Line 5) and therefore x′ [j ∗ a∗ ] = 0. Hence, to show that (8.16)
holds, we need to show that T(x′ )[j ∗ a∗ ] = 0. To show that, we start from applying
Lemma 8.4:
X

T(x′ )[j ∗ a∗ ] =X

λj ′ a′ xj ′ a′ [j ∗ a∗ ] x′ [j ′ a′ ].

j ′ ≼j ∗ a∗ a′ ∈Aj ′Now, using the fact that {j ′ ∈ J : j ′ ≼ [j ∗ a∗ ]} is equal to the disjoint union {j ′ ∈ J : j ′ ≼σp } ∪ {j ∗ }, and that x′ [(j ∗ , a′ )] = 0 for all a′ ∈ Aj∗ , we haveX

T(x′ )[j ∗ a∗ ] =X

λj ′ a′ xj ′ a′ [j ∗ a∗ ] x′ [j ′ a′ ].

(8.17)j ′ ≼σp a′ ∈Aj ′Since xj ′ a′ ∈ Q≽j ′ is a nonnegative vector, from Definition 2.4 it follows thatxj ′ a′ [σp ] =

X
a∈Aj∗

Hence, substituting (8.18) into (8.17),

xj ′ a′ [j ∗ a] ⩾ xj ′ a′ [j ∗ a∗ ].

(8.18)

179§8. UNCOUPLED LEARNING OF EXTENSIVE-FORM CORRELATED EQUILIBRIUM

T(x′ )[j ∗ a∗ ] ⩽XX

λj ′ a′ xj ′ a′ [σp ] x′ [j ′ a′ ]j ′ ≼σp a′ ∈Aj ′= T(x′ )[σp ] = x′ [σp ] = 0,
where the first equality follows again from Lemma 8.4, and the second equality follows
from the inductive hypothesis that x′ is a J-partial fixed point of T. Since x′ is anonnegative vector and T maps nonnegative vectors to nonnegative vectors, we conclude
that T(x′ )[j ∗ a∗ ] = 0 as we wanted to show.
• If x[σp ] ̸= 0, then b is a fixed point of the stochastic matrix x[σ1 p ] W, and therefore it
satisfiesX

W[a∗ , ac ] b[ac ] = x[σp ] b[a∗ ].

ac ∈Aj∗Hence, by using the fact that x′ [j ∗ a∗ ] = x[σp ] b[a∗ ] (Line 11), we can writeXx′ [j ∗ a∗ ] =W[a∗ , ac ] b[ac ].

ac ∈Aj∗By expanding the definition of W[a∗ , ac ] (Line 3) on the right-hand side
x′ [j ∗ a∗ ] =











X 
X 




r[a∗ ] + λj ∗ a xj ∗ a [j ∗ a∗ ] + 1 −


λσ̂
c
c



𝟙a∗ =ac x[σp ]b[ac ]σ̂∈Σ∗σ̂≼j ∗ ac

ac ∈Aj∗= r[a∗ ] + 1 −X

λσ̂ x′ [j ∗ a∗ ] +σ̂∈Σ∗ , σ̂≼j ∗ a∗X

λj ∗ ac xj ∗ ac [j ∗ a∗ ]x′ [j ∗ ac ],ac ∈Aj∗where in the second equality we used the fact that b ∈ ∆Aj∗ , and the fact that x′ [j ∗ a] =x[σp ] b[a] for all a ∈ Aj∗ (Line 11). Expanding the definition of r (Line 2),
x′ [j ∗ a∗ ] =Xj ′ ≼σXX

λj ′ a′ xj ′ a′ [j ∗ a∗ ] x[j ′ a′ ] + 1 −′p a ∈Aj ′λσ̂ x′ [j ∗ a∗ ]σ̂∈Σ∗ , σ̂≼j ∗ a∗X+

λj ∗ ac xj ∗ ac [j ∗ a∗ ]x′ [(j ∗ , ac )]ac ∈Aj∗= 1 −Xσ̂∈Σ∗ , σ̂≼j ∗ a∗λσ̂ x′ [j ∗ a∗ ] +XXj ′ ≼j ∗ a∗ a′ ∈Aj ′λj ′ a′ xj ′ a′ [j ∗ a∗ ] x′ [j ′ a′ ]

180

§8.A. APPENDIX: INDUCTIVE COMPUTATION OF FIXED POINTS OF TRIGGER DEVIATION
MATRICES= T(x′ )[j ∗ a∗ ],where we used the fact that the set {j ′ ∈ J : j ′ ≼ j ∗ a∗ } is equal to the disjoint union{j ′ ∈ J : j ′ ≼ σp } ⊔ {j ∗ } in the second equality, and Lemma 8.4 in the third equality.
Part 4: Complexity analysis In this part, we bound the number of operations required by
Algorithm 8.4.
• Line 2: each entry r[a] can be trivially computed in O(|Σ|) time by traversing all
predecessors of j ∗ . So, the vector r requires O(|Σ| |Aj∗ |) operations to be computed.
• Line 3: if ar = ac , then the number of operations required to compute W[ar , ac ]P
is dominated by the computation of σ̂≼(j ∗ ,ac ) λσ̂ , which requires O(|Σ|) operations.
Otherwise, if ar ̸= ac , the computation of W[ar , ac ] can be carried out in a constantnumber of operations. Hence, the computation of W[ar , ac ] for all ar , ac ∈ Aj∗ requires
O(|Σ| |Aj∗ | + |Aj∗ |2 ) time. Since |Aj∗ | ⩽ |Σ|, the total number of operations required tocompute all entries of W is O(|Σ| |Aj∗ |).
• Lines 4 to 8: if x[σp ] = 0, then the computation of w requires O(|Aj∗ |) operations. If,
on the other hand, x[σp ] ̸= 0, then the computation of w requires O(FP(|Aj∗ |) + |Aj∗ |)operation. Since clearly any fixed point oracle for a square matrix of order |Aj∗ | needs tospend time at least Ω(|Aj∗ |) time writing the output, O(FP(|Aj∗ |)+|Aj∗ |) = O(FP(|Aj∗ |)).
So, no matter the value of x[σp ], the number of iterations is bounded by O(FP(|Aj∗ |)).
• Line 11: finally, the algorithm spends O(|Aj∗ |) operations to fill out the entries of x.
Summing the number of operations of each of the different steps of the algorithm, we conclude
that each call to Extend(T, J, j ∗ , x) requires at most O(|Σ| |Aj∗ | + FP(|Aj∗ |)) operations.

Chapter 9

Geometry of correlated strategies,
and positive complexity results for
optimal EFCE
We have shown in Chapter 8 that one extensive-form correlated equilibrium can be obtained byfollowing simple, uncoupled no-trigger-regret dynamics. In this chapter we instead focus on thecomputation of an optimal (for example, social-welfare-maximizing) EFCE. Because an EFCE is bydefinition a correlated distribution over the Cartesian product of the strategies of the players, inorder to find one such optimal equilibrium point it is sufficient to have a concise characterizationof all such correlated distributions. That will be exactly the focus of this chapter:
Are there polynomially-sized descriptions of the set of all correlated strategies of players in animperfect-information extensive-form game?

9.1

Contributions

In this chapter, we greatly expand the set of games in which we prove the question can be
answered for the positive, obtaining the first positive complexity results around optimal correlated
equilibria in more than a decade. Specifically, we show the following:• In two-player games, correlated strategies have a polynomial description provided thata certain condition, which we coin triangle-freeness, is satisfied. We then show that thecondition holds, for example, when all chance moves are public, that is, both players observe
all chance moves. This result immediately implies that a social-welfare-maximizing EFCE
can be computed in polynomial time in the game tree size in these games. Furthermore, the182§9.2. PRELIMINARIES, NOTATION, AND PRIOR WORK

techniques that we introduce to establish the result might be of independent interest.
• As a byproduct of the proof technique, we show that in triangle-free games the polytope
of correlated strategies can be obtained via composition of the scaled extension operationthat we introduced in Chapter 4. This will set the stage for constructing a state-of-the-art
learning algorithm for computing optimal EFCE in Chapter 10.
• In Section 9.5 we mention that in games that are not triangle-free, we recently establisheda generalization of the result, proving that the polytope of correlated strategies can be
expressed as a linear transformation of a set obtained via a chain of scaled extension
operations. While the length of the chain is in the worst case exponential in the size ofthe game tree—this is unavoidable unless P = NP—we show that it can be bounded by afunction of a parameter of the game that intuitively measures the amount of asymmetricprivate information among the players, thereby leading to state-of-the-art parameterized
complexity results.
• Finally, in Section 9.6 we put to use the characterization of the polytope of correlated strategies
we develop in the chapter to empirically investigate the set of expected utilities that can bereached by EFCEs across nine games. In all cases, we observe that both the shape of thepolytope and the range of reachable payoffs are highly nontrivial. This empirically suggeststhat being able to search and optimize over the space of EFCE (rather than computing asingle EFCE without a priori guarantees over the social welfare, as in Chapter 8) is important.

9.2

Preliminaries, notation, and prior work

In this section, we define some notation to describe correlated strategies. We will focus on the caseof two correlating players, as that reduces the notational burden while exposing the complexity
in its generality. Extending the notation and concepts to more than two players is mechanical.
We will denote quantities that belong to the two correlating players using subscripts 1 and 2,
respectively.
A correlated distribution of play is a probability distribution over deterministic sequence-form
strategies Π1 × Π2 of the players. As summarized in Table 9.1, given two decision nodes
j1 ∈ J1 , j2 ∈ J2 , we say that j1 and j2 are connected, and write j1 ⇌ j2 , if there exists at least
one possible trajectory in the decision process is consistent with Player 1 acting at j1 and Player
2 acting at j2 . Equivalently, in game tree terms, j1 ⇌ j2 if and only if there exist nodes u1 ininformation set j1 and u2 in information set j2 , such that there exists a path from the root of thegame tree to u2 that passes through u1 , or vice versa. With the notation of connected decisionnodes, the notion of relevant sequence pairs can be introduced. We say that (σ1 , σ2 ) ∈ Σ1 × Σ2 form
a relevant sequence pair—denoted σ1 ▷◁ σ2 —if at least one of the sequences is the empty sequence,§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

183

SymbolDescription

j1 ⇌ j2

Σ1 ▷◁ Σ2

Connected decision nodes: if j1 ⇌ j2 , it is possible that the trajectory in thedecision process goes through both j1 ∈ J1 and j2 ∈ J2
Relevant sequence pair: σ1 ▷◁ σ2 if at least one between σ1 and σ2 is the empty
sequence ∅, or if σ1 = (j1 , a1 ), σ2 = (j2 , a2 ) with j1 ⇌ j2
Set of all relevant sequence pairs {(σ1 , σ2 ) ∈ Σ1 × Σ2 : σ1 ▷◁ σ2 }j1 ▷◁ σ2
σ1 ▷◁ j2

Shorthand for: (j1 , a1 ) ▷◁ σ2 for all a1 ∈ Aj1
Shorthand for: σ1 ▷◁ (j2 , a2 ) for all a2 ∈ Aj2

σ1 ▷◁ σ2

Table 9.1: Additional notation used when dealing with correlation of strategy spaces.

or if σ1 = (j1 , a1 ), σ2 = (j2 , a2 ) with j1 ⇌ j2 . Finally, we denote with the symbol Σ1 ▷◁ Σ2 the setof all relevant sequence pairs, and use the notation j ▷◁ σ to mean that (j, a) ▷◁ σ for all a ∈ Aj .

9.2.1

Polytope of correlation plans ΞConsider now a particular terminal state of the game. Let σ1 , σ2 be the last sequences that wereencountered by the two players. Clearly, σ1 ▷◁ σ2 because the underlying decision nodes must
be connected. In order to express the expected utility of such terminal state, it is important tomeasure the probability according to which strategies consistent with σ1 and σ2 were selected.
Let µ be the correlated distribution over deterministic sequence-form strategies for the agents.
Such a probability is given byXXµ[π, π ′ ].

(9.1)

′

π∈Π1 π ∈Π2
π[σ1 ]=1 π ′ [σ2 ]=1

Next, we extend (9.1) to any pair of relevant sequences.
Definition 9.1. Let µ ∈ ∆Π1 ×Π2 be a correlated distribution over deterministic strategies forthe correlating agents. We let f denote the function mapping µ to the vector
f (µ)[σ1 , σ2 ] :=XXµ[π, π ′ ]∀(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 .

(9.2)

π∈Π1 π ′ ∈Π2
π[σ1 ]=1 π ′ [σ2 ]=1

The function f just introduced defines a convenient mapping between a correlated distributionof play µ, an exponentially large object, to the vector f (µ), which only has a polynomial numberof entries in the sizes of the correlated strategy spaces. We call f (µ) the correlation plan equivalent
to µ. The set of all legal correlation plan is introduced next.

184§9.2. PRELIMINARIES, NOTATION, AND PRIOR WORKDefinition 9.2. The polytope of correlation plans Ξ is the image of f (Definition 9.1) as µ varies
over the set of all possible correlated distributions over Π1 × Π2 , that is,noΞ := Im f = f (µ) : µ ∈ ∆Π1 ×Π2 .

Since the function f (Definition 9.1) is linear and the probability simplex is a convex polytope,
it immediately follows that Ξ is indeed a convex polytope, as stated in the following lemma.
1 ▷◁Σ2
Lemma 9.1. The polytope of correlation plans Ξ is a convex polytope, and a subset of ℝΣ
.
⩾0Lemma 9.1 means good news: just like in the non-correlated case (Section 2.3), we canrepresent strategies as elements of low-dimensional (more precisely: with dimension bounded
as a polynomial in the size of the input decision spaces) convex polytope. However, unlike thenon-correlated case, in general the constraints that define Ξ are unknown. Furthermore, even if
the constraints were known, a polynomial number of constraints might not be enough to describe
Ξ. A simple computational complexity argument in the work by von Stengel and Forges (2008)shows that this hurdle cannot be avoided, unless P = NP.
Remark 9.1. Since f sums up distinct entries from the distribution µ, all entries of f (µ)
belong to [0, 1]. Hence, Ξ ⊆ [0, 1]Σ1 ▷◁Σ2 .

9.2.2

Optimal EFCE as a linear program

We now show that the concise representation defined by Ξ enables to express the set of all EFCEs
as a polytope whose number of constraints is polynomial whenever Ξ can be expressed via
polynomially many constraints. In order to elucidate the connection between the polytope of
all EFCEs and the polytope of correlation plans Ξ, we go back to the very definition of EFCE,
which was given in Section 8.2. According to the definition, a correlated strategy µ ∈ ∆Π1 ×Π2
is an EFCE when for any player i ∈ {1, 2}, there is no trigger sequence σ̂ = (j, a) ∈ Σ∗i andcontinuation strategy (deterministic or mixed) xσ̂ ∈ Qi, ≽j such that the (σ̂, xσ̂ )-trigger agent
for Player i attains more value in expectation than the player that always play according to thestrategies sampled from µ. As we show in Section 9.B, given any i ∈ {1, 2}, σ̂ = (j, a) ∈ Σ∗i , andcontinuation strategy xσ̂ ∈ Qi, ≽j , the difference in expected utility obtained by the (σ̂, xσ̂ )-trigger
agent is a bilinear function ξ ⊤ Ai,σ̂ xσ̂ of the correlated plan ξ ∈ Ξ that corresponds to µ, and xσ̂ .
Hence, by just using the definition of EFCE we get to the following intermediate result.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

185

Lemma 9.2. An EFCE ξ that maximizes the lineara objective c⊤ ξ is the solution to thefollowing nonlinear optimization problem:


max


 ξ
s.t.





c⊤ ξ
ξ∈Ξ
max

1
2

xσ̂ ∈Qi, ≽jξ ⊤ Ai,σ̂ xσ̂ ⩽ 0∀ i ∈ {1, 2}, σ̂ = (j, a) ∈ Σ∗i .

aWe remark that by the very construction of correlation plans, the expected utility of a player that plays according
to the strategy sampled from ξ is a linear function of ξ. Hence, linear objectives in ξ are in particular able to capture
any linear combination of the utilities of the players, including the sum of the utilities (that is, the social welfare).

We now show that the constraints 2 can be rewritten as linear constraints, by means of linear
programming duality. Indeed, letting Fi,≽j , f i,≽j denote the sequence-form constraints thatdefine Qi, ≽j , that is, Qi, ≽j = {xσ̂ : Fi,≽j xσ̂ = f i,≽j , xσ̂ ⩾ 0}, we have that the linear programs


 max

x

ξ ⊤ Ai,σ̂ xσ̂

s.t.




Fi,≽j xσ̂ = f i,≽jxσ̂ ⩾ 0,



 min

v
i,σ̂

σ̂

and

s.t.




f⊤
i,≽j v i,σ̂⊤F⊤
i,≽j v i,σ̂ ⩾ Ai,σ̂ ξv i,σ̂ ∈ ℝJi, ≽j

are dual to each other for all i ∈ {1, 2} and trigger sequences σ̂ = (j, a) ∈ Σ∗i , and therefore havethe same value. So, it follows that constraint 2 in the formulation of Lemma 9.2 holds if and only
if there exists v i,σ̂ ∈ ℝJi, ≽j such that⊤F⊤i,≽j v i,σ̂ ⩾ Ai,σ̂ ξand f ⊤i,≽j v i,σ̂ ⩽ 0.

Hence, we have the following.

Proposition 9.1. An EFCE ξ that maximizes the linear objective c⊤ ξ is the solution to thefollowing nonlinear optimization problem:

max


ξ




 s.t.

c⊤ ξ
1
2









3
4

ξ∈Ξ
f⊤
i,≽j v i,σ̂ ⩽ 0⊤F⊤i,≽j v i,σ̂ ⩾ Ai,σ̂ ξv i,σ̂ ∈ ℝJi, ≽j∀ i ∈ {1, 2}, σ̂ = (j, a) ∈ Σ∗i∀ i ∈ {1, 2}, σ̂ = (j, a) ∈ Σ∗i∀ i ∈ {1, 2}, σ̂ = (j, a) ∈ Σ∗i .

186§9.2. PRELIMINARIES, NOTATION, AND PRIOR WORK

9.2.3

The von Stengel-Forges polytope V

We now introduce a second important polytope. It lives in the same Euclidean space as Ξ but
unlike Ξ it is defined as the intersection of a polynomial number of linear constraints. This
polytope was introduced without name by von Stengel and Forges (2008)—we will refer to it as
the von Stengel-Forges polytope.
Definition 9.3 (von Stengel-Forges polytope). The von Stengel-Forges polytope is the convex
1 ▷◁Σ2polytope of non-negative vectors v ∈ ℝΣindexed over relevant sequence pairs, and⩾0defined as

V :=







1 ▷◁Σ2v ∈ ℝΣ
:⩾01v[∅, ∅] = 1

2

P

3

Pa∈Aj1
a∈Aj2





v[(j1 , a), σ2 ] = v[pj1 , σ2 ] ∀j1 ∈ J1 , σ2 ∈ Σ2 : j1 ▷◁ σ2 .
v[σ1 , (j2 , a)] = v[σ1 , pj2 ]∀j2 ∈ J2 , σ1 ∈ Σ1 : σ1 ▷◁ j2

We remark that, unlike the sequence-form strategy polytope, the constraints that define the
von Stengel-Forges polytope V (Definition 9.3) do not exhibit a natural hierarchical structure: thesame entry in a vector v ∈ V can appear in multiple constraints, and furthermore the constraints
will in general form cycles. This makes the problem of decomposing the structure of V as we didin Section 2.3 significantly more challenging.

9.2.4

von Stengel and Forges (2008)’s result for two-player games withoutchance

The following important inclusion was shown by von Stengel and Forges (2008), and holds ingeneral—that is, no matter the game.
Lemma 9.3. The von Stengel-Forges polytope is always a superset of the polytope ofcorrelation plans. In symbols, Ξ ⊆ V.

The reverse inclusion might or might not include. In their seminal paper, von Stengel andForges (2008) show that the reverse inclusion holds in all two-player games without chancemoves. From that statement, it follows that Ξ = V, and since V is a convex polytope defined by apolynomial number of linear constraints, optimization over the set of all EFCEs can be carried out
in polynomial time. For more than a decade, von Stengel and Forges (2008)’s result remained ourbest understanding of the classes of games in which V = Ξ. In this chapter we greatly extend theset of games for which the relationship holds.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

9.3

187

Characterization of the relationship between Ξ and V

We now give a complete characterization as to when that inclusion V ⊆ Ξ holds, by connecting itwith the integrality of the vertices of the von Stengel-Forges polytope. In doing so, we will findthe following lemma useful.
Lemma 9.4. Let 1(π1 ,π2 ) ∈ ∆Π1 ×Π2 denote the distribution over Π1 × Π2 that assigns mass 1
to the pair (π 1 , π 2 ), and mass 0 to any other pair of reduced-normal-form plans. Then,Ξ = co{f (1(π1 ,π2 ) ) : π 1 ∈ Π1 , π 2 ∈ Π2 }.
Proof. The “deterministic” distributions 1(π1 ,π2 ) are the vertices of ∆Π1 ×Π2 , so, in particular,∆Π1 ×Π2 = co{1(π1 ,π2 ) : π 1 ∈ Π1 , π 2 ∈ Π2 }.
Since by definition Ξ = Im f , and f is a linear function, the images (under f ) of the 1(π1 ,π2 )are a convex basis for Ξ, which is exactly the statement.
We are now ready to state our characterization.
Theorem 9.1. Let Γ be a two-player perfect-recall imperfect-information extensive-form
game, let V be its von Stengel-Forges polytope, and let Ξ be its polytope of correlation plans.
Then, Ξ = V if and only if all vertices of V have integer {0, 1} coordinates.
Proof. We prove the two implications separately.
(⇒) We start by proving that if Ξ = V, then all vertices of V have integer {0, 1} coordinates.
Since V = Ξ by hypothesis, from Lemma 9.4 we can write
V = co{f (1(π1 ,π2 ) ) : π 1 ∈ Π1 , π 2 ∈ Π2 }.
So, to prove this direction it is enough to show that f (1(π1 ,π2 ) ) has integer {0, 1}
coordinates for all (π 1 , π 2 ) ∈ Π1 × Π2 . To see that, we use the definition (9.2): each entry
in f (1(π1 ,π2 ) ) is the sum of distinct entries of 1(π1 ,π2 ) . Given that by definition 1(π1 ,π2 )has exactly one entry with value 1 and |Π1 × Π2 | − 1 entries with value 0, we conclude
that all coordinates of f (1(π1 ,π2 ) ) are in {0, 1}.
(⇐) We now show that if all vertices of V have integer {0, 1} coordinates, then V ⊆ Ξ. Thisis enough, since the reverse inclusion, V ⊇ Ξ, is trivial and already known von Stengel

188§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

and Forges, 2008. Let {v 1 , . . . , v n } be the vertices of V. To conclude that V ⊆ Ξ, we will
prove that v i ∈ Ξ for all i = 1, . . . , n. This is sufficient since V and Ξ are convex.
Let v ∈ {v 1 , . . . , v n } be any vertex of V. By hypothesis, v[σ1 , σ2 ] ∈ {0, 1} for all (σ1 , σ2 ) ∈Σ1 ▷◁ Σ2 . Because v satisfies the von Stengel-Forges constraints and furthermore v has{0, 1} entries by hypothesis, the two vectors q 1 , q 2 defined according to q 1 [σ1 ] = v[σ1 , ∅](σ1 ∈ Σ1 ) and q 2 [σ2 ] = v[∅, σ2 ] (σ2 ∈ Σ2 ) are pure sequence-form strategies. Now, let
π ∗1 and π ∗2 be the reduced-normal form plans corresponding to q 1 and q 2 , respectively.
We will show that v = f (1(π∗1 ,π∗2 ) ), which will immediately imply that v ∈ Ξ using
Lemma 9.4.
Since 1(π∗1 ,π∗2 ) has exactly one positive entry with value 1 in the position correspondingto (π ∗1 , π ∗2 ), by definition of the linear map f , for any (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 ,f (1(π∗1 ,π∗2 ) )[σ1 , σ2 ] = 𝟙σ1 ∈Π1 (σ1 ) · 𝟙σ2 ∈Π2 (σ2 ) .
So, using the known properties of pure sequence-form strategies, we obtainf (1(π∗1 ,π∗2 ) )[σ1 , σ2 ] = q1 [σ1 ] · q2 [σ2 ] = v[σ1 , ∅] · v[∅, σ2 ] = v[σ1 , σ2 ],where the last equality follows from Lemma 9.10. Since the equality holds for any(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , we have that v = f (1(π∗1 ,π∗2 ) ).

9.4

Scaled-extension-based structural decomposition for V

Theorem 9.1 shows that in all games for which the von Stengel-Forges polytope has integral
vertices, V = Ξ, and hance computing an optimal EFCE can be done in polynomial time. In thissection we will show that in games for which the von Stengel-Forges polytope V can be expressed
as a composition of scaled-extension operations (Definition 4.1) on probability simplexes, thevertices of V are integral. We later isolate a condition, which we coin scaled extension, which
implies the existence of such a scaled-extension-based decomposition. Finally, we will showthat all two-player games with public chance moves are automatically triangle-free (and not all
triangle-free have public chance moves), thereby greatly extending the set of games for which theequality V = Ξ is known to hold. The chain of implications that lead to result are summarized in
Figure 9.1.
Beyond enabling the proof that V = Ξ, we also remark that the existence of a scaled-extensionbased decomposition enables the construction of scalable no-external-regret algorithms for V
using the regret circuits construction for scaled extension we presented in Section 4.2.5. We willleverage this observation in Chapter 10.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

Thm. 9.3

Triangle-freeness(Definition 9.5)

Thm. 9.4

Scaled-extension-based
decomposition of V

Theorem 9.2

Integrality ofthe vertices of V

Theorem 4.4

Two-player gamewith public chance

189

Theorem 9.1

Efficient learning algorithm
for optimal EFCE (Chapter 10)

Ξ = V

Figure 9.1: Overview of the connections among this chapter’s results.

9.4.1

Examples and intuition

We give three examples of increasing complexity that capture the main intuition behind our
structural decomposition routine, and will set the stage for the rest of the theory to come.

Game tree

Example 9.1. We consider as examples the three small imperfect-information extensive-form
games whose game trees are depicted in the top row of Figure 9.2.

A
1

B
2

3

1 2

1 2

1 2

∅

1

1

4

1

4

3

4
4

3

A

1

4

D
1 2

∅

1 2

3

3 4

3 4

∅

1

2

3

4

1

4

4

4

4

1

C

3

2

1 2

∅

D

1 2

3 4

1

2

B
4
3 4

3

4

∅

3

1

3

2

2
2
3

3
4

2

C

2

2
3

2

1 2

B

1

C

∅

Correlation plan
fill-in order

A

4

3

3

3

3

4

2
4

Figure 9.2: Three examples of extensive-form games with increasingly complex information
partitions. As usual, the crossed nodes belong the chance player, the black round nodes
belong to Player 1, the white round nodes belong to Player 2, the gray round sets define
information sets, and the white squares denote terminal nodes (payoffs are omitted as they
are irrelevant). The numbers along the edges define the action names; for clarity we assign
unique action names at each information set.

First example Consider the leftmost game in Figure 9.2. The game starts with a chance node,

190§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

where two outcomes (say, heads or tails) are possible. After observing the outcome of
the chance node, Player 1 chooses between two actions (say, the “left” and the “right”
action). The choice as to whether to play the left or the right action can be different
based on the observed chance outcome. After Player 1 has played their action, Player 2
has to pick whether to play their left or right action—however, Player 2 does not observe
the chance outcome nor Player 1’s action. The chance outcome is not observed by Player
2, so, this is not a public-chance game.
The only information set C for Player 2 is connected to both information sets (denoted A
and B in Figure 9.2) of Player 1, so, all sequence pairs (σ1 , σ2 ) ∈ Σ1 × Σ2 are relevant.
Since Player 2 only has one information set, it is easy to incrementally generate the
von Stengel-Forges polytope. First, the fixed value 1 is assigned to v[∅, ∅] (step 1
in the fill-in order). Then, this value is split arbitrarily into the two (non-negative)
entries v[∅, C1], v[∅, C2] so that v[∅, C1] + v[∅, C2] = v[∅, ∅] in accordance with the
von Stengel-Forges constraints. This operation can be expressed using scaled extension
ash{(v[∅, ∅], v[∅, C1], v[∅, C2])} = {1} ◁ ∆2 ,where h is the identity function (step 2 in the fill-in order).
Then, v[∅, C1] is further split into v[A1, C1] + v[A2, C1] = v[∅, C1] and v[B3, C1] +v[B4, C1] = v[∅, C1], while v[∅, C2] is split into v[A1, C2] + v[A2, C2] = v[∅, C2] andv[B3, C2] + v[B4, C2] = v[∅, C2] (step 3 of the fill-in order). These operations can be
expressed as scaled extensions with ∆2 .
Now that the eight entries v[σ1 , σ2 ] for σ1 ∈ {A1, A2, B3, B4}, σ2 ∈ {C1, C2} have been
filled out, we fill in v[σ1 , ∅] for all σ1 ∈ {A1, A2, B3, B4} in accordance with the vonStengel-Forges constraint v[σ1 , ∅] = v[σ1 , C1] + v[σ1 , C2] (step 4 ). In this step, we are
not splitting any values, but rather we are summing already-filled-in entries in v toform new entries. Specifically, we can extend the set of partially-filled-in vectorsv = (v[∅, ∅], v[∅, C1], v[∅, C2], v[A1, C1], v[A2, C1],v[B3, C1], v[B4, C1], v[A1, C2], v[A2, C2], v[B3, C2], v[B4, C2])hwith the new entry v[A1, ∅] by using the scaled extension operation {v} ◁{1} where h isthe (linear) function that extracts the sum v[σ1 , C1] + v[σ1 , C2] from v. By doing so, we
have incrementally filled in all entries in v. Furthermore, by construction, we have that all
von Stengel-Forges constraints v[σ1 , ∅] = v[σ1 , C1] + v[σ1 , C2] (σ1 ∈ {∅, A1, A2, B3, B4})
and v[∅, σ2 ] = v[A1, σ2 ]+v[A2, σ2 ] = v[B3, σ2 ]+v[B4, σ2 ] (σ2 ∈ {C1, C2}) must hold. So,§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE191

the only two von Stengel-Forges constraints that we have ignored and might potentiallybe violated are v[∅, ∅] = v[A1, ∅] + v[A2, ∅] and v[∅, ∅] = v[B3, ∅] + v[B4, ∅]. This
concern is quickly resolved by noting that those constraints are implied by the otherones that we satisfy. In particular, by construction we havev[A1, ∅] + v[A2, ∅] = (v[A1, C1] + v[A1, C2]) + (v[A2, C1] + v[A2, C2])
= (v[A1, C1] + v[A2, C1]) + (v[A1, C2] + v[A2, C2])= v[∅, C1] + v[∅, C2]= v[∅, ∅],and an analogous statement holds for v[B3, ∅] + v[B4, ∅]. So, all constraints hold andthe scaled-extension-based decomposition is finished.
Remark 9.2. An approach that would start by splitting v[∅, ∅] into v[A1, ∅] +v[A2, ∅] = v[∅, ∅] and v[B3, ∅] + v[B4, ∅] = v[∅, ∅], thereby inverting the orderof fill-in steps 4 and 2 , would fail. Indeed, after filling v[σ1 , σ2 ] for all σ1 ∈{A1, A2, B3, B4}, σ2 ∈ {C1, C2}), there would be no clear way of guaranteeing thatv[A1, C1] + v[A2, C1] = v[B3, C1] + v[B4, C1] (= v[∅, C1]).

Second example We now consider a variation of the game from the first example, where
Player 2 observes the chance outcome but not the actions selected by Player 1. This
game, shown in the middle column of Figure 9.2, has public chance moves, because thechance outcome is observed by all players. In this game, not all pairs of information
sets are connected. In fact, only (A, C) and (B, D) are connected information set pairs.
Correspondingly, entries such as v[A1, D3], v[B4, C2], and v[A2, D4] are not defined in
the correlation plans for the game. This observation is crucial, and will set apart thisexample from the next one. To fill in any correlation plan, we can start by splittingv[∅, ∅] into v[A1, ∅] + v[A2, ∅] = v[∅, ∅] and v[B3, ∅] + v[B4, ∅] = v[∅, ∅] (fill-in
step 2 in the figure). Both operations can be expressed as a scaled extension ofpartially-filled-in vectors with ∆2 , scaled by the affine function that extracts v[∅, ∅] = 1
from the partially-filled-in correlation plans. Then, we further split those values into
entries v[σ1 , C1] + v[σ1 , C2] = v[σ1 , ∅] for σ1 ∈ {A1, A2} in accordance with the vonStengel-Forges constraint. Similarly, we will in v[σ1 , D3], v[σ1 , D4] for σ1 ∈ {B3, B4} inaccordance with the constraint v[σ1 , D3] + v[σ1 , D4] = v[σ1 , ∅] for σ1 ∈ {A1, A2} (fill-in
step 3 ). Finally, we recover the values of v[∅, σ2 ] for σ2 ∈ {C1, C2, D3, D4} with a scaled
extension with the singleton set {1} as discussed in the previous example. Again, it can be192§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

checked that despite the fact that we ignored the constraints v[∅, C1]+v[∅, C2] = v[∅, ∅]and v[∅, D3] + v[∅, D4] = v[∅, ∅], those constraints are automatically satisfied byconstruction. In this case, we were able to sidestep the issue raised in Remark 9.2
because of the particular connection between the information sets.
Third example Finally, we propose a third example in the third column of Figure 9.2. It is avariation of the first example, where Player 2 now observes Player 1’s action but not
the chance outcome. The most significant difference with the second example is thatthe information structure of the game is now such that all pairs of information sets ofthe players are connected. Hence, the problem raised in Remark 9.2 cannot be avoided.
Our decomposition algorithm cannot handle this example.

9.4.2

Triangle-freeness

The third example in the previous section highlights an unfavorable situation in which our
decomposition attempt based on incremental generation of the correlation plan. In order to
codify all situations in which that issue does not arise, we introduce the concept of rank of an
information set.
Definition 9.4. Let i ∈ {1, 2} be a player, and let −i denote the other player. Furthermore, letji ∈ Ji and σ−i ∈ Σ−i . The σ−i -rank of ji is the cardinality of the set{j−i ∈ J−i : j−i ⇌ ji , pj−i = σ−i }.

The issue in Remark 9.2 can be stated in terms of the ranks. Consider a relevant sequence pair
(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 and two connected information sets j1 ⇌ j2 such that pj1 = σ1 , pj2 = σ2 . If
the σ1 -rank of j2 and the σ2 -rank of j1 are both greater than 1, the issue cannot be avoided andthe decomposition will fail. For example, in the third example, where our decomposition fails,
all information sets have ∅-rank 2. We prove that such situations cannot occur, provided thegame satisfies the following condition, which can be verified in polynomial time in the size of the
EFG.
Definition 9.5 (Triangle-freeness). A two-player imperfect-information extensive-form
game is triangle-free if, for any choice of two distinct information sets j1 , j1′ ∈ J1 such that
pj1 = pj1′ = σ1 and two distinct information sets j2 , j2′ ∈ J2 such that pj2 = pj2′ = σ2 , it isnever the case that j1 ⇌ j2 ∧ j1′ ⇌ j2′ ∧ j1 ⇌ j2′ .
Using the definition of triangle-freeness, we can verify that the issue exposed in Remark 9.2
never occurs, as the next lemma clarifies.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

193

Lemma 9.5. Consider a triangle-free game, let (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , and let j1 ⇌ j2 be suchthat pj1 = σ1 , pj2 = σ2 . Then, at most one between the σ1 -rank of j2 and the σ2 -rank of j1 isstrictly larger than 1.
Proof. The results follows almost immediately from the definition of triangle-freeness. We
prove the statement by contradiction. Let (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 be a relevant sequence pair, andlet information sets j1 ∈ J1 , j2 ∈ J2 be such that pj1 = σ1 , pj2 = σ2 . Furthermore, assume thatthe σ1 -rank of j2 is greater than 1, and at the same time the σ2 -rank of j1 is greater than 1. Sincethe σ2 -rank of j1 is greater than 1, there exists an information set j2′ ∈ J2 , pj2′ = σ2 , distinct
from j2 , such that j1 ⇌ j2′ . Similarly, because the σ1 -rank of j2 is greater than 1, there exists an
information set j1′ ∈ J1 , pj1′ = σ1 , distinct from j1 , such that j1′ ⇌ j2 . But then, we have found
j1 , j1′ ∈ J1 and j2′ , j2 ∈ J2 such that pj1 = pj2 = σ1 , pj2′ = pj2 = σ2 such that j1 ⇌ j2′ , j1′ ⇌ j2 ,and j1 ⇌ j2 . So, the game is not triangle-free, contradiction.

9.4.3

Two-player games with public chance moves are triangle-free

In Theorem 9.2 we show that games with public chance (which includes games with no chancemoves at all) always satisfy the triangle-freeness condition of Definition 9.5.
Theorem 9.2. A two-player imperfect-information extensive-form game with public chancemoves is triangle-free.

Proof. For contradiction, let j1 , j2 be two distinct information sets for Player 1 such that pj1 = pj2 ,let J1 , J2 be two distinct information sets for Player 2 such that pJ1 = pJ2 , and assume thatj1 ⇌ J1 , j2 ⇌ J2 , j1 ⇌ J2 . By definition of connectedness, there exist nodes u ∈ j1 , v ∈ J1 suchthat v is on the path from the root to u, or vice versa. Similarly, there exist nodes u′ ∈ j2 , v ′ ∈ J2
such that u′ is on the path from the root to v ′ , or vice versa. Let w be the lowest common ancestorof u and u′ . It is not possible that w = u or w = u′ , because otherwise the parent sequencesof j1 and j2 would be different. So w must be a strict ancestor of both u and u′ , and u and u′
must be reached using different edges at w. Therefore, node w cannot belong to Player 1, orotherwise it again would not be true that pj1 = pj2 . So, there are only two possible cases: either
w belongs to Player 2, or it belongs to the chance player. We break the analysis accordingly.
First case: w belongs to Player 2. From above, we know that u and u′ are reached by following
different branches at w. So, if both v and v ′ were strict descendants of w, they would need
to be on two different branches of w (because they are connected to u and u′ respectively),
violating the condition pJ1 = pJ2 . So, at least one between v and v ′ is on the path from194

§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

the root to w (inclusive). But then either v is an ancestor of v ′ , or vice versa. Either caseviolates the hypothesis that pJ1 = pJ2 .
Second case: w belongs to the chance player. If any between v and v ′ is an ancestor of w, thennecessarily either v is an ancestor of v ′ , or v ′ is an ancestor of v. Either case violates thecondition pJ1 = pJ2 . So, both v and v ′ must be descendants of w. Because v is on thepath from the root to u (or vice versa), and v ′ is on the path from the root to u′ (or viceversa), then necessarily u, v and u′ , v ′ are on two different branches of the chance node w.
To fix names, call a the action at w that must be taken to (eventually) reach u and v, andlet b be the action that must be taken to (eventually) reach u′ and v ′ . Now, we use thehypothesis that j1 ⇌ J2 , that is, there exists u′′ ∈ j1 , v ′′ ∈ J2 such that u′′ is on the pathfrom the root to v ′′ or vice versa. Assume that u′′ is on the path from the root to v ′′ . Sinceu′′ belongs to the same information set as u (that is, j1 ), and since chance is public byhypothesis, then Player 1, when acting at u and u′′ , must have observed action a at w.
In other words, the path from the root to u′′ must pass through action a at w. But then,using the fact that u′′ is on the path from the root to v ′′ , this means that the path from theroot to v ′′ passes through action a. However, the path from the root to v ′ passes throughaction b. Since chance is public, nodes v ′ and v ′′ cannot be in the same information set,because Player 2 is able to distinguish them by means of the observed chance outcome.
We reached a contradiction. The symmetric case where v ′′ is on the path from the root tou′′ is analogous.
However, not all triangle-free games must have public chance nodes. For example, the leftmost
game in Figure 9.2 is triangle-free, but in that game the chance outcome is not public to Player 2.
So, our results apply more broadly than games with public chance moves.

9.4.4

Computation of the decomposition

Our algorithm consists of a recursive function, Decompose. It takes three arguments:(i) a sequence pair (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 ;(ii) a subset S of the set of all relevant sequence pairs; and(iii) a set D where only the entries indexed by the elements in S have been filled in.
High-level overview The decomposition for the whole von Stengel-Forges polytope V is computed by calling Decompose((∅, ∅), {(∅, ∅)}, {(1)})—this corresponds to the starting situationin which only the entry v[∅, ∅] has been filled in (denoted as fill-in step 1 in Figure 9.2). Eachcall to Decompose returns a pair (S ′ , D′ ) of updated indices and partial vectors, to reflect the new
entries that were filled in during the call.
Decompose((σ1 , σ2 ), S, D) operates as follows (we denote with −i the opponent for Player i):§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

195

1. Let Bi := {I ∈ Ji : I ▷◁ σ−i , pI = σi } for all i ∈ {1, 2}, and J ∗ ← { }.
2. For each (j1 , j2 ) ∈ B1 × B2 such that j1 ⇌ j2 , if the σ2 -rank of I1 is greater than or equal tothe σ1 -rank of j2 , we update J ∗ ← J ∗ ∪ {j1 }. Else, we update J ∗ ← J ∗ ∪ {j2 }.
3. For each i ∈ {1, 2} and I ∈ Bi such that the σ−i -rank of I is 0, do J ∗ ← J ∗ ∪ {I}.
4. For each I ∈ J ∗ : (Below we assume that I ∈ J1 , the other case is symmetrical)(a) Fill in all entries {v[(I, a), σ2 ] : a ∈ AI } by splitting v[σ1 , σ2 ]. This can be expressedhusing a scaled extension operation as D ← D ◁ ∆|AI | where h extracts v[σ1 , σ2 ] from any
partially-filled-in vector.
(b) Update S ← S ∪ {((I, a), σ2 )} to reflect that the entries corresponding to (I, a) ▷◁ σ2 havebeen filled in.
(c) For each a ∈ AI we assign (S, D) ← Decompose(((I, a), σ2 ), S, D). End for.
(d) Let K := {J ∈ J2 : I ⇌ J}. For all J ∈ J2 such that pJ ≽ (J ′, a′ ) for some J ′ ∈ K, a′ ∈ AJ ′ :• If I ⇌ J, then for all a ∈ AJ we fill in the sequence pair v[σ1 , (J, a)] by assigning its value in accordance with the von Stengel-Forges constraint v[σ1 , (J, a)] =Ph∗ ∗a∗ ∈AI ∗ v[(I , a ), (J, a)] via the scaled extension D ← D ◁{1} where the linearPfunction h maps a partially-filled-in vector to the value of a∗ ∈AI ∗ v[(I ∗ , a∗ ), (J, a)].
P
Since this is done for all a ∈ AJ , automatically a∈AJ v[σ1 , (J, a)] = v[σ1 , σ2 ], andwe can safely ignore the latter constraint.
• Otherwise, we fill in the entries {v[σ1 , (J, a)] : a ∈ AJ }, by splitting the valuehv[σ1 , pJ ]. In this case, we let D ← D ◁ ∆|AJ | where h extracts the entry v[σ1 , pJ ] froma partially-filled-in vector in D.
5. At this point, all the entries corresponding to indices S̃ = {(σ1′ , σ2′ ) : σ1′ ≽ σ1 , σ2′ ≽ σ2 } havebeen filled in, and we return (S ∪ S̃, D).
The above algorithm formalizes and generalizes the first two examples of Figure 9.2. For
example, step 2 of the fill-in order in either example is captured in Step 4(a), while fill-in step 3
corresponds to Step 4(c). Finally, fill-in step 4 corresponds to Step 4(d). In the rest of the section,
we formalize the construction and prove the correctness of the algorithm.
9.4.4.1

Two useful subroutines

We start by presenting two simple subroutines that capture Step 4(d) of Section 9.4.4 (whichcorrespond to fill-in step 4 in Figure 9.2). The two subroutines are symmetric and have the roleof filling rows and columns of the correlation plans.
The following inductive contract will be important for the full algorithm.

§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

196

Algorithm 9.1: FillOutRow((σ1 , σ2 ), j1 , S, D)

Algorithm 9.2: FillOutColumn((σ1 , σ2 ), j2 , S, D)

Preconditions :(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , j1 ∈ I1 , pj1 =
σ1 , (σ1 , σ2 ) ∈ S
1 for j2 such that pj2 = σ2 and σ1 ▷◁ j2 do2
if j1 ⇌ j2 then3for σ2′ ∈ {(j2 , a) : a ∈ Aj2 } do[▷ Fill (σ1 , σ2′ ) by summing up all entries{v[(j1 , a′ ), σ2′ ] : a′ ∈ Aj1 }]4
S ← S ⊔ {(σ1 , σ2′ )}Preconditions :(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , j2 ∈ I2 , pj2 =σ2 , (σ1 , σ2 ) ∈ S
1 for j1 such that pj1 = σ1 and σ2 ▷◁ j1 do
2
if j1 ⇌ j2 then3for σ ′ ∈ {(j1 , a) : a ∈ Aj1 } do[▷ Fill (σ1′ , σ2 ) by summing up all entries{v[σ1′ , (j2 , a′ )] : a′ ∈ Aj2 }]4
S ← S ⊔ {(σ1′ , σ2 )}h5

D ← D ◁{1}
P whereh : v 7→ a′ ∈A v[(j1 , a′ ), σ2′ ]hD ← D ◁{1}
P whereh : v 7→ a′ ∈A v[σ1′ , (j2 , a′ )]

5

j1

6

7

else
[▷ Fill all {v[σ1 , (j2 , a)] : a ∈ Aj2 } bysplitting v[σ1 , σ2 ] accordance with the vonStengel-Forges constraints]
S ← S ⊔ {(σ1 , (j2 , a)) : a ∈ Aj2 }hD ← D ◁ ∆Aj2 where h : v 7→ v[σ1 , σ2 ]9for σ2′ ∈ {(j2 , a) : a ∈ Aj2 } do
10
FillOutRow((σ1 , σ2′ ), j1 )11 return (S, D)
8

j2

6

7

else
[▷ Fill all {v[(j1 , a), σ2 ] : a ∈ Aj1 } by splittingv[σ1 , σ2 ] accordance with the vonStengel-Forges constraints]
S ← S ⊔ {((j1 , a), σ2 ) : a ∈ Aj1 }hD ← D ◁ ∆Aj1 where h : v 7→ v[σ1 , σ2 ]9for σ ′ ∈ {(j1 , a) : a ∈ Aj1 } do
10
FillOutColumn((σ1′ , σ2 ), j2 )11 return (S, D)
8

Lemma 9.6 (Inductive contract for FillOutRow). Suppose that the following preconditions
hold when FillOutRow((σ1 , σ2 ), j1 , S, D)) is called:(Pre1) (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 ;(Pre2) j1 ∈ I1 is such that pj1 = σ;(Pre3) S contains only relevant sequence pairs and D consists of vectors indexed by exactly
the indices in S;(Pre4) (σ1 , σ2 ) ∈ S, but (σ1 , σ2′ ) ∈/ S for all σ2′ ≻ σ2 ;(Pre5) For all a ∈ j1 and σ2′ ≽ σ2 such that j1 ▷◁ σ2′ , ((j1 , a), σ2′ ) ∈ S;
(Pre6) If j1 ▷◁ σ2 , all v ∈ D satisfy the von Stengel-Forges constraint v[σ1 , σ2 ] =Pa∈j1 v[(j1 , a), σ2 ];(Pre7) All v ∈ D satisfy the von Stengel-Forges constraintsv[(I, a), pj2 ] =

X
a′ ∈A

v[(I, a), (j2 , a′ )]j2

for all a ∈ j1 , and j2 ∈ I2 : j1 ▷◁ j2 , pj2 ≽ σ2 .
Then, the sets (S ′ , D′ ) returned by the call are such that
(Post1) S ′ contains only relevant sequence pairs and D′ consists of vectors indexed by§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

197

exactly the indices in S ′ ;(Post2) S ′ = S ⊔ {(σ1 , σ2′ ) : σ2′ ≻ σ2 , σ ▷◁ σ2′ };(Post3) All v ∈ D′ satisfy the von Stengel-Forges constraintsv[σ1 , pj2 ] =

X
a′ ∈A

v[σ1 , (j2 , a′ )]∀j2 ∈ I2 : σ ▷◁ j2 , pj2 ≽ σ2j2

and all von Stengel-Forges constraintsv[σ1 , σ2′ ] =Xv[(I, a), σ2′ ] ∀σ2′ ∈ Σ2 : σ2′ ▷◁ j1 , σ2′ ≽ σ2 .

a∈Aj1

Proof. By induction.
Base case The base case corresponds to σ2 ∈ Σ2 such that no information set j2 ∈ I2 : pj2 =σ2 ∧ σ1 ▷◁ j2 exists. In that case, Algorithm 9.1 returns immediately, so (Post1) holds
trivially from (Pre3). Since no j2 such that pj2 = σ2 ∧ σ1 ▷◁ j2 exists, no σ2′ ≻ σ2 such that
σ1 ▷◁ σ2′ exists, so (Post2) holds as well. The first set of constraints of (Post3) is empty, andthe second set reduces to (Pre6).
Inductive step Suppose that the inductive hypothesis holds when σ2′ ≻ σ2 . We will showthat it holds when σ2′ = σ2 as well. In order to use the inductive hypothesis, we firstneed to check that the preconditions are preserved at the time of the recursive call onLine 10. (Pre1) holds since σ1 ▷◁ j2 . (Pre2) holds trivially since σ does not chance. (Pre3)
holds since we are updating S and D in tandem on lines 4, 5 and 7, 8. (Pre4) holds sinceby the time of the recursive call we have only filled in entries (σ1 , σ2′ ) where σ2′ is animmediate successor of σ2 . (Pre5) at Line 10 holds trivially, since it refers to a subset ofthe entries for which the condition held at the beginning of the call. (Pre6) holds because
j1 ▷◁ σ2′ ⇐⇒ j1 ⇌ j2 . Hence, if j1 ▷◁ σ2′ then Lines 4 and 5 must have run. (Pre7) at Line
10 holds trivially, since it refers to a subset of the constraints for which the condition heldat the beginning of the call. Using the inductive hypothesis, (Post1), (Post2), and thesecond set of constraints in (Post3) follow immediately. The only constraints that are left
to be verified arev[σ1 , σ2 ] =Xv[σ1 , (j2 , a′ )]∀j2 ∈ I2 : σ ▷◁ j2 , pj2 = σ2 .

(9.3)

a′ ∈Aj2

That constraint is guaranteed by Lines 7 and 8 for all j2 ̸⇌ j1 . So, we need to verify
that it holds for all those j2 such that pj2 = σ2 , σ ▷◁ j2 and j1 ⇌ j2 . Let j2 be one such

198

§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

information set. Then, from Lines 4 and 5 we have thatv[σ1 , (j2 , a)] =

X
a′ ∈A

v[(I, a′ ), (j2 , a)] ∀a ∈ Aj2 .

j1

Summing the above equations across all a ∈ Aj2 and using (Pre7) yields
Xv[σ1 , (j2 , a)] =X

X

v[(I, a′ ), (j2 , a)]a∈Aj2 a′ ∈Aj1

a∈Aj2

=

X
a′ ∈A

=

X

v[(I, a′ ), (j2 , a)]j1 a∈Aj2

X

v[(I, a′ ), pj2 ]a′ ∈Aj1

=

X
a′ ∈A

v[(I, a′ ), σ2 ],j1where we used the hypothesis that pj2 = σ2 in the last equality. Finally, since j1 ⇌ j2
and pj2 = σ2 , it must be j1 ▷◁ σ2 and so, using (Pre6), we obtain thatXv[σ1 , (j2 , a)] = v[σ1 , σ2 ],a∈Aj2completing the proof of Equation (9.3). So, (Post3) holds as well and the proof of theinductive step is complete.
The inductive contract for FillOutColumn is symmetric.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

9.4.4.2

The full decomposition algorithm

We are now ready to present the full decomposition algorithm.
Algorithm 9.3: Decompose((σ1 , σ2 ), S, D)
Preconditions :(σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , (σ1 , σ2 ) ∈ S
B ← {}
for all i ∈ {1, 2}, I ∈ Ii , pI = σi , σ−i ▷◁ I do
3
if the σ−i -rank of I is 0 then4
B ←B⊔I
5 for (j1 , j2 ) ∈ I1 × I2 such that pj1 = σ1 , pj2 = σ2 , j1 ⇌ j2 do
6
if the σ2 -rank of j1 is ⩾ the σ1 -rank of j2 then7
B ← B ⊔ j1
8else
9B ← B ⊔ j2
10 for I ∈ B do
11
if I ∈ I1 then[▷ Fill all {v[(I, a), σ2 ] : a ∈ AI } by splitting v[σ1 , σ2 ] accordance with the von Stengel-Forges
constraints]
12
S ← S ⊔ {((I, a), σ2 ) : a ∈ AI }
1
2

h

13
14
15

16
17

18

D ← D ◁ ∆AI where h : v 7→ v[σ1 , σ2 ]for σ1′ ∈ {(I, a) : a ∈ AI } do
Decompose((σ1′ , σ2 ), S, D)[▷ Fill a portion of the row for σ1 ]for j2 ∈ I2 : σ1 ▷◁ j2 , pj2 = σ2 dofor σ2′ ∈ {(j2 , a′ ) : a′ ∈ Aj2 } do[▷ Fill (σ1 , σ2′ ) by summing up all entries {v[(I, a′ ), σ2′ ] : a′ ∈ AI }]S ← S ⊔ {(σ1 , σ2′ )}h19

D ← D ◁{1} where h : v 7→

20

FillOutRow((σ1 , σ2′ ), I)

21

22[▷ Recursive call]Pa′ ∈AIv[(I, a′ ), σ2′ ]else[▷ Fill all {v[σ1 , (I, a)] : a ∈ AI } by splitting v[σ1 , σ2 ] accordance with the von Stengel-Forges
constraints]
S ← S ⊔ {(σ1 , (I, a)) : a ∈ AI }
h

23
24
25

26
27

28

D ← D ◁ ∆AI where h : v 7→ v[σ1 , σ2 ]for σ2′ ∈ {(I, a) : a ∈ AI } do
Decompose((σ1 , σ2′ ), S, D)[▷ Fill a portion of the column for σ2 ]for j1 ∈ I1 : σ2 ▷◁ j1 , pj1 = σ1 dofor σ1′ ∈ {(j1 , a′ ) : a′ ∈ Aj1 } do[▷ Fill (σ1′ , σ2 ) by summing up all entries {v[σ1′ , (I, a′ )] : a′ ∈ AI }]S ← S ⊔ {(σ1′ , σ2 )}
h

29
30
31

[▷ Recursive call]D ← D ◁{1} where h : v 7→
FillOutColumn((σ1′ , σ2 ), I)
return (S, D)P

a′ ∈AIv[σ1′ , (I, a′ )]199§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

200

Lemma 9.7 (Inductive contract for Decompose). Assume that at the beginning of each call to
Decompose((σ1 , σ2 ), S, D) the following conditions hold
(Pre1) S contains only relevant sequence pairs and D consists of vectors indexed by exactly
the indices in S.
(Pre2) S does not contain any relevant sequence pairs which are descendants of (σ1 , σ2 ),with the only exception of (σ1 , σ2 ) itself. In formulas,S ∩ {(σ1′ , σ2′ ) ∈ Σ1 × Σ2 : σ1′ ≽ σ1 , σ2′ ≽ σ2 } = {(σ1 , σ2 )}.
Then, at the end of the call, the returned sets (S ′ , D′ ) are such that
(Post1) S ′ contains only relevant sequence pairs and D′ consists of vectors v indexed byexactly the indices in S ′ .
(Post2) The call has filled in exactly all relevant sequence pair indices that are descendantsof (σ1 , σ2 ) (except for (σ1 , σ2 ) itself, which was already filled in). In formulas,
S ′ = S ⊔ {(σ1′ , σ2′ ) ∈ Σ1 × Σ2 : σ1′ ≽ σ1 , σ2′ ≽ σ2 , (σ1′ , σ2′ ) ̸= (σ1 , σ2 ), σ1′ ▷◁ σ2′ }.
(Post3) D′ satisfies the subset of von Stengel-Forges constraintsX

v[(I, a), σ2′ ] = v[pI , σ2′ ] ∀σ2′ ≽ σ2 , I ∈ I1 s.t. σ2′ ▷◁ I, pI ≽ σ1

a∈AI

X

v[σ1′ , (J, a)] = v[σ1′ , pJ ]∀σ1′ ≽ σ1 , J ∈ I2 s.t. σ1′ ▷◁ J, pJ ≽ σ2 .

a∈AJ

Proof. By induction.
Base case The base case is any (σ1 , σ2 ) such that there is no σ1′ ≽ σ1 , σ2′ ≽ σ2 , σ1′ ▷◁ σ2′ . In thatcase, the set B is empty, so the algorithm terminates immediately without modifying
the sets S and D. Consequently, (Post1) and (Post2) hold trivially from (Pre1) and (Pre2).
(Post3) reduces to an empty set of constraints, so (Post3) holds as well.
Inductive step In order to use the inductive hypothesis, we will need to prove that thepreconditions for Decompose hold on Lines 15 and 25. We will focus on Line 15 (I ∈ I1 ),as the analysis for the other case (I ∈ I2 ) is symmetric. (Pre1) clearly holds, since
we always update S and D in tandem. Since all iterations of the for loop on Line 10
touch different information sets, at the time of the recursive call on Line 15, and given(Post2) for all previous recursive calls, the only relevant sequence pairs (σ1′′ , σ2′′ ) such that
σ1′′ ≽ σ1′ , σ2′′ ≽ σ2 that have been filled are the ones on Lines 12 and 13. So, (Pre2) holds.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE201

We now check that the preconditions for FillOutRow hold at Line 20. (Pre1), (Pre2),(Pre3), and (Pre4) are trivial. (Pre5) and (Pre7) are guaranteed by (Post2) and (Post3) ofDecompose applied to Line 15. (Pre6) holds because of Lines 18 and 19.
Using the inductive contracts of FillOutRow, FillOutColumn and Decompose for the recursive
calls, we now show that all postconditions hold at the end of the call. (Post1) is trivial
since we always update S and D together. (Post2) holds by keeping track of what entries
are filled in Lines 12, 13, 18, 19, 22, 23, 28, 29, as well as those filled in the calls to
FillOutRow, FillOutColumn and Decompose, as regulated by postcondition (Post2) in the
inductive contracts of the functions. In order to verify (Post3), we need to verify that theconstraints that are not already guaranteed by the recursive calls hold. In particular, weneed to verify thatAX

v[(I, a), σ2 ] = v[σ1 , σ2 ] ∀I ∈ I1 s.t. σ2 ▷◁ I, pI = σ1 , I ∈
/B

a∈AI

B

X

v[σ1 , (J, a)] = v[σ1 , σ2 ]∀J ∈ I2 s.t. σ1 ▷◁ J, pJ = σ2 , J ∈
/ B.

a∈AJ

We will show that constraints A hold; the proof for B is symmetric. Using Lemma 9.5
together with the definition of B (Lines 1-9), any information set I ∈ Ii : pI = σi , σ−i ▷◁ I
that is not in B must have σ−i -rank exactly 1. Let I ∈ I1 be such that σ2 ▷◁ I, pI = σ1 , I ∈/ B,
as required in A . Since the σ2 -rank of I is 1, let J be the only information set in I2 suchthat I ⇌ J, pJ = σ2 . Note that J ∈ B. The entries v[(I, a), σ2 ] : a ∈ AI were filled inLines 28 and 29 when the for loop picked up J ∈ B. So, in particular,v[(I, a), σ2 ] =

X
a′ ∈A

v[(I, a), (J, a′ )]∀a ∈ AI .

J

Summing the above equations across a ∈ AI , we obtainXv[(I, a), σ2 ] =X Xv[(I, a), (J, a′ )]a∈AI a′ ∈AJ

a∈AI

=

X X

v[(I, a), (J, a′ )]a′ ∈AJ a∈AI=

X

v[σ1 , (J, a′ )]a′ ∈AJ= v[σ1 , σ2 ],where the last equation follows from the way the entries v[σ1 , (J, a′ )] : a′ ∈ AJ werefilled in (Lines 22 and 23). This shows that the set of constraints A hold.
We are now ready to conclude the proof of correctness for the decomposition algorithm.

§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

202

Theorem 9.3. The von Stengel-Forges polytope V of a two-player perfect-recall triangle-free
EFG can be expressed via a sequence of scaled extensions with simplexes and singleton sets:
h1

h2

h3

hn

V = {1} ◁ X1 ◁ X2 ◁ · · · ◁ Xn ,(9.4)

where, for i = 1, . . . , n, either Xi = ∆si for some simplex dimension si ∈ ℕ, or Xi = {1}, andhi is a linear function. Furthermore, an exact algorithm exists to compute such expression in
linear time in the dimensionality of V, and so, in time at most quadratic in the size of thegame.

Proof. The correctness of the algorithm follow from (Post3) in the inductive contract. Everytime the set of partially-filled-in vectors D gets extended, it is extended with either the singleton
set {1} or a simplex. In either case the nonnegative affine functions h used are linear. So, thedecomposition structure is as in the statement. Finally, since the overhead of each call (on topof the recursive calls) is linear in the number of relevant sequence pairs (σ, τ ) ∈ Σ1 ▷◁ Σ2 thatare filled, and each relevant sequence pair is filled only once, the complexity of the algorithm
is linear in the number of relevant sequence pairs.

9.4.5

Integrality of the vertices of V in triangle-free games

The scaled-based decomposition of V can be used to conclude the integrality of the vertices of V,
by leveraging the following analytical result about the scaled extension operation.
Lemma 9.8.

Let X , Y, and h be as in Definition 4.1. If X is a convex polytope withhvertices {x1 , . . . , xn }, and Y is a convex polytope with vertices {y 1 , . . . , y m }, then X ◁ Y is aconvex polytope whose vertices are a nonempty subset of {(xi , h(xi )y j ) : i ∈ {1, . . . , n}, j ∈{1, . . . , m}}.

h

Proof. Take any point z ∈ X ◁ Y. By definition of scaled, extension, there exist x ∈ X , y ∈ Ysuch that z = (x, h(x)y). Since {x1 , . . . , xn } are the vertices of X , x can be written as a convex
Pn
Pm
combination x = i=1 λi xi where (λ1 , . . . , λn ) ∈ ∆n . Similarly, y = i=1 µi y i for some(µ1 , . . . , µm ) ∈ ∆m . Hence, using the hypothesis that h is affine, we can write


! m
n
n
X
X
X
z = (x, h(x)y) = 
λi xi , hλi xi
µj y j 
i=1

i=1

j=1

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE


=

n
X

n
X

λi xi ,

i=1

203


! m
n X
m
X
X

λi h(xi )µj y j  =λi µj xi , h(xi )y j .

i=1

j=1

i=1 j=1

Since λi µj ⩾ 0 for all i ∈ [[n]], j ∈ [[m]] andn XmX

λi µj =

i=1 j=1

n
X


! m
X
λi 
µj  = 1,

i=1

j=1

we conclude that z ∈ co{(xi , h(xi )y j ) : i ∈ [[n]], j ∈ [[m]]}. On the other hand, (xi , h(xi )y j ) ∈h

X ◁ Y, sohX ◁ Y = conoxi , h(xi )y j : i ∈ [[n]], j ∈ [[m]] .

Since the vertices of a (nonempty) polytope are a (nonempty) subset of any convex basis for thehpolytope, the vertices of X ◁ Y must be a nonempty subset of (xi , h(xi )y j ) : i ∈ [[n]], j ∈ [[m]] ,which is the statement.
In particular, by applying Lemma 9.8 inductively on the structure of the scaled-extension-based
structural decomposition of V, we obtain the following theorem.
Theorem 9.4. Let V be the von Stengel-Forges polytope of a two-player triangle-free game(Definition 9.5). All vertices of V have integer {0, 1} coordinates.

Proof. We prove the statement by induction over the scaled-extension-based decomposition
h1

h2

hn

V = {1} ◁ X1 ◁ · · · ◁ Xn .
In particular, we will show that for all k = 0, . . . , n, the coordinates of the vertices of thepolytopeh1

hk

Vk = {1} ◁ · · · ◁ Xk
constructed by considering only the first k scaled extensions in the decomposition are all
integer. Since V ⊆ [0, 1]Σ1 ▷◁Σ2 (Remark 9.1), this immediately implies that each coordinate is in{0, 1}.
Base case: k = 0. In this case, V0 = {1}. The only vertex is {1}, which is integer. So, base casetrivially holds.

204§9.4. SCALED-EXTENSION-BASED STRUCTURAL DECOMPOSITION FOR V

Inductive step Suppose that the polytope Vk (k < n) has integer vertices. We will show
hk+1

that the same holds for Vk+1 . Clearly, Vk+1 = Vk ◁ Xk+1 . From the properties of thestructural decomposition, we know that Kk+1 is either the singleton {1}, or a probability
simplex ∆sk+1 for some appropriate dimension sk+1 . We break the analysis accordingly.
• If Xk+1 = {1}, the scaled extension represents filling in a linearly-dependent entry
in v ∈ V by summing already-filled-in entries. So, hk+1 takes a partially-filled-in
vector from Vk and sums up some of its coordinates. Let v 1 , . . . , v n be the vertices
of Vk . Using Lemma 9.8, the vertices of Vk+1 are a subset of{(v i , h(v i ) · 1) : i = 1, . . . , n}.

(9.5)

Since by inductive hypothesis v i have integer coordinates, and h sums up some ofthem, h(v)i is integer for all i = 1, . . . , n. So, all of the vectors in (9.5) have integer
coordinates, and in particular this must be true of the vertices of Vk+1 .
• If Xk+1 = ∆sk+1 , the scaled extension represents the operation of partitioning an
already-filled-in entry v[σ, τ ] of Vk into si non-negative real values. The affine
function hk+1 extracts the entry v[σ, τ ] from each vector v ∈ Vk . Let v 1 , . . . , v n be thevertices of Vk . The vertices of ∆sk+1 are the canonical basis vectors {e1 , . . . , esk+1 }.
From Lemma 9.8, the vertices of Vk+1 are a subset of{(v i , h(v i )ej ) : i = 1, . . . , n, j = 1, . . . , sk+1 }= {(v i , vi [σ, τ ]ej ) : i = 1, . . . , n, j = 1, . . . , sk+1 }.

(9.6)

Since by inductive hypothesis the vertices v i have integer coordinates, vi [σ, τ ] is an
integer. Since the canonical basis vector only have entries in {0, 1}, all of the vectorsin (9.6) have integer coordinates. So, in particular, this must be true of the vertices
of Vk+1 .

Finally, combining Theorem 9.4 and Theorem 9.1, we obtain the central theorem of thischapter.

Theorem 9.5. In a two-player perfect-recall imperfect-information extensive-form game thatsatisfies the triangle-freeness condition (Definition 9.5), the polytope of correlation plans
coincides with the von Stengel-Forges polytope. Consequently, an optimal EFCE can becomputed in polynomial time (in the size of the input imperfect-information extensive-formgame) in two-player triangle-free games.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

9.5

205

Beyond triangle-freenessThe result established in Theorem 9.3 shows that, in certain games, the polytope of correlationplans can be expressed as a polynomially-sized chain of scaled extension operations. This
characterization implies that an optimal EFCE can be found in polynomial time, and, as we
will show in Chapter 10, it also enables the construction of practical no-regret algorithms via
the formalism of regret circuits (Chapter 4). In work following the introduction of the result,
B. H. Zhang, Farina, Celli, and Sandholm (2022) showed that a characterization of Ξ based onscaled extension exists in any game, though it might not be of polynomial size.[9.a] To establishthat result, which will not be discussed in detail in this dissertation, we used techniques fromthe theory of tree decompositions to construct a polytope of dimension typically exponential in
the input game tree size, whose vertices are guaranteed to map onto the vertices of the polytope
of correlation plans via a linear projection. We remark that the length of the chain of scaled
extension operations that define the polytope whose projection is Ξ is exponential in a parameterthat intuitively represent the amount of uncommon information between the players, thereby
yielding the current state-of-the-art parameterized complexity results for the computation of
optimal EFCE in general imperfect-information extensive-form games.
We refer the interested reader to the paper by B. H. Zhang, Farina, Celli, and Sandholm (2022)
for the result, and to prior work by B. H. Zhang, Farina, and Sandholm (2023) and B. H. Zhang
and Sandholm (2022a) as a gentle introduction on how the machinery of tree decomposition canbe applied in correlated strategy spaces, and how that relates to scaled extension.

9.6

Experimental investigation of utilities reached by EFCE

The characterization of the polytope of correlation plans given in this chapter enables visualizing,for the first time, the polytope of all expected utility profiles that can be reached by EFCEs. Indeed,recall that the expected utility for any player is a linear function of the correlation plan ξ. Hence,
set of all utilities reachable by EFCE is a linear transformation of the polytope of EFCE defined byconstraints 1 - 4 in Proposition 9.1, and can thus be computed efficiently.
In the next two subsections, we use the characterization of the polytope of correlation plans toinvestigate empirically the set of expected utilities that can be reached by EFCE in game instances
drawn from a variety of standard parametric classes, identified with an alphabetical mnemonic:B – battleship, D – liar’s dice, K – Kuhn poker, L – Leduc poker, RS – ridesharing game,S – sheriff, T – three-player tricks game, TP – double dummy bridge game. As in the otherchapters, a full description of the games is available in Appendix A.
[9.a] As discussed, the exponential size cannot be avoided, unless P = NP. See also von Stengel and Forges (2008).

206§9.6. EXPERIMENTAL INVESTIGATION OF UTILITIES REACHED BY EFCE

9.6.1

Two-player general-sum games

Figure 9.3 shows the polytopes of expected utilities that can be reached by EFCE in three two-player
general-sum games. We observe that both the shape of the polytope and the range of reachable
payoffs is highly nontrivial. This empirically suggests that being able to search and optimize overthe space of EFCE (rather than computing a single EFCE without a priori guarantees over thesocial welfare, as in Chapter 8) is important.

S2122 — Sheriff

Player 2’s utility →RS212 — Ridesharing

Player 2’s utility →

2.844
2.869
2.894
2.919
2.943
2.968
2.993
3.018
3.043
3.067
3.092

0.077
0.538
0.999
1.460
1.921
2.382
2.843
3.304
3.765
4.226

3.092
3.067
3.043
3.018
2.993
2.968
2.943
2.919
2.894
2.869
2.844
−0.384

7.906
7.133
6.360
5.587
4.814
4.041
3.268
2.495
1.722
0.949
0.175
−0.585−0.558−0.531−0.504−0.477−0.450−0.423−0.396−0.369−0.342−0.315Player 1’s utility →B2222 — Battleship−0.015−0.034−0.053−0.071−0.090−0.109−0.128−0.147−0.166−0.184−0.203Player 2’s utility →Figure 9.3: Polytope of expected utilities for the players that can be reached via extensive-form correlated
equilibria in three standard two-player general-sum imperfect-information extensive-form games.

9.6.2

Three-player zero-sum games

Figure 9.3 shows the polytopes of expected utilities that can be reached by EFCE across six
three-player zero-sum games. We remark that despite the appearance of curved boundaries, theset of payoffs reachable by EFCE, being the linear projection of a polytope with finitely many
facets, must itself be a polytope defined by finitely many facets. The experimental data confirms
the conclusion we drew in the case of two-player games (Figure 9.3): both the shape of thepolytope and the range of reachable payoffs is highly nontrivial. Once again, this empiricallysuggests that being able to search and optimize over the space of EFCE (rather than computing asingle EFCE without a priori guarantees over the social welfare, as in Chapter 8) is important.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE−
− 1.0
−0 0.71 99
2.3
0
0 .061 25
0.8 .448315
1 .221
1 .6082.3 .995
2.7 82
68

1’s
uti
lity
→

Pla
yer

0
0 .0240 .0280.0 .032
36
0
0 .0390.0 .043
47
0
0 .051
0.0 .055
5
0.0
62 8

tili
ty
→

’s u

Pla
yer
1

→

uti
lity

1’s

Pla
yer

→
lity

T3[50] — Tricks

TP3 — Double-dummy bridge

→

→

ty

tili

’s u

→

ty

tili

’s u

Pla
yer
1

lity

uti

→

→

Pla
yer
1

→

lity

uti

3’s

ty
tili

u
3’s

lity

uti

15
0.8
94
0 .8
73
0 .9
52
1.0
31
1 .1
10
1 .2
89
1.2
68
1 .3
47
1 .4
26
1.5
05
1 .6

76
0 .8
34
0.9
91
0.9
49
1 .0
06
1.1
64
1.1
22
1 .2
79
1.2
37
1.3
94
1 .3
52
1.4
← Player 2’s utility

1.799
1.720
1.641
1.562
1.483
1.404
1.325
1.246
1.167
1.088
1.009

yer
Pla

yer
Pla

3’s

5
.27
−0 5
.21
−0 55
.1
− 0 95
.0
−0 5
.03
−0
25
0 .0
85
0 .0
45
0.1
05
0 .2
65
0 .2
25
0.3
← Player 2’s utility

1.708
1.650
1.593
1.535
1.478
1.420
1.362
1.305
1.247
1.190
1.132

0
0 .386
0.5 .465
40
40 .6230 .7020.8 .781
60
0
1 .9391.0 .018
1.1 9775

D32 — Liar’s dice00 .416
0.5 .474
31
00 .5890 .646
0.7 .704
62
00 .8190.9 .877
0 . 9 34
92

← Player 2’s utility−− 0 .2−0 0.21 75
5.1−− 0.0 55
0.0 0.03 95
525
0
0 .0850.2 .14505
00.3 .265
25← Player 2’s utility

yer
Pla

1’s

uti

← Player 2’s utility

0.550
0.490
0.430
0.370
0.310
0.250
0.190
0.130
0.070
0.010
−0.050

Pla
yer

3’s

2
.28
−1 95
.8
−0 08
.5
−0 2
.12
−0
65
0 .2
52
0.6
39
1 .0
25
1 .4
12
1.8
99
2 .1
86
2 .5

6
.01
−0 12
.0
−0 08
.0
−0 04
.0
−0 1
.00
−0
03
0 .0
07
0.0
11
0 .0
14
0 .0
18
0.0
22
0 .0

5
.02
−0 23
.0
−0 21
.0
−0 19
.0
−0 7
.01
−0 15
.0
−0 13
.0
−0 1
.01
−0 09
.0
−0 07
.0
−0 05
.0
−0

2.381
1.994
1.607
1.221
0.834
0.447
0.060
−0.326−0.713−1.100−1.487

yer
Pla

→
ty
tili

y→

t
tili

u
3’s

u
3’s

−0.009−0.012−0.016−0.020−0.024−0.027−0.031−0.035−0.039−0.043−0.046

yer
Pla

yer
Pla

−0.016−0.018−0.020−0.022−0.024−0.026−0.028−0.030−0.032−0.034−0.036

L3223 — Leduc poker

K35 — Kuhn poker

0
0 .041
0.0 .043
45
00 .047
0 .0490.0 .051
53
0
0 .0550.0 .057
0.0 5961

K34 — Kuhn poker

207← Player 2’s utility

Figure 9.4: Polytope of expected utilities for the players that can be reached via extensive-form correlated
equilibria in three standard three-player zero-sum imperfect-information extensive-form games.

9.A

Appendix: Additional lemmas on the structure of V

Lemma 9.9. Let v ∈ V. For all σ1 ∈ Σ1 such that v[σ1 , ∅] = 0, v[σ1 , σ2 ] = 0 for all σ2 ▷◁ σ1 .
Similarly, for all σ2 ∈ Σ2 such that v[∅, σ2 ] = 0, v[σ1 , σ2 ] = 0 for all σ1 ▷◁ σ2 .

Proof. We prove the theorem by induction on the depth of the sequences σ1 and σ2 . The depth
depth(σ) of a generic sequence σ = (I, a) ∈ Σi of Player i is defined as the number of actions
that Player i plays on the path from the root of the tree down to action a at information set I
included. Conventionally, we let the depth of the empty sequence be 0.

§9.A. APPENDIX: ADDITIONAL LEMMAS ON THE STRUCTURE OF V

208

Take σ1 ∈ Σ1 such that v[σ1 , ∅] = 0. For σ2 of depth 0 (that is, σ2 = ∅), clearlyv[σ1 , σ2 ] = 0. For the inductive step, suppose that v[σ1 , σ2 ] = 0 for all σ2 ∈ Σ2 , σ1 ▷◁ σ2such that depth(σ2 ) ⩽ d2 . We will show that v[σ2 , σ2 ] = 0 for depth(σ2 ) ⩽ d2 + 1. Indeed,let (I, a′ ) = σ2 ▷◁ σ1 of depth d2 + 1. Since v ∈ V, in particular the von Stengel-ForgesP
constraint a∈AI v[σ1 , (I, a)] = v[σ1 , pI ] must hold. The depth of pI is d2 , so by the inductivePhypothesis, it must be v[σ1 , pI ] = 0, and therefore a∈AI v[σ1 , (I, a)] = 0. But all entries of v
are nonnegative, so it must be v[σ1 , (I, a)] = 0 for all a ∈ AI , and in particular for (I, a′ ) = σ2 .
This completes the proof by induction.
The proof for the second part is analogous.

Lemma 9.10. Let v ∈ V have integer {0, 1} coordinates. Then, for all (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 , itholds thatv[σ1 , σ2 ] = v[σ1 , ∅] · v[∅, σ2 ].

Proof. We prove the theorem by induction on the depth of the sequences, similarly to Lemma 9.9.
The base case for the induction proof corresponds to the case where σ1 and σ2 both have
depth 0, that is, σ1 = σ2 = ∅. In that case, the theorem is clearly true, because v[∅, ∅] = 1 aspart of the von Stengel-Forges constraints (Definition 9.3).
Now, suppose that the statement holds as long as depth(σ1 ), depth(σ2 ) ⩽ d. We will showthat the statement will hold for any (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 such that depth(σ1 ), depth(σ2 ) ⩽ d + 1.
Indeed, consider (σ1 , σ2 ) ∈ Σ1 ▷◁ Σ2 such that depth(σ1 ), depth(σ2 ) ⩽ d + 1. If any of thesequences is the empty sequence, the statements holds trivially, so assume that neither is theempty sequence and in particular σ1 = (I, a), σ2 = (J, b). If v[σ1 , ∅] = 0, then from Lemma 9.9v[σ1 , σ2 ] = 0 and the statement holds. Similarly, if v[∅, σ2 ] = 0, then v[σ1 , σ2 ] = 0, and thestatement holds. Hence, the only remaining case given the integrality assumption on thecoordinates of v is v[σ1 , ∅] = v[∅, σ2 ] = 1.
From the von Stengel-Forges constraints,v[pI , ∅] =X
a′ ∈AIv[(I, a′ ), ∅] = 1 +

X
a′ ∈A

I

v[(I, a′ ), ∅] ⩾ 1.

,a′ ̸=aHence, because all entries of v are in {0, 1}, it must be v[pI , ∅] = 1 and v[(I, a′ ), ∅] = 0 for alla′ ∈ AI , a′ ̸= a. With a similar argument we conclude that v[∅, pJ ] = 1 and v[∅, (J, b′ )] = 0
for all b′ ∈ AJ , b ̸= b′ . Using the inductive hypothesis,v[pI , pJ ] = v[pI , ∅] · v[∅, pJ ] = 1.

§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE

209

Now, using the von Stengel-Forges constraints together with the equality v[pI , pJ ] = 1 wejust proved, we conclude that
X X
a′ ∈A

v[(I, a′ ), (J, b′ )] = 1.

(9.7)

′
I b ∈AJ

On the other hand, since v[(I, a′ ), ∅] = 0 for all a′ ∈ AI , a′ ̸= a and v[∅, (J, b′ )] = 0 for allb′ ∈ AJ , b′ ̸= b, from Lemma 9.9 we have thata′ ̸= a ∨ b′ ̸= b =⇒ v[(I, a′ ), (J, b′ )] = 0.

(9.8)

From (9.8) and (9.7), we conclude that v[(I, a), (J, b)] = v[σ1 , σ2 ] = 1 = v[σ1 , ∅] · v[∅, σ2 ], as
we wanted to show.

9.B

Appendix: Optimal EFCE as a linear program

In order to express the utility of a trigger agent, it is necessary to compute the probability of thegame ending in each of the terminal states. To do that, for the purposes of this appendix we
introduce the following additional notation:• Πi (I), is the set of reduced-normal-form strategies that can lead to information set I(which belongs to Player i) assuming that the other player acts to do so as well. This isequivalent (assuming no zero-chance nodes or disconnected game trees) to saying that all
reduced-normal-from strategies in Πi (I) have some action which belongs to information set
I. Formally,Πi (I) := {π ∈ Πi : π[pI ] = 1}.
• Πi (Ia) is the set of reduced normal form strategies which will lead to information set I
and recommend the action a at I. This is equivalent to the set of reduced normal form
strategies which contain a as part of their recommendation (this set is typically a subset of
Πi (Ia)).Formally,Πi (Ia) := {π ∈ Πi : π[Ia] = 1}.
• σi (z) is the last sequence belonging to Player i on the path from the roof of the game tree tothe terminal state z ∈ Z.
• Πi (z) is the set of reduced-normal-form strategies which can lead to the terminal state z(assuming the other player players to do so). This is equivalent to Πi (σi (z)).

210§9.B. APPENDIX: OPTIMAL EFCE AS A LINEAR PROGRAM

For simplicity, we focus on the two-player case; generalizing to the general case is straightˆ → Z that maps a triple offorward. Consider the random variable tσ̂ : Π1 × Π2 × Π1 (I)reduced-normal-form strategies (π 1 , π 2 , π̂ 1 ) to the terminal state of the game that is reachedwhen Player 1 is a σ̂-trigger agent and Player 2 follows all recommendations. We are interested in
expressing the probability of terminating at each z ∈ Z for a σ̂-trigger agent, given the mediator’s
joint distribution µ over reduced normal form strategies and the trigger strategy µ̂ for the deviating
player, which we will assume to be Player 1 without loss of generality. For each trigger σ̂, theterminal leaves may be partitioned into the following 3 sets.
• Zσ̂ (or equivalently ZI,âˆ ) is the set of terminal nodes that are descendants of the trigger
ˆ â). In order for the game to end in one of these terminal nodes, it is necessary that theσ̂ = (I,recommendation device recommended to Player 1 the trigger sequence σ̂, and therefore theagent must have deviated. Furthermore, Player 2 must have been recommended the terminal
sequence σ2 (z) corresponding to the terminal state, and finally π̂ 1 must be compatible withσ1 (z). We can capture all these constraints concisely by saying that the sampled (π 1 , π 2 , π̂ 1 )must be such that π 1 ∈ Π1 (σ̂), π 2 ∈ Π2 (z) and π̂ 1 ∈ Π1 (z). Therefore the probability that aσ̂ trigger agent terminates at some z ∈ Zσ̂ is given by,ℙµ,µ̂ [tσ̂ = z ∈ Zσ̂ ] = Xπ 1 ∈Π1 (σ̂)π 2 ∈Π2 (z)µ(π 1 , π 2 )

Xµ̂1 (π̂ 1 ),π̂ 1 ∈Π1 (z)where the first term in the product is the probability that Player 2 plays to z and Player 1gets triggered, and the second term is the probability that the deviation strategy from Player
1 upon getting triggered is one that reaches z.
ˆ except σ̂. In• ZIˆ is the set of terminal states that are descendant of any sequence in I,
order for the game to reach this terminal state, recommendations issued to Player 1 by theˆ There are two cases: either
correlation device must have been such that Player 1 reached I.
ˆ or it did not. In the former case, Player 1 started
the correlation device recommended σ̂ at I,
deviating (using the sampled reduced-normal-form plan π̂ 1 ); hence, in this case it must beπ̂ 1 ∈ Π1 (z). In the latter case, Player 1 does not deviate from the recommendation, andtherefore it must be π 1 ∈ Π1 (z). Either way, Player 2 must have been recommended theterminal sequence z corresponding to the terminal state z; that is, π 2 ∈ Π2 (z). Collecting allthese constraints, it must beˆ(π 1 , π 2 , π̂ 1 ) ∈ Π1 (σ̂) × Π2 (z) × Π1 (z) ∪ Π1 (z) × Π2 (z) × Π1 (I).
Using the fact that the two cases as to whether or not Player 1 was recommended σ̂ or not at
Iˆ are disjoint, we can write§9. GEOMETRY OF CORRELATED STRATEGIES, AND POSITIVE COMPLEXITY RESULTS FOR
OPTIMAL EFCE
ℙµ,µ̂ [tσ̂ = z ∈ ZIˆ] = X

π 1 ∈Π1 (σ̂)π 2 ∈Π2 (z)µ(π 1 , π 2 )Xπ̂ 1 ∈Π1 (z)








µ̂1 (π̂ 1 ) + 


211

X

π 1 ∈Π1 (z)π 2 ∈Π2 (z)µ(π 1 , π 2 )
.

The first term in the summation may be understood as the probability that the agent was
triggered and its deviation was to play something other than σ̂. The second term is thatprobability that the agent was not triggered and the game simply terminates at z based on µ.
• Finally, Z−Iˆ is the set of terminal nodes that are neither in Zσ̂ nor in ZIˆ. If the gamehas ended in any terminal state that belongs to Z−Iˆ, Player 1 has not deviated from theˆrecommended strategy, since they have never even reached the trigger information set, I.
Hence, in this case it must be (π 1 , π 2 ) ∈ Π1 (z) × Π2 (z). Hence,ℙµ,µ̂ [tσ̂ = z ∈ Z−Iˆ] =Xµ(π 1 , π 2 ).

π 1 ∈Π1 (z)π 2 ∈Π2 (z)With the above, we can finally express the constraint that no deviation strategy µ̂ can lead to ahigher utility for Player 1 than simply following each recommendation. Indeed, for all µ̂, theutility of the trigger agent is expressed asX

u1 (z)ℙµ,µ̂ [tσ̂ = z],z∈Z

where the correct expression for ℙµ,µ̂ [tσ̂ = z] must be selected depending on whether z ∈ Zσ̂ ,z ∈ ZIˆ or z ∈ Z−Iˆ. On the other hand, the utility of an agent that follows all recommendations isX

u1 (z)ℙµ,µ̂ [π 1 ∈ Π1 (z), π 2 ∈ Π2 (z)] =

z∈Z



X
u1 (z)z∈ZX
π 1 ∈Π1 (z)π 2 ∈Π2 (z)µ(π 1 , π 2 ).

Therefore, following all recommendations is a best response for the σ̂-trigger agent if and only ifµ is chosen so that

X
z∈Z


u1 (z)
ℙµ,µ̂ [tσ̂ = z] −X
π 1 ∈Π1 (z)π 2 ∈Π2 (z)µ(π 1 , π 2 )⩽0

ˆ∀µ̂ ∈ ∆Π1 (I) .

(9.9)

The crucial observation is that all the probabilities ℙµ,µ̂ [t = z] defined above can be expressed
via the following quantities:

212

y1,σ̂ (z) :=§9.B. APPENDIX: OPTIMAL EFCE AS A LINEAR PROGRAM

Xµ̂1 (π̂ 1 ) ∀z ∈ Z;

ξ1 (σ1 ; z) :=π̂ 1 ∈Π1 (z)Xµ(π 1 , π 2 ) ∀σ1 ∈ Σ1 , z ∈ Z.

π 1 ∈Π1 (σ1 )π 2 ∈Π2 (z)For example, for all z ∈ ZIˆ we can writeℙµ,µ̂ [tσ̂ = z] = ξ1 (σ̂; z)yi,σ̂ (z) + ξ1 (σ1 (z); z).
When deviations relative to Player 2 are brought into the picture, the following two sets ofsymmetric quantities also become relevant:y2,σ̂ (z) :=Xπ̂ 2 ∈Π2 (z)µ̂1 (π̂ 2 ) ∀z ∈ Z;

ξ2 (σ2 ; z) :=Xµ(π 1 , π 2 ) ∀σ2 ∈ Σ2 , z ∈ Z.

π 1 ∈Π1 (z)π 2 ∈Π2 (σ2 )It is now natural to perform a change of variables, and pick (correlated) distributions for the
random variables y1,σ̂ (·), y2,σ̂ (·), ξ1 (·; ·) and ξ2 (·; ·) instead of µ, µ̂1 and µ̂2 . Since there are only apolynomial number (in the game tree size) of combinations of arguments for these new randomvariables, this approach would allow one to remove the redundancy of realization-equivalentnormal-form plans and focus on a polynomially-small search space. In the case of the randomvariables y1,σ̂ and y2,σ̂ , it is clear that the change of variables is possible via the sequence form (vonStengel, 2002). Therefore, the only difficulty is in characterizing the space spanned by ξ1 and ξ2
as µ varies across the probability simplex, which has been the focus of the present chapter.

Chapter 10

Learning optimal extensive-form
correlated equilibria
10.1

Contributions

In this chapter, we propose the first no-regret learning algorithm that guarantees convergence toan optimal EFCE. Specifically, we show that computing optimal EFCE in multiplayer imperfectinformation extensive-form games can be cast as solving a bilinear saddle-point problem. This
unlocks applying rich technology, both theoretical and experimental, developed so far forcomputing bilinear saddle points (for example, Nash equilibria in zero-sum games) for the morechallenging—and much less understood—problem of computing optimal equilibria.
To establish the result, we build on the structural understanding of the polytope of correlationplans Ξ, which we developed in Chapter 9, to cast the computation of optimal EFCE as a max-min
equilibrium over two heavily structured domains: Ξ on the one side, and a Cartesian product of
rescaled sequence-form strategy polytopes on the other. By then leveraging the machinery ofregret circuits, together with the insight regarding the scaled-extension-based decomposition ofΞ, we are able to construct learning algorithms that are able to compute EFCE at significantly
faster and at a significantly larger scale than the best prior approach based on linear programming(Proposition 9.1).
We experimentally evaluate our learning-based method on 23 game instances from 8 different
classes of established parametric benchmark games. Our experiments once again confirm theconclusion that learning dynamics form the basis for some of the most scalable technique available
today to compute equilibrium points in large games, and computation of optimal EFCE inimperfect-information extensive-form games is no exception.
Beyond computational advances, we remark that our bilinear formulation also allows us toobtain last-iterate guarantees to the optimal equilibria when using predictive online gradient214§10.2. OPTIMAL EFCE AS A BILINEAR SADDLE-POINT PROBLEM

descent as the learning algorithm, instead of the time-average guarantees traditionally derived
within the no-regret framework.

10.2

Optimal EFCE as a bilinear saddle-point problem

In this section we show that the problem of computing an optimal (for example, social welfare
maximizing) EFCE can be converted to the problem of finding a Nash equilibrium in a two-player
zero-sum game. This realization is important algorithmically, in that it will enable us to tap into
the richness of approaches studied in Part II of this dissertation to compute an optimal EFCE atscale, for the first time.
Our approach relies on a Lagrangian relaxation of the optimization problem that was given in
Lemma 9.2, reproduced below for convenience:


max


 ξ
s.t.





c⊤ ξ
ξ∈Ξ
max

1
2

(10.1)⊤xσ̂ ∈Qi, ≽j∀ i ∈ [[n]], σ̂ = (j, a) ∈ Σ∗i .

ξ Ai,σ̂ xσ̂ ⩽ 0

We note that the feasible set of the program above does not change if constraint 2 is replaced by
max 0,

maxxσ̂ ∈Qi, ≽jξ ⊤ Ai,σ̂ xσ̂∀ i ∈ [[n]], σ̂ = (j, a) ∈ Σ∗i .

⩽0Hence, using the fact that max{0, maxx∈X f (x)} = maxx∈co{{0},X } f (x) for any linear function fand set X , the following program is equivalent to (10.1):


max


 ξ
s.t.





c⊤ ξ
1

ξ∈Ξ

(10.2)max

2xσ̂ ∈co{{0},Qi, ≽j }

ξ ⊤ Ai,σ̂ xσ̂ ⩽ 0∀ i ∈ [[n]], σ̂ = (j, a) ∈ Σ∗i .

Letting now Qi,σ̂ := co{{0}, Qi, ≽j for all i ∈ [[n]] and σ̂ = (j, a) ∈ Σ∗i , the Lagrangian relaxationof the above program is the saddle-point problem
max
ξ∈Ξ

min

λ∈ℝ⩾0 ,
xi,σ̂ ∈Qi,σ̂ ∀i,σ̂c⊤ ξ − λX Xξ ⊤ Aσ̂ xi,σ̂ .

(10.3)i∈[[n]] σ̂∈Σ∗i

We first point out that the above saddle-point optimization problem admits a solution (ξ ⋆ , x⋆ , λ⋆ ).

215§10. LEARNING OPTIMAL EXTENSIVE-FORM CORRELATED EQUILIBRIA

Proposition 10.1. The problem (10.3) admits a finite saddle-point solution (ξ ∗ , x∗ , λ∗ )Moreover, for all fixed λ > λ∗ , the problems (10.3) and (10.1) have the same value and same
set of optimal solutions.

Proof. Let v be the optimal value of (10.2), which, as we argued, is equal to the optimal valueof (10.1). The Lagrangian of (10.2) is
max
ξ∈Ξ

min

λi,σ̂ ∈ℝ⩾0 ,xi,σ̂ ∈Qi,σ̂∀i∈[[n]],σ̂∈Σ∗ic⊤ ξ −X X

λi,σ̂ ξ ⊤ Aσ̂ xi,σ̂ .

i∈[[n]] σ̂∈Σ∗iNow, making the change of variables x̄i,σ̂ := λi,σ̂ xi,σ̂ , the above problem is equivalent to
max
ξ∈Ξ

min

c⊤ ξ −

x̄i,σ̂ ∈Q̄i,σ̂∀i∈[[n]],σ̂∈Σ∗i

X X

ξ ⊤ Aσ̂ x̄i,σ̂ .

(10.4)i∈[[n]] σ̂∈Σ∗iwhere Q̄i,σ̂ is the conic hull of Qi,σ̂ : Q̄i,σ̂ := [0, +∞)Qi,σ̂ . The problem (10.4) is a bilinear
saddle-point problem, where Ξ is compact and convex and Q̄i,σ̂ is convex. Thus, Sion’s
minimax theorem Sion, 1958 applies, and we have that the value of (10.4) is equal to the valueof the problem
min

max

x̄i,σ̂ ∈Q̄i,σ̂ ξ∈Ξ∀i∈[[n]],σ̂∈Σ∗ic⊤ ξ −X X

ξ ⊤ Aσ̂ x̄i,σ̂ .

(10.5)i∈[[n]] σ̂∈Σ∗iSince this is a linear program (by taking a dual of the inner maximization problem) with a finite
value, its optimum value must be achieved by some x̄ := (. . . , x̄i,σ̂ , . . . ) := (. . . , λi,σ̂ xi,σ̂ , . . . ).
Let λ∗ := maxi,σ̂ λi,σ̂ . Using the fact that 0 ∈ Qi,σ̂ ⊆ Q̄i,σ̂ and clearly ξ ⊤ Ai,σ̂ 0 = 0 for all µ,
the profilex̄′ := (. . . , λ∗ x̄i,σ̂ , . . . )wherex′i,σ̂ := 0 +λi,σ̂(xi,σ̂ − 0)λ∗is also an optimal solution of (10.5). Therefore, for any λ ⩾ λ∗ , x′ := (x′1 , . . . , x′n ) is an optimal
solution for the minimizer in (10.3) that achieves the value of (10.1), so (10.1) and (10.3) havethe same value.
Now take λ > λ∗ , and suppose for contradiction that (10.3) admits some optimal µ ∈ Ξ that is
not optimal in (10.1). Then, either c⊤ µ < v, or µ violates some constraint maxxi,σ̂ µ⊤ Ai,σ̂ xi,σ̂ ⩽0. The first case is impossible because then setting xi,σ̂ = 0 for all i yields value less than v
in (10.3). In the second case, since we know that (10.3) and (10.1) have the same value when

216

§10.3. NO-REGRET LEARNING ALGORITHM

λ = λ∗ , we havec⊤ ξ − λ max
x∈X

10.3

X X

ξ ⊤ Ai,σ̂ xi,σ̂ < c⊤ ξ − λ∗ max
x∈Xi∈[[n]] σ̂∈Σ∗i

X X

ξ ⊤ Ai,σ̂ xi,σ̂ ⩽ v.

i∈[[n]] σ̂∈Σ∗iNo-regret learning algorithm

Since (10.3) is a bilinear saddle point, its solution can be found by self play of no-externalregret algorithms. However, a challenge that arises in this approach is that the domain of theminimization is unbounded—the Lagrange multiplier is allowed to take any nonnegative value.
Nevertheless, we show in the theorem below that it suffices to set the Lagrange multiplier to a
fixed value (that may depend on the time horizon); appropriately setting that value will allow us
to trade-off between the equilibrium gap and the optimality gap. Before we proceed, we remarkthat the minimization domain in (10.3) is the Cartesian product of Qi,σ̂ over all possible (i, σ̂).
Hence, a no-regret algorithm for the minimization problem can be obtained by instantiating
individual no-regret algorithm for each Qi,σ̂ using the approach described in Section 4.2.2; thisPP
(T )justifies the notation i∈[[n]] σ̂∈Σ∗ RegQi,σ̂ used for the regret of the min deviator below.
i

Theorem 10.1. Suppose that the max deviator in the saddle-point problem (10.3) incurs regret
P
P
(T )
(T )
RegΞ and the min deviator incurs regret i∈[[n]] σ̂∈Σ∗ RegQi,σ̂ after T ∈ ℕ repetitions, foriPTa fixed λ = λ(T ) > 0. Then, the average mediator strategy Ξ ∋ ξ̄ := T1 t=1 ξ (t) satisfies thefollowing:
1. For any strategy ξ ∗ ∈ Ξ such that maxi∈[[n]],σ̂∈Σ∗i maxx∗i,σ̂ ∈Qi,σ̂ (ξ ∗ )⊤ Ai,σ̂ x∗i ⩽ 0,


X X
1
(T
)
(T
)
RegQi,σ̂ ;c⊤ ξ̄ ⩾ c⊤ ξ ∗ − RegΞ +T∗i∈[[n]] σ̂∈Σi

2. The equilibrium gap of ξ̄ decays with a rate of λ−1 :


′
⊤
X
X
′
max
c
(ξ
−
ξ
)
1
ξ,ξ ∈Ξ
(T )Reg(T ) +maxmax ξ̄ Ai,σ̂ x∗i ⩽+RegQi,σ̂.
Ξx∗∈Qi,σ̂
λλT
i∈[[n]],σ̂∈Σ∗ii∗⊤i∈[[n]] σ̂∈Σi

Proof. Let ξ̄ ∈ Ξ be the average strategy of the mediator, and x̄i ∈ Qi,σ̂ be the average strategy
of each deviator i ∈ [[n]], σ̂ ∈ Σ∗i over the T iterations. We first argue about the approximate
optimality of ξ̄. In particular, we have that§10. LEARNING OPTIMAL EXTENSIVE-FORM CORRELATED EQUILIBRIA

217






 1 X X
X X
(T )
(T )
RegQi,σ̂ + RegΞ (10.6)
ξ ⊤ Ai,σ̂ x̄i − c⊤ ξ̄ ⩾ max c⊤ ξ − λ
 T
ξ∈Ξ 
∗
∗
i∈[[n]] σ̂∈Σii∈[[n]] σ̂∈Σi


X X
X X
1
(T )
(T )
(ξ ∗ )⊤ Ai,σ̂ x̄i − ⩾ c⊤ ξ ∗ − λRegQi,σ̂ + RegΞ (10.7)
T∗∗i∈[[n]] σ̂∈Σii∈[[n]] σ̂∈Σi


X X
X X
1
(T )
(T )
⩾ c⊤ ξ ∗ − λRegQi,σ̂ + RegΞ max (ξ ∗ )⊤ Ai,σ̂ x∗i − x∗∈Qi,σ̂
Ti∗∗i∈[[n]] σ̂∈Σii∈[[n]] σ̂∈Σi


1 X X
(T )
(T )
⩾ c⊤ ξ ∗ − 
RegQi,σ̂ + RegΞ ,(10.8)
T∗i∈[[n]] σ̂∈Σiwhere:• (10.6) follows from the fact that the Lagrangian of the problem,
L : Ξ × X ∋ (ξ, (xi,σ̂ )ni=1 ) 7→ c⊤ ξ − λ

X Xξ ⊤ Ai,σ̂ xi,σ̂ ,i∈[[n]] σ̂∈Σ∗isatisfies


X X
1
(T
)
(T
)
RegQi,σ̂ + RegΞ , (10.9)
max L(ξ ∗ , (x̄i )ni=1 ) − ∗min
L(ξ̄, (x∗i )ni=1 ) ⩽ ξ∗ ∈ΞT(xi )n∈X
i=1∗i∈[[n]] σ̂∈Σiin turn implying (10.6) since
X X⊤

max ξ̄ Ai,σ̂ x∗i,σ̂ ⩾x∗∈Qi,σ̂
ii∈[[n]] σ̂∈Σ∗
i

X X

⊤

ξ̄ Ai,σ̂ 0 = 0;i∈[[n]] σ̂∈Σ∗i• (10.7) uses the notation ξ ∗ to represent any equilibrium strategy optimizing the objectivec⊤ ξ; and• (10.8) follows from the fact that, by assumption, ξ ∗ satisfies the equilibrium constraint
maxx∗i ∈Qi,σ̂ (ξ ∗ )⊤ Ai,σ̂ x∗i ⩽ 0 for any Player i ∈ [[n]], σ̂ ∈ Σ∗i , as well as the nonnegativityof the Lagrange multiplier.
This establishes Item 1 of the statement.
Next, we analyze the equilibrium gap of ξ̄. Consider any mediator strategy ξ ∈ Ξ such that
ξ ⊤ Ai,σ̂ xi,σ̂ ⩽ 0 for any xi,σ̂ ∈ Qi,σ̂ and deviator i ∈ [[n]], σ̂ ∈ Σ∗i . By (10.9),

218§10.3. NO-REGRET LEARNING ALGORITHM

c⊤ ξ − λX Xξ ⊤ Ai,σ̂ x̄i − c⊤ ξ̄ + λi∈[[n]] σ̂∈Σ∗
i

X X

max

∈Qi,σ̂
x∗i,σ̂i∈[[n]] σ̂∈Σ∗i⊤ξ̄ Ai,σ̂ x∗i,σ̂



X X
1
(T ) 
(T )⩽RegQi,σ̂ . (10.10)
RegΞ +
T∗i∈[[n]] σ̂∈ΣiBut, by the equilibrium constraint for ξ, it follows that ξ ⊤ Ai,σ̂ xi,σ̂ ⩽ 0 for any xi,σ̂ ∈ Qi,σ̂ andPP
deviator i ∈ [[n]], σ̂ ∈ Σ∗i , in turn implying that i∈[[n]] σ̂∈Σ∗ ξ ⊤ Ai,σ̂ xi,σ̂ ⩽ 0. So, combiningiwith (10.10),

λ



X
X
1
⊤
(T )
(T )
RegQi,σ̂ + RegΞ .
max ξ̄ Ai,σ̂ x∗i,σ̂ ⩽ c⊤ ξ̄ − c⊤ ξ + 
∗ ∈Q
x
T
i,σ̂
i
∗
∗

X X

(10.11)i∈[[n]] σ̂∈Σii∈[[n]] σ̂∈Σi⊤⊤Finally, given that maxx∗i′ ∈Xi′ ξ̄ Ai′ x∗i′ ⩾ ξ̄ Ai′ di′ = 0 for any deviator i′ , it follows that
X X

max

⊤

x∗
∈Qi,σ̂
i,σ̂i∈[[n]] σ̂∈Σ∗iξ̄ Ai,σ̂ x∗i,σ̂ ⩾

max

max

x∗
∈Qi,σ̂
i∈[[n]],σ̂∈Σ∗
i,σ̂
i

⊤

ξ̄ Ai,σ̂ x∗i,σ̂ ,and (10.11) implies Item 2 of the statement.
As a result, if we can simultaneously guarantee that λ(T ) → +∞ and


X X
1
(T )
(T ) 
RegΞ +
RegQi,σ̂ → 0,T∗

as T → +∞,i∈[[n]] σ̂∈Σi

Theorem 10.1 guarantees that both the optimality gap (Item 1) and the equilibrium gap (Item2) converge to 0. We show that this is indeed possible in the sequel (Corollaries 10.1 and 10.2),
obtaining favorable rates of convergence as well.
It is important to stress that while there exists a bounded critical Lagrange multiplier for our
problem (Proposition 10.1), thereby obviating the need for truncating its value, such a bound is
not necessarily polynomial.

10.3.1 Construction of weakly-predictive algorithms via regret circuits (Chapter 4)
Next, we combine Theorem 10.1 with suitable regret minimization algorithms in order toguarantee fast convergence to optimal equilibria. A particularly natural approach in this case is
via the regret circuits introduced in Chapter 4. Indeed, the regret minimization problem overthe domain of the minimization in the saddle-point problem is the Cartesian product of thesets Qi,σ̂ , each of which is the convex hull between the singleton set {0} and the sequence-form§10. LEARNING OPTIMAL EXTENSIVE-FORM CORRELATED EQUILIBRIA

219

polytope Qi, ≽j , for which several no-regret algorithms are known (including those presented inthis dissertation, in Chapters 4 to 7).
Similarly, as we discussed in Chapter 9, the domain Ξ of the maximization problem—thepolytope of correlation plans—can be expressed as (possibly a linear projection of an exponentiallylong) composition of scaled extension operations and probability simplexes. Since scaled extension
admits a regret circuit, we can repeatedly use the circuit in combination with no-regret algorithms
for probability simplexes (for example, Discounted RM or PRM+ ) to construct a parameter-freeno-external-regret algorithm that is rather competitive in practice.
Given that the deviators’ observed utilities have range OT (λ), the regret of each deviator
under suitable choices of local algorithm (for example, Discounted RM or PRM+ as we use in the√experiments) will grow as OT (λ T ). Selecting a bound of λ := T 1/4 on the Lagrange multiplier,so as to optimally trade off Items 1 and 2 of Theorem 10.1, leads to the following conclusion.
Corollary 10.1. There exist regret minimization algorithms such that when employed in thePTsaddle-point problem (10.3), the average strategy of the mediator ξ̄ := T1 t=1 ξ (t) converges
to the set of optimal equilibria at a rate of T −1/4 . The per-iteration complexity is polynomialin the number of scaled extensions required to represent X (see Chapter 9).

10.3.2

Faster rates using RVU-predictive algorithms (Chapter 5)

As already discussed in Chapter 5, proximal algorithms such as predictive online mirror descent
beget RVU-predictive algorithms, which typically imply faster convergence rate compared toonly weakly-predictive algorithm such as those constructed in Section 10.3.1. In particular, using
predictive mirror descent we can guarantee that the sum of the agents’ regrets in the saddle-point√
problem (10.3) will now grow as OT (λ), instead of the previous bound OT (λ T ). Thus, letting
λ = T 1/2 leads to the following improved rate of convergence.
Corollary 10.2 (Improved rates using predictivity). There exist regret minimization algorithms
PTthat guarantee that the average strategy of the mediator ξ̄ := T1 t=1 ξ (t) converges to theset of optimal equilibria at a rate of T −1/2 . The per-iteration complexity is analogous to
Corollary 10.1.

Proof sketch. By the RVU bound (Syrgkanis, Agarwal, Luo, and Schapire, 2015; S. Rakhlin
diam2Ξ + diam2X
and Sridharan, 2013), the sum of the agents’ regrets can be bounded as, for aη
sufficiently small η = OT (1/λ), where diamΞ and diamX denote the ℓ2 -diameter of Ξ and X,
respectively. Thus, taking η = Θ(1/λ) to be sufficiently small implies the statement.
Improving the T −1/2 rate of Corollary 10.2 is an interesting direction for future research.

220

10.3.3§10.3. NO-REGRET LEARNING ALGORITHM

Remarks on last-iterate convergence

The results we have stated thus far apply for the average strategy produced by the no-regret
algorithm—a typical feature of traditional guarantees in the no-regret framework. In contrast,
there is a recent line of work that endeavors to recover last-iterate guarantees as well (Daskalakis
and Panageas, 2019; Gorbunov, Loizou, and Gidel, 2022; Abe, Sakamoto, and Iwasaki, 2022; Cai,
Oikonomou, and Zheng, 2022; Azizian, Iutzeler, Malick, and Mertikopoulos, 2021; C. Wei, C. Lee,
M. Zhang, and Luo, 2021; C.-W. Lee, Kroer, and Luo, 2021; Golowich, Pattathil, and Daskalakis,
2020; Lin, Zhou, Mertikopoulos, and Jordan, 2020; Gorbunov, Berard, Gidel, and Loizou, 2022).
Yet, despite many efforts, the known last-iterate guarantees of no-regret learning algorithms
apply only for restricted classes of games, such as two-player zero-sum games. There is an
inherent reason for the limited scope of those results: last-iterate convergence is inherently tied toNash equilibria, which in turn are hard to compute in general games (Daskalakis, Goldberg, andPapadimitriou, 2009; Chen, Deng, and Teng, 2009)—let alone computing an optimal one (Gilboa
and Zemel, 1989; Conitzer and Sandholm, 2008). Indeed, any given joint strategy profile ofthe deviators induces a product distribution, so ϵ-iterate convergence requires at the very least
computing an ϵ-Nash equilibrium.
Proposition 10.2 (Informal). Any uncoupled learning dynamics without a mediator require
superpolynomial time to guarantee ϵ-last-iterate convergence, for a sufficiently small ϵ =(1/ poly), even for two-deviator normal-form games, unless PPAD ⊆ P.

There are also unconditional exponential communication-complexity lower bounds for uncoupled
methods (Babichenko and Rubinstein, 2022; Hirsch, Papadimitriou, and Vavasis, 1989; Roughgarden and Weinstein, 2016; Hart and Mansour, 2010), as well as other pertinent impossibility
results (Hart and Mas-Colell, 2003; Milionis, Papadimitriou, Piliouras, and Spendlove, 2022) that
document the inherent persistence of limit cycles in general-sum games. In contrast, an important
advantage of our formulation using Ξ is that we can guarantee last-iterate convergence to bilinear√
saddle points, using the known bound of OT (λ/ T ) for the last-iterate gap of predictive online
gradient descent can be employed (Anagnostides, Panageas, Farina, and Sandholm, 2022).
Theorem 10.2 (Last-iterate convergence to bilinear saddle points). There exist regret
minimization algorithms that guarantee that the last strategy of the mediator ξ (T ) converges
to the set of optimal equilibria at a rate of T −1/4 . The per-iteration complexity is analogous
to Corollaries 10.1 and 10.2.

As such, our mediator-augmented learning paradigm bypasses the hardness of Proposition 10.2
since last-iterate convergence is no longer tied to convergence to Nash equilibria.

221§10. LEARNING OPTIMAL EXTENSIVE-FORM CORRELATED EQUILIBRIA

10.4

Experimental evaluation

We extensively evaluate the empirical performance of the regret-circuit-based learning algorithm
given in Section 10.3.1 for computing optimal EFCE across 23 game instances and different
objective functions including social welfare. The game instances we use are described in detail in
Appendix A, and belong to following eight different classes of established parametric benchmark
games, each identified with an alphabetical mnemonic: B – Battleship, D – Liar’s dice, GL– Goofspiel, K – Kuhn poker, L – Leduc poker, RS – ridesharing game, S – Sheriff, TP –double dummy bridge game.
For each of the 23 games, we compare the runtime required by a commercial linear programming
method instantiated on the optimal EFCE LP given in Proposition 9.1 (and originally proposedby B. H. Zhang, Farina, Celli, and Sandholm, 2022) (‘LP’) and the runtime required by ourregret-circuit-based learning algorithm defined in Section 10.3.1 (‘Ours’) for computing ϵ-optimalequilibrium points.
Game

Max. Pl.1’s utility Max. Pl.2’s utility
LP
Ours
LP
Ours

Max. soc. welf.
LP
Ours

B2222
B2322
B2323
B2324

0.00s
9.00s
3m 54stimeout

0.00s
0.02s
9.00s
1.60s
3m 40s
44.87stimeout 10m 48s

0.01s
1.23s
48.40s
9m 3s

0.00s
0.05s
9.00s
4.63s
4m 9s 1m 27s
timeout 10m 21s

S2122
S2123
S2133
S2254
S2264

0.00s
1.00s
4.00s
timeout
timeout

0.01s
0.09s
0.52s
2m 10stimeout

0.00s
1.00s
3.00s
timeout
timeout

0.02s
0.34s
1.31s
timeout
timeout

0.00s
1.00s
3.00s
timeout
timeout

0.02s
0.15s
0.49s
3m 34stimeout

RS212
RS213
RS222
RS223

0.00s
timeout
0.00s
timeout

0.00s
35.37s0.00stimeout

0.00s
timeout
0.00s
timeout

0.00s
32.49s0.00stimeout

0.00s
timeout
0.00s
timeout

0.00s
23.37s0.00stimeout

Game

Max. Pl.1’s utility Max. Pl.2’s utility Max. Pl.3’s utility
LP
Ours
LP
Ours
LP
Ours

D32
D33

12.00stimeout

0.40s
timeout

11.00stimeout

0.35s
timeout

10.00stimeout

0.66s
timeout

GL33

0.00s

0.01s

0.00s

0.01s

0.00s

0.01s

K35

57.00s0.55s55.00s1.03s60.00s

1.26s

L3132
L3133
L3151
L3223
L3523

8m 18s
21m 25s
timeout2m 2stimeout

6.10s
6.84s
timeout
5.52s
timeout

8m 57s
21m 43s
timeout
1m 50stimeout

7.65s
10.76stimeout
6.46s
timeout

7m 35s
19m 58s
timeout
2m 0s
timeout

6.78s
10.28stimeout
5.94s
timeoutTP3

timeout

13.46stimeout

14.25stimeout

14.48sTable 10.1: Experimental comparison between our learning-based approach (‘Ours’, Section 10.3.1)
and using the commercial LP solver Gurobi on the linear programming formulation of optimal
EFCE (Proposition 9.1), both for computing an optimal EFCE within an optimality and feasibility
tolerance set to 1% of the payoff range of the game.

Table 10.1 and Table 10.2 show experimental results for the case in which the threshold ϵ is set
to be, respectively, 1% and 0.1% of the payoff range of the game. Each table is split into a leftpart, which contains two-player general-sum games, and a right part with three-player zero-sum
games. For each game, three objective functions are considered:• For the two-player general-sum games, we consider the three objectives of (i) maximizing
Player 1’s utility, (ii) maximizing Player 2’s utility, and (iii) maximizing social welfare, that is,
the sum of the two players’ utilities.
• For the three-player zero-sum games, we consider the three objectives corresponding to222§10.4. EXPERIMENTAL EVALUATION

maximizing Player 1, 2, and 3’s utility, respectively.
Each row corresponds to a game, whose identifier begins with the alphabetical mnemonic of thegame class.
The regret-circuit-based construction detailed in Section 10.3.1 allows a degree of flexibility as
to what no-regret algorithms are used for each of the probability simplexes that make up Ξ andthe Qi,σ̂ . In the experimental tables, the column ‘Ours’ reports the minimum between the timerequired by the regret-circuit-based algorithm instantiated with Discounted RM and PRM+ at eachdecision point. For each run of the algorithms, the timeout was set at one hour.
We observe that our learning-based approach is faster—often by more than an order ofmagnitude—and more scalable than the linear program. This shows the promise of ourcomputational approach.
Game

Max. Pl.1’s utility Max. Pl.2’s utility
LP
Ours
LP
Ours

Max. soc. welf.
LP
Ours

B2222
B2322
B2323
B2324

0.00s
9.00s
4m 12stimeout

0.00s
9.00s
3m 40stimeout

0.16s
11.04s
3m 41stimeout

0.00s
9.00s
4m 9s
timeout

0.10s
10.87s3m 35stimeout

0.03s
4.11s
1m 28stimeout

S2122
S2123
S2133
S2254
S2264

0.00s
2.00s
5.00s
timeout
timeout

0.04s
0.24s
1.12s
6m 24stimeout

0.00s
1.00s
3.00s
timeout
timeout

0.07s
1.63s
4.80s
timeout
timeout

0.00s
1.00s
4.00s
timeout
timeout

0.07s
0.37s
0.95s
9m 2s
timeout

RS212
RS213
RS222
RS223

0.00s
timeout
0.00s
timeout

0.00s
5m 1s
0.02s
timeout

0.00s
timeout
0.00s
timeout

0.01s
4m 19s0.02stimeout

0.00s
timeout
0.00s
timeout

0.00s
2m 28s0.02stimeout

Game

Max. Pl.1’s utility Max. Pl.2’s utility Max. Pl.3’s utility
LP
Ours
LP
Ours
LP
Ours

D32
D33

14.00stimeout

0.92s
timeout

GL33

0.00s

K35

1m 5s

L3132
L3133
L3151
L3223
L3523
TP3

13.00stimeout

0.74s
timeout

12.00stimeout

1.60s
timeout

0.06s

0.00s

2.99s

1m 3s

0.05s

0.00s

0.05s

4.80s

1m 4s

9m 41s26m 52s
timeout
2m 38stimeout

26.68s22.31stimeout
20.44stimeout

4.66s

10m 28s
27m 22s
timeout
2m 32stimeout

27.57s39.44stimeout
28.27stimeout

9m 54s
25m 32s
timeout
2m 31stimeout

30.00s
40.75stimeout
49.95stimeout

timeout

26.28stimeout

40.64stimeout

41.96sTable 10.2: Experimental comparison between our learning-based approach (‘Ours’, Section 10.3.1)
and using the commercial LP solver Gurobi on the linear programming formulation of optimal
EFCE (Proposition 9.1), both for computing an optimal EFCE within an optimality and feasibility
tolerance set to 0.1% of the payoff range of the game.

Chapter 11

Learning dynamics for teamcoordination and collusion
The positive results regarding the geometry of correlated strategies we investigated in Chapter 9
also imply positive results about the computation of optimal team strategies in imperfectinformation two-team zero-sum games, helping answer questions such as:
How should two players colluding against a third at a poker table play?
Or, how would the two defenders in Bridge (who are prohibited from communicating privately
during the game) play optimally against the declarer?
Depending on the amount of communication allowed among the team members, we canidentify at least three solution concepts.
• If the team members can freely (and privately) communicate during play, the team effectively
becomes a single player. Hence, all the tools we have seen so far (for example, learning
an optimal strategy using CFR or linear programming) directly apply. We will refer to thisequilibrium as ‘Team Nash equilibrium’.
• If the team members cannot communicate at all, then the strategies of the team members
should be picked as the pair of strategies that maximizes the expected utility of the teamagainst a best-responding agent. This solution concept is called a team maxmin equilibrium(TME). As we show in Section 11.2, the minmax theorem does not hold in general for TME.
So, perhaps, the term “equilibrium” should be used carefully when referring to TME.
• The third case is intermediate, and models the setting where the team members have an
opportunity to discuss and agree on tactics before the game starts, but are otherwise unable224§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION

to communicate during the game, except through their publicly-observed actions. Thismodels, for example, multiplayer poker in the presence of collusion, and the game of bridge.
This solution concept is called a team maxmin equilibrium with correlation device (TMECor).
TMECor can be thought of as follows. Before playing, the team members get together insecret, and discuss about tactics for the game. They come up with m possible plans, each ofwhich specifies a deterministic strategy for each team member, and write them down inm separate envelopes. Then, just before the game starts, they together pick one of the menvelopes according to a shared probability distribution, and play according to the plan in
the chosen envelope.
It is important to realize that the sampling of the envelope can happen even if the teammembers cannot communicate before the game starts, as long as they can agree on someshared signal, such as for example a common clock. With that signal as input, the teammembers could seed a random number generator and use that to agree on the same random
envelope to use, without having communicated among each other.
Table 11.1 compares the properties of the three solution concepts defined above. Generally
speaking, as the amount of allowed communication increases (from TME to TMECor to TeamNash), the utility of the team increases, while the complexity of computing the optimal strategy
for the team decreases. In particular, as the first three rows of Table 11.1 show, the problem ofcomputing a TME strategy for the team is significantly less well-behaved than TMECor.
For all these reasons, we believe that TMECor is often a superior solution concept for adversarial
team games when no in-game private communication is allowed. It yields higher utility to theteam, is computationally and game-theoretically better behaved, and can be implemented inmany practical interaction.

Convex problem
Bilinear saddle-point problem
Low-dimensional strat. polytope
Minmax theorem
Team utility
Complexity

TME
no communication
ever

TMECorno communication
during play

Team Nash Eq.
private communication
during play

✗
✗
✗
✗
low
very hard✓✓✓✓highersometimes hard✓✓✓✓highestpolynomial

Table 11.1: Comparison between TME, TMECor and Team Nash equilibrium.

§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION

11.1

225

Contributions and related work

After demonstrating that TME does not satisfy the minmax theorem, in this chapter we showthat learning dynamics for the teams can be devised to compute TMECor, by connecting TMECor
computation with EFCE-related technology introduced in Chapters 9 and 10. As we show, thisapproach leads to theoretical and practical state-of-the-art learning algorithms for computing
TMECor strategies in imperfect-information two-team zero-sum games, adding to the conclusion,
already drawn several times in this dissertation, that uncoupled learning dynamics often leadto both the theoretical and practical state-of-the-art technique for computing equilibria inimperfect-information extensive-form games.
The study of TME dates back to at least the work by von Stengel and Koller (1997). The studyof TMECor—the focus of this chapter—is more recent. Basilico, Celli, De Nittis, and Gatti (2017)
show that TMECor can lead to significantly higher utility (up to a factor linear in the number ofgame tree leaves) than TME. The study of the computational aspects of TMECor was initiated by
Celli and Gatti (2018). Until recently, the best techniques for solving adversarial team games inabsence of communication were based on column generation (Farina, Celli, Gatti, and Sandholm,
2018; Farina, Celli, Gatti, and Sandholm, 2021; Y. Zhang, An, and Černỳ, 2021; B. H. Zhang, Farina,
Celli, and Sandholm, 2022). Those techniques work well in some small and medium-sized games
in practice, but generally have no or weak theoretical guarantees. More recently, other techniques
emerged. B. H. Zhang and Sandholm (2022b) developed an algorithm for solving adversarial
team games based on a novel tree decomposition of each player’s strategy space, and use it to devise
a linear program. They show parameterized complexity bounds that scale with the amount of
uncommon information among team members. Simultaneously, Carminati, Cacciamani, Ciccone,and Gatti (2022) developed an algorithm for converting the game into a strategically equivalent(but exponentially-larger) two-player game with perfect recall, inspired by prior research in themultiagent reinforcement learning community (e.g., Nayyar, Mahajan, and Teneketzis, 2013;
Sokota, Lockhart, Timbers, Davoodi, D’Orazio, Burch, Schmid, Bowling, and Lanctot, 2021).
The technique we develop in this chapter is based on learning dynamics, and yields analgorithm significantly faster than all prior approaches.

11.2

Failure of minmax theorem for team maxmin equilibrium

As mentioned in the introduction, one of the major shortcomings of TME (at least from a gametheoretic point of view) is that it fails the minmax theorem. We demonstrate this by explicitly
showing a game in which the maxmin and minmax values of the game are different.
Consider the matching pennies game of Figure 11.1 (Left), where a team ▲ of two players(denoted 1▲ and 2▲ respectively) compete against an opponent. Each player has a penny. First,
the opponent (denoted ▼) secretly turns their penny heads (H) or tails (T). Then, Player 1▲ secretly

226§11.2. FAILURE OF MINMAX THEOREM FOR TEAM MAXMIN EQUILIBRIUM

H

LegendOpponent, ▼TTeam ▼
Team ▲

Player 1▲
H

T

H

Terminal history

Y Information set

TPlayer 2▲
H T

H T

H T

(1, −1) (0, 0) (0, 0) (0, 0)

H T(0, 0) (0, 0) (0, 0) (1, −1)
Figure 11.1: Matching pennies game.

turns their penny heads or tail. Finally, Player 2▲ turns their penny heads or tail. After all pennies
have been turned, all of them are revealed. If all three pennies match (that is, they are all heads orthey are all tails), the team wins a payoff of 1 and the opponent suffers a loss of −1. Otherwise,
nobody gains or loses anything.
Maxmin value We will use x, y, z to denote strategies for Players 1▲, 2▲, and ▼, respectively.
The TME strategy for team ▲ is the solution to the maxmin problem

vmaxmin :=







max


x,y






 min
z
s.t.





s.t.









3

xH · yH · zH + xT · yT · zTzH + zT = 1
2 zH , zT ⩾ 0
1(♠)

xH + xT = 1
4 yH + yT = 1
5 xH , xT , yH , yT ⩾ 0.

Lemma 11.1. The solution to the maxmin problem defined in (♠) is vmaxmin = 1/4.

Proof. The internal minimization problem in (♠) is minimizing a linear function over the
2-simplex. So, the solution to the internal minimization problem is the minimum of thefunction over the two vertices, that is, min{xH · yH , yT · yT }. Hence, we can rewrite (♠) as

max


x,y



s.t.
vmaxmin =






min{xH · yH , yT · yT }xH + xT = 1
4 yH + yT = 1
5 xH , xT , yH , yT ⩾ 0.
3

§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION227

We can now perform the substitutions xT = 1 − xH and yT = 1 − yH (justified by 3 and 4 ),and get to(vmaxmin =max

min{xH · yH , (1 − xH ) · (1 − yH )}

s.t.

5

x,y

0 ⩽ xH ⩽ 1,

0 ⩽ yH ⩽ 1.

Not that the problem is completely symmetric: if (x∗H , yH∗ ) is an optimal solution, then(1 − x∗H , 1 − yH∗ ) is also optimal. Hence, there exists at least one optimal solution in which
xH · yH ⩽ (1 − H) · (1 − yH )⇐⇒xH + yH ⩽ 1

and consequently

vmaxmin =




 max
x,y

xH · yH

s.t.




6

xH + yH ⩽ 1
5 0 ⩽ xH ⩽ 1, 0 ⩽ yH ⩽ 1.

Since xH is nonnegative, the objective is maximized when yH is maximized, which happens for
yH = 1 − xH . So,(
vmaxmin =max

xH · (1 − xH )

s.t.

5

xH

0 ⩽ xH ⩽ 1,

which is the maximum of a one-dimensional parabola. Taking the gradient of the objective, it’s
immediate to check that the maximum is obtained when xH = 1/2, and hence the statement of
the lemma follows.
Minmax value Conversely, the TME strategy for the opponent is the solution to the maxmin
problem

vminmax :=











 min
z







s.t.





max


 x,y


s.t.






xH · yH · zH + xT · yT · zT
xH + xT = 1
yH + yT = 1
3 xH , xT , yH , yT ⩾ 0.
1
2

zH + zT = 1
5 zH , zT ⩾ 0
4

(♣)

228

§11.3. TMECOR AS A BILINEAR SADDLE-POINT PROBLEM

Lemma 11.2. The solution to the minmax problem defined in (♣) is vminmax = 1/2.

Proof. The choices (xH , xT , yH , yT ) = (1, 0, 1, 0) and (xH , xT , yH , yT ) = (0, 1, 0, 1) are feasible forthe internal maximization problem in (♣). Hence, we have


 min
z
vminmax ⩾s.t.
max{zH , zT }(zH + zT = 1 =5 zH , zT ⩾ 0

min

max{zH , 1 − zH }

s.t.

5

z

4

0 ⩽ zH ⩽ 1.

It’s immediate to see that the solution to the one-dimensional optimization problem on theright-hand side is 1/2, attained for zH = 1/2. So vminmax ⩾ 1/2. To complete the proof, it isenough to show that the there exists an assignment of z for which (♣) attains the value 1/2.
To that end, consider the feasible assignment zH = zT = 1/2. Then, the objective of (♣) is

max


x,y



s.t.






1/2 · x

H · yH +1/2 · xT · yTxH + xT = 1
2 yH + yT = 1
3 xH , xT , yH , yT ⩾ 0.
1

(

max

1/2(x · y + (1 − x )(1 − y ))
H
H
H
H

s.t.

3

max

(xH − 1/2)(yH − 1/2) + 1/4

s.t.

3

x,y

=

(

x,y

=

0 ⩽ xH ⩽ 1,

0 ⩽ xH ⩽ 1,

0 ⩽ yH ⩽ 1.

0 ⩽ yH ⩽ 1.

= 1/2,
where the last equality follows from since the product in the objective is maximized for
xH = yH = 1.
Given that the maxmin and minmax values are different, we conclude that, unlike regularminmax equilibrium, TME does not satisfy the minmax theorem in general.

11.3

TMECor as a bilinear saddle-point problem

In this section, we show how TMECor in imperfect-information two-team zero-sum games canbe cast as a bilinear saddle-point problem over polytopes embedded in Euclidean spaces ofpolynomial dimension in the game size. To simplify the exposition, we will assume that theteams—denoted ▲ and ▼ respectively—each consist of two team members, denoted 1▲, 2▲ and1▼, 2▼ respectively. The case where a team has less or more players extends directly; in theexperimental evaluation we test on teams of varying size.

§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION

229

Similar to maxmin strategies in regular (non-team) two-player zero-sum games, a TMECor
strategy for team (say, ▲) is a distribution µ▲ ∈ ∆Π1▲ ×Π2▲ over tuples of deterministic strategies
for the team members’ (1▲ and 2▲), such that it maximizes the team’s expected expected utility
against a best-responding, independent opposing team:
max

min

µ▲ ∈∆Π1▲ ×Π2▲ µ▼ ∈∆Π1▼ ×Π2▼hi
𝔼(π1▲ ,π2▲ )∼µ▲ 𝔼(π1▼ ,π2▼ )∼µ▼ u▲ (π 1▲ , π 2▲ , π 1▼ , π 2▼ ) ,(11.1)

where u▲ = u1▲ + u2▲ denotes the utility function of team ▲, defined as the sum of the utilities of
its team members.
By definition of expectation, the objective function in (11.1) is bilinear in µ▲ and µ▼ . Thisshows that the computation of a TMECor is a bilinear saddle-point problem over convex polytopes(and therefore in particular it can be converted into a linear program, which is itself a convex
problem, cf. Table 11.1). Another consequence of the bilinear saddle-point structure is the factthat the minmax theorem must hold for TMECor strategies, unlike TME, as we have hinted in
Section 11.2. However, as formulated, the optimization problem (11.1) has the major drawbackof requiring optimization over exponentially-large domains, since so is typically the numberof deterministic strategies of every player. We show in the next subsection that by operating
an appropriate change of variables, we can retain the convexity-concavity and the bilinear
saddle-point structure, while at the same time gaining that the domains of the maximizing andminimization problems belong to a space of polynomial dimension in the game tree size.

11.3.1

Realization vectors and low-dimensional parameterization

Letting u▲ (z) = u1▲ (z) + u2▲ (z) denote the utility received by team ▲ at terminal node z ∈ Z,
pc (z) denote the product of the probability of all chance actions on the path from the root of thegame tree to terminal node z, and σi (z) denote the last sequence on the path from the root of thegame tree to z, by definition we have that the utility u▲ (π 1▲ , π 2▲ , π 1▼ , π 2▼ ) can be written asXz∈Zu▲ (z) · pc (z) · π 1▲ [σ1▲ (z)] · π 2▲ [σ2▲ (z)] · π 1▼ [σ1▼ (z)] · π 2▼ [σ2▼ (z)]
{z
} |
{z
}
|
probability of all ▲’s
actions on the pathfrom root to z

probability of all ▼’s
actions on the pathfrom root to z

By using (11.2) together with the independence of µ▲ and µ▼ , the expected utilityhiū▲ (µ▲ , µ▼ ) := 𝔼(π1▲ ,π2▲ )∼µ▲ 𝔼(π1▼ ,π2▼ )∼µ▼ u▲ (π 1▲ , π 2▲ , π 1▼ , π 2▼ )satisfies the equation(11.2)

230§11.3. TMECOR AS A BILINEAR SADDLE-POINT PROBLEM

ū▲ (µ▲ , µ▼ ) =

X

h
i
h
i
u▲ (z) · pc (z) · 𝔼µ▲ π 1▲ [σ1▲ (z)]·π 2▲ [σ2▲ (z)] · 𝔼µ▼ π 1▼ [σ1▼ (z)]·π 2▼ [σ2▼ (z)] ,z∈Z(11.3)
where for brevity we used 𝔼µ▲ and 𝔼µ▼ as shorthands for 𝔼(π1▲ ,π2▲ )∼µ▲ and 𝔼(π1▲ ,π2▲ )∼µ▲ ,respectively. The above formulation shows that it is not necessary to optimize over µ▲ and µ▼
directly, as the objective function only depends bilinearly on the aggregate quantitieshiω ▲ [z] := 𝔼µ▲ π 1▲ [σ1▲ (z)]·π 2▲ [σ2▲ (z)] =Xµ▲ [π 1▲ , π 2▲ ] · π 1▲ [σ1▲ (z)] · π 2▲ [σ2▲ (z)],(π 1▲ ,π 2▲ )hi
ω ▼ [z] := 𝔼µ▼ π 1▼ [σ1▼ (z)]·π 2▼ [σ2▼ (z)] =Xµ▼ [π 1▼ , π 2▼ ] · π 1▼ [σ1▼ (z)] · π 2▼ [σ2▼ (z)](π 1▼ ,π 2▼ )(11.4)
for z ∈ Z. Hence, at least conceptually, we can operate a change of variables in the optimization
problem, trading the distributions µ▲ , µ▼ in (11.1), for the vectors ω ▲ , ω ▼ , called realization vectors.
Doing so greatly reduces the number of variables in the optimization problem: the distributionsµ are over an exponential number of elements (the product of deterministic plans for each of theteam’s members), while ω ▲ , ω ▼ only require |Z| scalar variables—a linear amount in the gametree size.
In order to perform the change of variable effectively, we need to study the new domain of the
optimization problem. In particular, as we move away from µi ∈ ∆Πi1 ×Πi2 and allow the team to
specify a ω i instead (i ∈ {▲, ▼}), we need to characterize the domain of ω i that can be induced
from µi according to (11.4). The key observation here is that the mapping from µi to ω i is linear.
Hence, since the set of feasible µi is a convex polytope (the simplex ∆Πi1 ×Πi2 ), we can use thefollowing lemma to conclude that the set Ωi of all feasible ω i is itself a convex polytope. Hence, a
TMECor can be expressed as a bilinear saddle-point problem
max

min

ω ▲ ∈Ω▲ ω ▼ ∈Ω▼

Xz∈Zu▲ (z) · pc (z) · ω ▲ [z] · ω ▼ [z] =: max

min ω ⊤▲ U▲ ω ▼ ,(11.5)

ω ▲ ∈Ω▲ ω ▼ ∈Ω▼

where U▲ is an appropriate matrix that encodes the bilinear objective.
While the formulation (11.5) is reminescent of the bilinar formulation of Nash equilibrium in
two-player zero-sum games, several important differences exist. Most importantly, unfortunately,in general the convex polytopes Ω▲ and Ω▼ cannot be captured via a polynomial number of linear
constraints, as it is known that the computation of TMECor is computationally intractable inpractice (Chu and Halpern, 2001). This makes the formulation (11.5) conceptually similar to thatof EFCE (Chapters 9 and 10). In the next subsection we formalize this latter connection, and givenew positive complexity results for TMECor, as well as learning dynamics for teams inspired by
the technique pioneered by Chapter 10.

§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION

11.3.2

231

Connection with correlation plans and triangle-freenessFor each team i ∈ {▲, ▼}, consider the game tree Γi obtained by converting all nodes of the
opposing team into chance nodes. The polytope of correlation plans Ξi of Γi encode a superset of
the information encoded by realization plans. Indeed, it is immediate to see from the definitionof the polytope of correlation plans (Definitions 9.1 and 9.2) that the following holds.
Lemma 11.3. The polytope of valid realization vectors for each team i ∈ {▲, ▼} can beobtained by considering a subset of the dimensions of the polytope of correlation plans Ξi ofthe team, specifically
Ωi =

n


ξ[σi1 (z), σi2 (z)]z∈Zo: ξ ∈ Ξi .

(11.6)As a direct consequence, we have the following.
Corollary 11.1. When the information structure of the members of a team i ∈ {▲, ▼} istriangle-free (Section 9.4.2), then Ωi can be defined using only polynomially many constraints
in the size of the input game tree.
Hence, when both teams are triangle-free, or a triangle-free team plays against a single
opponent (for which the set of correlation plans is simply the sequence-form polytope), a
TMECor can be found in polynomial time.

As far as we are aware, this positive complexity result has not been noted before in theliterature. As an example application, we remark that in many cases the G Goofspiel, GL
limited-information Goofspiel, and B battleship games involve triangle freeness, and therefore a
TMECor in those instances can be computed in polynomial time.
More generally, the connection between Ωi and Ξi enables us to use the results discussed in
Section 9.5 to characterize an upper bound on the number of constraints required to define Ωi , andconsequently arrive at parameterized complexity results for the computation of TMECor.
Corollary 11.2. For any team i ∈ {▲, ▼}, the set Ωi of realization vectors is a linear
transformation of a set that can be defined via a chain of scaled extension operations
h1

h2

h3

hm

{1} ◁ X1 ◁ X2 ◁ · · · ◁ Xm ,(11.7)

where, for j = 1, . . . , m, Xj = ∆sj for some simplex dimension sj ∈ ℕ and hj is a linear
function.

232§11.4. EXPERIMENTAL EVALUATION

Remark 11.1. In fact, we remark that since Ωi only depends on a subset of Ξi , sharper bounds
Pm
can be given on the number of simplexes m used in (11.7) and the total dimension j=1 sj
that what could be done directly using the results known for Ξi alone. We will not discusssuch improvements in this dissertation, but refer the interested reader to the preprint by
B. H. Zhang, Farina, and Sandholm (2023).

Following the same approach as Chapter 10, by leveraging the structural decompositionof Ω▲ , Ω▼ via scaled extension operations (Corollary 11.2) via the framework of regret circuits
described in Chapter 4, we obtain no-external-regret learning dynamics. By using the folklore
connection between no-external-regret algorithms and solutions to bilinear saddle-point problems,
we can then use these learning dynamics to compute TMECors in games. We show in Section 11.4
that this approach indeed leads to the current practical state-of-the-art algorithm for computing
TMECor.

11.4

Experimental evaluation

In this section we experimentally evaluate our no-external-regret dynamics for solving the
TMECor bilinear saddle-point formulation (11.5). Following the approach laid out in Chapter 10,
we construct the dynamics by applying the regret circuit of Section 4.2.5 to the scaled-extensionbased decomposition of the sets (Corollary 11.2), employing both PRM+ and Discounted RM as
local no-external-regret minimizers for the probability simplexes that form the scaled-extensiondecomposition of Ξ▲ , Ξ▼ .
We compare our method against the prior state-of-the-art algorithms from three different setof techniques that have been used for this problem (see also Section 11.1):• The tree-decomposition-based LP solver proposed by B. H. Zhang and Sandholm (2022b)(henceforth “ZS22”). We used the original implementation of the authors, which internallyuses the barrier algorithm implemented by the commercial solver Gurobi. As recommended
by the authors, we turned Gurobi’s presolver off to avoid numerical instability and increase
speed. We allowed Gurobi to use up to four threads.
• The single-oracle algorithm of B. H. Zhang, Farina, Celli, and Sandholm (2022) (henceforth“ZFCS22”), which is an improved version of a prior single-oracle algorithm by Farina, Celli,Gatti, and Sandholm (2021). It represents currently the fastest single-oracle benchmark, andtherefore it fairly represents the state of the literature based on such technique. ZFCS22
iteratively refines the strategy of each team by solving best-response problems using a tightinteger program derived from the theory of extensive-form correlation (von Stengel and Forges,
2008; Farina, Kroer, and Sandholm, 2021b). We used the original code by the authors, which233§11. LEARNING DYNAMICS FOR TEAM COORDINATION AND COLLUSION

was implemented for three-player games in which a team of two players faces an opponent.
Like ZS22 and our LP-based solver, ZFCS22 uses the commercial solver Gurobi to solve linear
and integer linear programs. We allowed Gurobi to use up to four threads.
• The coordinator-based representation of Carminati, Cacciamani, Ciccone, and Gatti (2022)(henceforth, “CCCG22”), which computes a TMECor by converting the problem into computing a Nash equilibrium of a strategically-equivalent by exponentially-sized imperfectinformation extensive-form game, using a technique inspired by the work of Nayyar, Mahajan,
and Teneketzis (2013).
All experiments were run on a 64-core AMD EPYC 7282 processor. Each algorithm wasallocated a maximum of 4 threads, 60GBs of RAM, and a time limit of 6 hours. Experimental
results are summarized in Table 11.2.

Game {▼}

Decompos. of Ω▲

Decompos. of Ω▼

Simplexes Dimension Simplexes Dimension

K33
K34
K36
K38
K3c
K45
K45

ZS22
ZFCS22‡
ϵ = 10−3 ϵ = 10−4

37
49
73
97
—
4,621
121

36
48
72
96
—
15,415120

4,108
66,3497,002,763
488,157,721——
202,660,366

0.00s
0.00s
0.03s
4.73s
oom
0.03s
1.59s

0.00s
0.00s
0.12s
32.36s
oom
0.05s
6.34s

0.01s
0.02s
1.22s
3m 23soom
0.79s
3m 25s23,98349,005
139,964
417,027
150,707496,196855,397 3,486,091
32,75045,913

685
1,201
1,501
1,861
2,437

684
1,691,158
1,200
61,983,093
1,500—
1,860 1,973,610,366
2,436
538,111

0.02s
0.10s
0.18s
1.24s
0.03s

0.05s
0.48s
0.50s
4.94s
0.08s

0.50s
7.58s
9.30s
4m 24s0.27s

L3523 {3} 2,911,352 4,183,685
L4133 {3,4}79,351
158,058

220,705
75,157220,704
155,475

11.26s0.21s

24.86s0.92s

2m 26s
7.30s

> 6h
—

> 6h
—

1,522

1,521

—

0.11s

0.40s

2.10s

11.05s11.05s16,381
16,380486,442 1,155,144
218,570
389,995
46,23660,717
5,551
5,550

—
—
—
—
—

22.54s
2.31s
1.72s
3.80s
30.20s

1m 28s
4.70s
4.26s
11.09s
1m 11s

L3133
L3143
L3151
L3153
L3223

D33

{3}
487
918
{3}
2,100
6,711
{3}
54,255336,944{3} 1,783,926 15,564,765
{3}
—
—
{3,4}
26,566
124,875{4} 998,471 4,658,070

CCCG22 This chapter
Game size ϵ = 10−3 ϵ = 10−4

{3}
{3}
{3}
{3}
{3}

{3}

91,858215,967

D34{3} 4,043,377 13,749,608
D43 {2,4} 514,120 1,217,310
D62 {2,4,6} 254,758
457,795
D62 {4,6} 991,861 2,029,546
D62{6} 3,158,364 7,395,885

222,239,487
277,714,570

0.00s
0.01s
0.14s
0.23s
0.84s
—
—

0.00s
0.02s
0.14s
0.32s
1.39s
—
—

24.89s 45.96s
2m 4s
6m 3s
3.06s 13.98s
7m 23s 28m 13s
13.48s 18.53s

8m 29s 3h 19m 3h 19m
1m 32s
——16.55s——
1m 35s——
8m 53s——Table 11.2: Runtime of our learning algorithm (column ‘This chapter’), compared to prior state-of-theart algorithms based on linear programming (‘ZS22’, B. H. Zhang and Sandholm, 2022b) andcolumn generation (‘ZFCS22’, B. H. Zhang, Farina, Celli, and Sandholm, 2022) respectively, on
several standard parametric benchmark games. ‘—’: Missing or unknown value.

Column ‘Game‘ indicates the game, and the set of players on Team ▼. The games belong to234§11.4. EXPERIMENTAL EVALUATION

different classes of standard parametric benchmark games, each identified with an alphabeticalmnemonic whose starting letters and numbers identify the game class and number of players:
K3 K4 3- and 4-player Kuhn poker; L3 L4 3- and 4-player Leduc poker; D3 D4 D6 3-, 4-,
and 6-player liar’s dice. Depending on the game instance, we test with different time sizes, raging
from all-versus-one scenarios, to balanced splits. As usual, we refer the reader to Appendix A
for a description of each game class and their parameters. Columns ‘Decompos. of Ω▲ ’ and‘Decompos. of Ω▼ ’ report the total number of simplexes and dimension of the scaled-extension
decomposition of Ω▲ , Ω▼ (see Remark 11.1) for each game. Column ‘CCCG22’ indicates thenumber of histories in the equivalent game produced by Carminati, Cacciamani, Ciccone, andGatti (2022)’s algorithm. Column ‘This chapter’ reports the time to convergence of the best
variant of our learning dynamics to an average team exploitability of less than ϵ times the rangeof payoffs of the game. Column ‘ZS22’ reports the time it took ZS22 to compute an equilibrium
strategy for team ▲, to Gurobi’s default precision. Finally, column ‘ZFCS22’ reports the time it
took ZFCS22 to compute an equilibrium strategy for team ▲ with exploitability of less than ϵtimes the range of payoffs of the game. The missing values in that column are due to the fact thatthe implementation of ZFCS22 by the original authors only supported 3-player games.
Overall, our algorithms based on the team belief DAG are generally 2-3 orders of magnitude
faster than ZS22. In games with high parameters (e.g., K38 and K3c ), on the other hand, ZFCS22
is significantly more scalable, as it avoids the exponential dependence in the parameters at thecost of requiring the solution to integer programs, for which runtime guarantees are hard to
give. Compared to the converted game of Carminati, Cacciamani, Ciccone, and Gatti (2022), ourteam belief DAG is much smaller, often by orders of magnitude, which allows our algorithms tosimilarly be faster by orders of magnitude. Since Carminati, Cacciamani, Ciccone, and Gatti (2022)
do not give detailed timing results for their implementation for most of the games they tested, we
have not included a runtime comparison. However, they reported a runtime of approximately 3
minutes to achieve an exploitability of 0.021 in L3133 , whereas our algorithm took 0.02 seconds
to achieve a lower exploitability of 0.001—a difference of about four orders of magnitude.

Part IV

Beyond perfect rationality

Chapter 12

Positive complexity results fortrembling-hand perfect equilibria
The Nash equilibrium prescribes optimal strategies against perfectly rational opponents. However,
it is known that it has serious shortcomings when used to prescribe strategies to be deployedagainst imperfect opponents who may make mistakes. This is problematic when operationalizing
equilibria in the real world among imperfect players. In this chapter we will provide newpositive complexity results for the computation of a refinement of the Nash equilibrium—called an
extensive-form perfect equilibrium (EFPE)—that guarantees rational strategies even upon mistakesof the opponent.
Undesirable behavior can be prescribed by the Nash equilibrium already in the simple andwell-understood case of two-player zero-sum games, as we illustrate with the following example.
Example 12.1. As an example, consider the perfect-information game in Figure 12.1.
A
a

b

B
(1, −1)l(0, 0)

r(−5, 5)Figure 12.1: Perfect-information game in which a sequentially-irrational Nash equilibrium ishighlighted.

The bold lines show one of the Nash equilibria of the game. It does not matter whether
the white player acting at B chooses action l or action r because he never gets to action if

238§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

the black player acting at A plays rationally. So, in Nash equilibrium, the white player canchoose action l. However, if the black player makes a mistake and chooses action b, it would
be better for the white player to choose action r (thus getting a payoff of 5 instead of 0). So, inthat part of the game where the black player has made the mistake, the white player’s Nashequilibrium strategy is not rational. In game-theoretic terms, it is not sequentially rational.

While in the particular example of Figure 12.1 the issue could be resolved by using anequilibrium refinement called subgame perfect Nash equilibrium, that solution concept is notparticularly useful in imperfect-information extensive-form games, where few subgames (nodesof the game tree that are alone in their information set) typically exist.
Example 12.2. For example, consider the example in Figure 12.2.

aLegendAcbB(1, −1)lr(0, 0) (−5, 5)lPlayer 1

Terminal history

Player 2

Y Information set

r(−1, 1) (−5, 5)

Figure 12.2: Small imperfect-information game in which a sequentially-irrational Nash equilibrium
is highlighted.

The white player acting at information set B does not have any subgame, and therefore thehighlighted sequentially-irrational Nash equilibrium is subgame perfect.

Another refinement of Nash equilibrium is undominated Nash equilibrium (UNE), that is, Nashequilibrium where the pure strategies in the support of the equilibrium do not include strongly
dominated strategies. UNE would remove the unreasonable Nash equilibria in the games of bothFigures 12.1 and 12.2, but there are other games where UNEs can be sequentially irrational, andin general undomination and sequential rationality are incomparable in the sense that neither
implies the other (Miltersen and Sørensen, 2006). We included an example of a situation where an
undominated equilibrium is sequentially irrational in an appendix to this chapter, Section 12.A.
Trembling-hand refinements are the standard solution proposed in the literature to avoid theissue of sequential irrationality. One of these, extensive-form perfect equilibrium (EFPE) will then be
the focus of this chapter. We review trembling-hand refinements, and in particular EFPE, in thenext section.

§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

12.1

239

Preliminaries on trembling-hand refinements

The issue of sequential irrationality stems from the fact that some parts of the game tree are
unreachable at equilibrium. For those excluded parts of the game tree, any strategy can bepicked without affecting the equilibrium. The idea behind trembling-hand refinements can beconceptualized easily. To avoid sequential irrationality, trembling-hand refinements force all
players to explore the whole game tree. They do so by requiring the players to tremble, that is, play
all actions at all decision points with some strictly positive lower bound probability controlled
by a hyperparameter ϵ > 0. A trembling-hand refinement is then any limit point of such Nashequilibria as ϵ → 0+ .
Different equilibrium notions differ as to how the lower bounds are set as a function of ϵ. In
this chapter we are interested in extensive-form perfect equilibria (EFPE), due to Selten (1975),
which define perhaps the conceptually simplest trembling scheme. Specifically, in an EFPE, thetrembles are behavioral: given ϵ > 0, the perturbed game simply mandates that every action at
every decision point must be picked with probability at least ϵ.
Definition 12.1 (Extensive-form perfect equilibrium, EFPE). Given a game Γ, an EFPE isa limit point, as ϵ → 0+ , of Nash equilibria of the perturbed Γ(ϵ) in which each player isconstrained to play only strategies that put probability at least ϵ on every action.

It is well-known (e.g., Kreps and Wilson, 1982) that every game has at least one EFPE, and that
EFPEs are sequentially rational.
Other notions of trembling-hand refinements exist, most notably the quasi-perfect equilibrium(QPE) due to van Damme (1984). The definition of QPE is significantly more complicated, andwill be given in Definition 13.5 (in Chapter 13), together with its one-sided counterpart called
one-sided QPE (Farina and Sandholm, 2021a).

12.2

Contributions and related results

The main contribution of this chapter will be to establish the following result:
Finding an EFPE in a two-player game is not harder than finding a Nash equilibrium.
Specifically, in zero-sum games, an EFPE can be found in polynomial time in the size of the
input game, and in general-sum games it can be computed using a path-following algorithmwith polynomial-time steps.
The result closes a problem left explicitly open by Miltersen and Sørensen (2006), andcomplements a similar result that was known for the quasi-perfect equilibrium (see Table 12.1).

240§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE

Solution concept

Two-player general-sum games

Nash equilibrium (NE)(Nash, 1950)

Two-player zero-sum games

PPAD-complete

FP(Daskalakis, Goldberg, and Papadimitriou, 2009)(Romanovskii, 1962)(von Stengel, 1996)

Quasi-perfect equilibrium (QPE)(van Damme, 1984)

PPAD-complete

FP(Miltersen and Sørensen, 2010)(Miltersen and Sørensen, 2010)

Extensive-form perfect eq. (EFPE)(Selten, 1975)

PPAD-completeFP(this chapter)(this chapter)

Table 12.1: Complexity of computing trembling-hand equilibrium refinements in two-player games.

Building on some of the ideas and techniques introduced in this chapter, in Chapter 13 wewill propose the first scalable algorithm for computing exact QPE, EFPE, and other refinements of
the Nash equilibrium in two-player zero-sum games.

12.3

Positive complexity results for two-player EFPEIn two-player games, a Nash equilibrium can be expressed as a linear complementarity problem(LCP). In this section, we show that the same holds even when each strategy is required to selectevery action with probability at least ϵ as required to compute the EFPE defined in Definition 12.1.

12.3.1

Behavioral perturbation matrices

The constraint that each action in a strategy xi ∈ Qi for Player i ∈ {1, 2} must be selected withprobability is at least ϵ is captured in the sequence-form representation by requiring thatxi [ja] ⩾ ϵ · xi [pj ]∀ ja ∈ Σ∗i .

By including the dummy constraint xi [∅] ⩾ 0 so as to make the matrix square, we can collect
the linear constraints above in matrix form as Ri (ϵ) xi ⩾ 0, where the matrix Ri (ϵ) ∈ ℝΣi ×Σi isdefined byRi (ϵ)[σ, σ ′ ] :=




1

if σ = σ ′otherwise,−ϵ if σ = ja and σ ′ = pj
0

∀ σ, σ ′ ∈ Σi .

We call Ri (ϵ) the behavioral perturbation matrix of Player i. We illustrate these matrices with an
example.

§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

241

2

C

3

4

B

5

6

 ∅

1 0 0 0 0 ∅ −ϵ 1 0 0 0  A1
R1 (ϵ) =  −ϵ 0 1 0 0  A2

 0 0 −ϵ 1 0  B5
0 0 −ϵ 0 1 B6

C3
C4

1

B5
B6

A

A2

A1

Example 12.3. Consider the small game below. The matrices R1 (ϵ) and R2 (ϵ) in this case areas follows: ∅
1 0 0 ∅

R2 (ϵ) =  −ϵ 1 0  C3.
−ϵ 0 1 C4Assuming that the rows and columns of behavioral perturbation matrices are ordered inaccordance with the partial order ≼ of sequences, we remark that behavioral perturbation matrices
are lower triangular square matrices having only 0 or −ϵ as entries. We now show that they arealways invertible and that their inverse is a matrix of polynomials in ϵ.
Lemma 12.1. Let Ri (ϵ) be a n × n behavioral perturbation matrix. Then Ri (ϵ) is invertible.
Furthermore, assuming that the rows and columns of behavioral perturbation matrices areordered in accordance with the partial order ≼ of sequences, its inverse is
Ri (ϵ)−1 = I + ϵE(ϵ),
where I is the identity matrix, and E(ϵ) is a lower triangular matrix whose entries arepolynomials in ϵ having non-negative integer coefficients.

Proof. By induction on the submatrix formed by the first k rows and k columns. When k = 1,
the statement is trivial. Now, suppose the theorem holds for the k × k top-left submatrix of
Ri (ϵ), denoted Rik (ϵ); we will show that it holds for the (k + 1) × (k + 1) top-left submatrix
mRik+1 (ϵ) of Ri (ϵ). By definition of Ri (ϵ), we can write
Rik+1 (ϵ) = Rik (ϵ)

0

⊤

1

b(ϵ)


where b(ϵ)⊤ = (0, . . . , 0, −ϵ, 0, . . . , 0).

,It is immediate to check by direct inspection that the matrix

Rik+1 (ϵ)−1 = 

Rik (ϵ)−1−b(ϵ)⊤

Rik (ϵ)−1

0
1




242

§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE

is the inverse of Rik+1 (ϵ). Using the inductive hypothesis, we have Rik (ϵ)−1 = I + ϵEk (ϵ), andtherefore
Ek (ϵ)
0.
Rik+1 (ϵ)−1 = I + ϵ(−b(ϵ)/ϵ)⊤ Rik (ϵ)−1 0
Since (−b(ϵ)/ϵ)⊤ = (0, . . . , 0, 1, 0 . . . , 0) is a non-negative real vector, the row vector b′ (ϵ) :=(−b(ϵ)/ϵ)⊤ Rik (ϵ)−1 has entries that are polynomials with non-negative integer coefficients, andsoEk (ϵ) 0 =: I + ϵEk+1 (ϵ),
Rik+1 (ϵ)−1 = I + ϵ′b (ϵ) 0where Ek+1 (ϵ) is a lower triangular matrix whose entries are polynomials in ϵ having nonnegative integer coefficients, as we wanted to show.

A1
A2
B5
B6

Example 12.4. For the matrix R1 (ϵ) of Example 12.3, we have:∅
1 0 0 0 0 ∅ ϵ 1 0 0 0  A1−1
R1 (ϵ) =  ϵ 0 1 0 0  A2 .
 2

 ϵ 0 ϵ 1 0  B5
ϵ2 0 ϵ 0 1 B6

12.3.2

EFPE as a trembling linear complementarity problem (LCP)

In this subsection, we show that the problem of finding a Nash equilibrium of a two-player gameΓ(ϵ) subject to the constraint that each player pick each action with lower bound probability ϵ canbe expressed as a linear complementarity program whose entries are polynomials in ϵ. In what
follows, we let Fi ∈ ℝJi ×Σi and f i ∈ ℝJi define the sequence-form polytopeiQi = {x ∈ ℝΣ
⩾0 : Fi x = f i , x ⩾ 0}of Player i ∈ {1, 2}, and Ui define the payoff matrix of Player i in sequence form.
Lemma 12.2. An EFPE is a limit point as ϵ → 0+ of any solution of the perturbed standardform LCP§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA


find



 s.t.
P (ϵ) :





243

z, w
z⊤w = 0w = M(ϵ)z + b
z, w ⩾ 0

where, using the shorthand R1 ≡ R1 (ϵ), R2 ≡ R2 (ϵ) for brevity,

x̃1
 
 x̃2  +
v 1 
z=
v − , 1
 +
v 2 
v−
2



0


 0 


 f1 ,b=−f 1 
 f2 
−f 2



−⊤ ⊤
0
−R1−⊤ U1 R2−1 R1−⊤ F⊤
F1
0
0
1 −R1

−⊤
−1
−⊤ ⊤ 
0
0
0R2−⊤ F⊤F2 
−R2 U⊤
2 R1
2 −R2


 −F1 R1−1

0
0
0
0
0
.
M(ϵ) = −1F1 R 1
0
0
0
0
0






0
−F2 R2−100000
F2 R2−10000Proof. The problem of finding the best response of Player i in the perturbed game Γ(ϵ) where
every action is selected with probability at least ϵ is a linear problem maxxi
BRi (ϵ) :
s.t.



x⊤
i Ui x−i
Fi x i = f i
Ri (ϵ)xi ⩾ 0

Notice that Ri (ϵ) is invertible (Lemma 12.1), hence by changing variable x̃i := Ri (ϵ)xi , we findthe equivalent problem: maxx̃i
BRi (ϵ) :s.t. 1
2Taking the dual:−⊤x̃⊤Ui R−i (ϵ)−1 x̃−i
i Ri (ϵ)
Fi Ri (ϵ)−1 x̃i = f ix̃i ⩾ 0.

244

§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE
 minviBRi (ϵ) :s.t. 3


4

f⊤
i vi
−⊤
Ri (ϵ)−⊤ F⊤
Ui R−i (ϵ)−1 x̃−i
i v i ⩾ Ri (ϵ)
v i free in sign.

Finally, complementarity slackness requires that
5

−⊤ ⊤
x̃⊤
Fi v i − Ri (ϵ)−⊤ Ui R−i (ϵ)−1 x̃−i ) = 0.
i (Ri (ϵ)

Solving problem BRi (ϵ) or BRi (ϵ) is equivalent to solving the feasibility problem defined byconstraints 1 to 5 . It is now easy to see that we can cast the problem of satisfying conditions
1 to 5 for both players as a standard-form LCP whose parameters are as defined in thislemma.
In particular, we show that a polynomial (in the game size) number of bits is sufficient torepresent the polynomial entries of the LCP introduced in Lemma 12.2.
Lemma 12.3. Consider the LCP formulation of Lemma 12.2, where ϵ is treated as a symbolic
variable, so that the entries of M(ϵ) are polynomials in ϵ. A number of bits polynomial in the
input game size is sufficient to store all coefficients appearing in P (ϵ).
Proof. Consider the LCP formulation of Lemma 12.2. We begin by showing that each coefficient
appearing in P (ϵ) requires a polynomial amount of memory to be store. This property triviallyholds for vector b. On the other hand, all numbers appearing in matrix M(ϵ) are either
zeros, or they are obtained by multiplying two or more of the following matrices together:⊤⊤R1 (ϵ)−⊤ , R2 (ϵ)−⊤ , U1 , U⊤
2 , F1 , F2 . Hence, as long as each of the coefficients appearing in
the above-mentioned matrices requires a polynomial number of bits in the input game size,⊤
the property is true. This is clearly true for U1 , U⊤
2 , F1 , F2 , so we are left with the task of−1−1
proving this property for R1 (ϵ) and R2 (ϵ) . However, since det R1 (ϵ) = 1 (indeed, noticethat R1 (ϵ) is lower triangular), using the adjoint matrix theorem and the Leibniz formula forthe determinant, we conclude that each entry in R1 (ϵ)−1 is obtained as a sum of |Σ1 |! terms,
each of which is a product of |Σ1 | entries of R1 (ϵ), since |Σ1 | is the dimension of R1 (ϵ) (thesame holds for R2 (ϵ)). Therefore, the property holds, showing that each coefficient in M(ϵ)
and b requires a polynomial amount of memory to be stored.
We now show that the maximum degree appearing in M(ϵ) is 2|Σ1 |. This is a consequenceof the observation above: since each entry in R1 (ϵ)−1 is obtained as sum of |Σ1 |! terms, eachof which is a product of |Σ1 | entries of R1 (ϵ), the maximum degree appearing in R1 (ϵ)−1 is|Σ1 |. Now, since each element of M(ϵ) is obtained from the product of at most two matrices
dependent on ϵ, the maximum degree appearing in M(ϵ) (and therefore in P (ϵ)) is 2|Σ1 |.
Thus, we have a polynomial amount of coefficients to store, each of which requires a§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

245

polynomial amount of memory. The required space is therefore polynomial.
Finally, we show that for ϵ sufficiently small, the LCP of Lemma 12.2 is feasible.

12.3.3

Existence of a negligible positive perturbation (NPP)

Before turning our attention to the computational aspects of finding an EFPE, we introduce some
general concepts, pertaining to perturbed linear optimization problems. While we target thedevelopment of these concepts with our specific use-case in mind, it should be noted that thissection’s definitions and lemmas are of broader interest, being applicable to any linear program(LP) or LCP. We recall that a basis B for a standard-form LP with constraints Mx = b or astandard-form LCP with linear equality constraints w = Mz + b is a set of linearly independent
columns of M such that the associated solution (called basic solution) is feasible.
Definition 12.2 (Negligible positive perturbation, NPP). Let P (ϵ) be an LCP dependent onsome perturbation ϵ. The value ϵ∗ > 0 is a negligible positive perturbation (NPP) if any optimalbasis B for P (ϵ∗ ) is optimal for P (ϵ), for all ϵ ∈ (0, ϵ∗ ].

Definition 12.3 (Optimality certificate for a basis). Given an LCP P (ϵ), and a basis B for it, wecall the finite-dimensional column vector γ B (ϵ) an optimality certificate for B if for all ϵ > 0
γ B (ϵ) ⩾ 0 ⇐⇒ B is optimal for P (ϵ).

Lemma 12.4. In the case of a perturbed LCP in standard form

find



 s.t. 1
P (ϵ) :

2



3

z, w
z⊤w = 0w = M(ϵ)z + b(ϵ)
z, w ⩾ 0

an optimality certificate for the complementary basis B isγ B (ϵ) = B(ϵ)−1 b(ϵ)
where B is the basis matrix corresponding to B.

246§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE

Proof. Since the basis B is complementary by hypothesis, constraint 1 is always satisfied.
Constraint 2 is satisfied by the definition of B(ϵ). Constraint 3 is satisfied if and only if
B(ϵ)−1 b(ϵ) = γ B (ϵ) ⩾ 0.
Finally, before proceeding, we introduce three lemmas that come in handy when dealing with
optimality certificates. Indeed, it is often the case that γ B (ϵ) has polynomial or rational functions(with respect to ϵ) as entries.
Lemma 12.5. Let p(ϵ) := a0 + a1 ϵ1 + · · · + an ϵn be a real polynomial such that a0 ̸= 0, and letµ := maxi |ai |. Then p(ϵ) has the same sign of a0 for all 0 ⩽ ϵ ⩽ ϵ∗ , whereϵ∗ := |a0 |/(µ + |a0 |).

Proof. We prove that when a0 > 0, p(ϵ) is positive for all 0 ⩽ ϵ ⩽ ϵ∗ . Indeed,p(ϵ) = a0 + a1 ϵ1 + · · · + an ϵn > a0 − µϵ∞Xϵi = a0 −

i=0

µϵ
.
1−ϵ

Since ϵ ⩽ ϵ∗ := a0 /(µ + a0 ) we havep(ϵ) > a0 −

µa0
= 0.
µ + a0 − a0

The case a0 < 0 is symmetric, and in that case p(ϵ) is negative for all 0 ⩽ ϵ ⩽ ϵ∗ .
As a corollary, we extend the result of Lemma 12.5 to rational functions.
Lemma 12.6. Letp(ϵ) :=

a0 + a1 ϵ1 + · · · + an ϵnb0 + b1 ϵ1 + · · · + bm ϵm

be a rational function such that a0 , b0 ̸= 0, and let µa := maxi |ai |, µb := maxi |bi |. Then p(ϵ)
has the same sign of a0 /b0 for all 0 ⩽ ϵ ⩽ ϵ∗ , where

|a0 |
|b0 |
ϵ := min,.
µ + |a0 | µ + |b0 |∗§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

247

Proof. The proof follows immediately by applying Lemma 12.5 to the numerator and thedenominator of p(ϵ).

Lemma 12.7. Letp(ϵ) :=

a0 + a1 ϵ1 + · · · + an ϵnb0 + b1 ϵ1 + · · · + bm ϵm

be a rational function with integer coefficients, where the denominator is not identically zero;let µa := maxi |ai |, µb := maxi |bi |, µ := max{µa , µb } and ϵ∗ := 1/(2µ). Then exactly one of
the following holds:• p(ϵ∗ ) = 0 for all 0 < ϵ ⩽ ϵ∗ ,• p(ϵ∗ ) > 0 for all 0 < ϵ ⩽ ϵ∗ ,• p(ϵ∗ ) < 0 for all 0 < ϵ ⩽ ϵ∗ .

Proof. If the numerator of p(ϵ) is identically zero, the thesis follows trivially, as p(ϵ) = 0 for allϵ, while the denominator is never zero for all 0 < ϵ ⩽ ϵ∗ due to Lemma 12.5. If, on the other
hand, the numerator of p(ϵ) is not identically zero, there exist j and x, both non-negative, suchthat p can be written asp(ϵ) =ϵq (aq + aq+1 ϵ + · · · + an ϵn−q ),ϵr (br + br+1 ϵ1 + · · · + bn ϵn−r )with aq , br ̸= 0. Since aq and br are integer, |aq |, |br | ⩾ 1 and we have
ϵ∗ =



1
|aq |
|br |
⩽ min
,
.
2µ
µa + |aq | µb + |br |Using Lemma 12.6 we conclude that the sign of p(ϵ) is constant, and equal to that of aq /br , for
all 0 < ϵ ⩽ ϵ∗ .

Theorem 12.1. Given a (general-sum) two-player game Γ with ν := maxi∈{1,2}, j∈Ji |Aj |, theproblem P (ϵ) of computing any EFPE for Γ admits an NPP ϵ∗ ⩽ 1/ν that can be computedfrom Γ in polynomial time. In particular, ϵ∗ = 1/V ∗ , where the integer value V ∗ can be
represented in memory with a number of bits polynomial in the input game size.

248§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE

Proof. We illustrate the steps that lead to the determination of such V ∗ . The central idea is asfollows: we want to determine ϵ∗ so that, whatever the feasible base B for P (ϵ∗ ) may be, theoptimality certificate for ϵ∗ is positive for all ϵ ∈ (0, ϵ∗ ]. Indeed, it is immediate to see that suchϵ∗ is necessarily an NPP.
• Optimality certificate. We begin by studying the optimality certificate for the LCP P (ϵ),
that is, by Lemma 12.4,
B(ϵ)−1 b(ϵ) ⩾ 0,
where B(ϵ) is base matrix corresponding to the feasible base B found by Lemke’s algorithm.
Introducing C(ϵ) = cof B(ϵ), the cofactor matrix of matrix B(ϵ), and leveraging the wellknown identity B(ϵ)−1 = C(ϵ)⊤ / det B(ϵ), we can rewrite the optimality certificate aboveasC(ϵ)⊤ b(ϵ)⩾ 0.
det B(ϵ)
The vectorial condition above is equivalent to a system of m scalar conditions, each of theform
gk (ϵ) =ck (ϵ)⊤ b(ϵ)⩾ 0,det B(ϵ)

where ck (ϵ) is the k-th row of C(ϵ)⊤ . Evidently, gk (ϵ) is a rational function in ϵ, having
only integer coefficients, for all k = 1, . . . , m.
• Denominator coefficients. We now give an upper bound on the coefficients of thedenominator of gk (ϵ), that is det B(ϵ). Let VB be the largest coefficient that could
potentially appear in B(ϵ) and b(ϵ), and let d be the largest polynomial degree appearingin B(ϵ). Notice that d ∈ O(poly(m)). By using Hadamard’s inequality, we can write
coeff(det B(ϵ)) ⩽ mm/2 VBm coeff((1 + ϵ + · · · + ϵd )m ),where coeff(·) is the largest coefficient of its polynomial argument. Since coeff((1 + ϵ +· · · + ϵd )m ) ⩽ dm , we havecoeff(det B(ϵ)) ⩽ VD := mm/2 (dVB )m .
Notice that this bound is valid for all possible basis matrices B(ϵ). Furthermore, noticethat§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

249

log VD = m/2 log m + m log d + m log VB ,and by Lemma 12.3 we conclude that VD requires a number of bits polynomial in the
input game size in order to be stored in memory.
• Numerator coefficients. Since the elements of ci (ϵ) are cofactors for B(ϵ), they are
upper-bounded by det B(ϵ), which in turn is upper-bounded by VD . Therefore,
coeff(ci (ϵ)⊤ b(ϵ)) ⩽ VN := VB VD .
Again, it is worthwhile to notice that this bound is valid for all possible basis matrices
B(ϵ).
• Wrapping up. Define V ∗ = 2 max{VN , VD } = 2VB VD . We now argue that ϵ∗ = 1/V ∗ isan NPP for P (ϵ). Indeed, let B ∗ be a feasible basis (since ϵ∗ ⩽ 1/m, B always exists—see
Lemma 12.9) for P (ϵ∗ ), and let B(ϵ∗ ) be the corresponding base matrix. Being B feasible
for P (ϵ∗ ), each row gk in the optimality certificate is non-negative when evaluated at ϵ∗ forall k. Therefore, we know from Lemma 12.7 that gk (ϵ) ⩾ 0 for all ϵ ∈ (0, 1/V ∗ ] = (0, ϵ∗ ].
Hence, the optimality certificate for B is non-negative for all 0 < ϵ ⩽ ϵ∗ , which isequivalent to say that ϵ∗ is an NPP. Finally, note that V ∗ = 2VB VD be stored in memory
with a number of bits polynomial in the game size. This completes the proof.

12.3.4Computation of extensive-form perfect equilibria

We finally delve into the computational details of finding extensive-form perfect equilibria. The
central result of this section (Theorem 12.1) roughly states that the EFPE LCP (Lemma 12.2) always
admits a “small” NPP. Leveraging this fact, we quickly derive a path-following algorithm for thecomputation of EFPE in general-sum games in which each pivoting step has a polynomial-time cost
(Theorem 12.2), and a polynomial-time algorithm for the zero-sum counterpart (Theorem 12.3).
These two algorithms put the two search problems in PPAD and FP classes, respectively.
To establish the result, we will make use of Lemke’s algorithm (Lemke, 1970), a classic iterativealgorithm that is able to solve a linear complementarity problem, provided it satisfies the following
conditions.
Lemma 12.8 (Theorem 4.1, Koller, Megiddo, and von Stengel, 1996). Consider a linear
complementarity problem in standard form

250

§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE


find



 s.t. 1

2



3

z, w
z⊤w = 0w = Mz + b
z, w ⩾ 0.

If the two conditions(a) z ⊤ Mz ⩾ 0 for all z ⩾ 0, and(b) z ⩾ 0, Mz ⩾ 0, z ⊤ Mz = 0 =⇒ z ⊤ b ⩾ 0,
are satisfied, then Lemke’s algorithm computes a solution of the LCP.
We start by observing that, as long as the perturbation ϵ is “reasonably small”, the LCP definedin Lemma 12.2 always admits a solution.
Lemma 12.9. If 0 < ϵ ⩽ 1/ν, where ν := maxi∈{1,2}, j∈Ji |Aj | is the maximum number ofactions available at any information set of the game. Lemke’s algorithm always finds asolution for P (ϵ).
Proof. We follow the same proof structure as that in Koller, Megiddo, and von Stengel, 1996,
Section 4. In particular, we prove that if U1 , U2 < 0, then conditions (a) and (b) of Lemma 12.8
hold for all problems P (ϵ) defined in Lemma 12.2. Notice that we can always assume U1 , U2 < 0without loss of generality, as we can apply an offset to the payoff matrices leaving the gameunaltered.
• Condition (a). We need to show that when U1 , U2 < 0, then z ⊤ M(ϵ)z ⩾ 0 for all z ⩾ 0.
We have:−⊤z ⊤ M(ϵ)z = x̃⊤(−U1 − U2 )R2 (ϵ)−1 x̃2 .
1 R1 (ϵ)

Substituting U = −U1 − U2 > 0 and using Lemma 12.1:⊤
z ⊤ M(ϵ)z = x̃⊤1 (I + ϵE1 (ϵ)) U(I + ϵE2 (ϵ))x̃2⩾ x̃⊤
1 Ux̃2 .

(12.1)

When z ⩾ 0, then x̃1 , x̃2 ⩾ 0 and we conclude that x̃⊤
1 Ux̃2 ⩾ 0, which implies the thesis.
• Condition (b). We already proved (Equation (12.1)) thatz ⊤ M(ϵ)z ⩾ x̃⊤
1 Ux̃2 ,§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

251

where U > 0. In order for z ⊤ M(ϵ)z to be zero given z ⩾ 0, it is necessary that x̃1 , x̃2 = 0.
−−+
Defining v 1 = v +1 − v 1 and v 2 = v 2 − v 2 , we have
R1 (ϵ)−⊤ F⊤
1 v1
R2 (ϵ)−⊤ F⊤
2 v2 0⊤, z ⊤ b = b⊤ z = f ⊤M(ϵ)z = 1 v1 + f 2 v2 .


0






0
0
Hence, in order to complete the proof, it suffices to show that⊤Ri (ϵ)−⊤ F⊤i v i ⩾ 0 =⇒ f i v i ⩾ 0(i ∈ {1, 2}).

To this end, we consider the following linear optimization problem Yi (ϵ), and its dualȲi (ϵ):Yi (ϵ)Ȳi (ϵ):: maxx̃i
s.t.



0
Fi Ri (ϵ)−1 x̃i = f i ,x̃i ⩾ 0

(

f⊤
i vi
.
Ri (ϵ)−⊤ F⊤
i vi ⩾ 0

minvi
s.t.

Notice that Yi (ϵ) is feasible since ϵ ⩽ 1/ν by hypothesis (indeed, the strategy that picks
every action uniformly at random satisfies that each action is select with probability⩾ 1/ν). By the strong duality theorem, we conclude that whenever the constraint of
the dual problem is satisfied, the objective value is non-negative, that is Ri (ϵ)−⊤ F⊤
i vi ⩾⊤
0 =⇒ f i v i ⩾ 0 as we aimed to show.
We remark that when ϵ is a given value, the task of finding an NE of P (ϵ) has a powerfulinterpretation. Indeed, it captures the situation in which the actions of a player are subject toexecution uncertainty and therefore a player cannot perfectly control their actions.
Theorem 12.2. The problem of computing an EFPE of a general-sum two-player game Γ is
PPAD-complete.
Proof. Let ϵ∗ = 1/V ∗ be an NPP as defined in Theorem 12.1, and let B be a feasible base for the(numerical) problem P (ϵ∗ ), found using Lemke’s algorithm. Since ϵ∗ is an NPP, the pair ofstrategies (x∗1 , x∗2 ) corresponding to B retain their feasibility with respect to the LCP P (ϵ) as

252§12.3. POSITIVE COMPLEXITY RESULTS FOR TWO-PLAYER EFPE

ϵ → 0+ , meaning that (x∗1 , x∗2 ) is in fact an EFPE.
Furthermore, given that V ∗ requires a number of bits polynomial in the input game size,
each iteration of Lemke’s algorithm takes time polynomial in the game size. This proves thatthe algorithm described is a path-following algorithm requiring a polynomial-time cost at each
step, and therefore the problem of finding an EFPE in two-player games is in the PPAD class.
The hardness easily follows from the fact that EFPE is a refinement of Nash equilibrium, an
EFPE always exists, and finding a Nash is PPAD-complete. Therefore, if finding an EFPE were
not PPAD-hard, then one could use the EFPE-finding algorithm with the aim of finding an NE
and therefore not even finding an NE would be PPAD-hard. This concludes the proof.
We remark that the proof of theorem above also applies for an arbitrary ϵ (potentially non-NPP),
showing that finding an NE for any ϵ < maxi∈{1,2}, j∈Ji |Aj | is in the PPAD class. We summarize
the procedure to find an EFPE of general-sum games in Algorithm 12.1.
Algorithm 12.1: Find-EFPE
Compute ϵ∗ from Γ as in the proof of Theorem 12.1∗
2 Determine a basis B for the numerical LCP P (ϵ )3 Let B(ϵ) be the base matrix corresponding to B in P (ϵ), as ϵ varies
1[▷ Since B(ϵ)−1 b is a rational bounded function in a neighborhood of 0, B(0)−1 b exists]
−
− ⊤
+
+
−1
4 (x̃1 , x̃2 , v 1 , v 1 , v 2 , v 2 ) = B(0)b5[▷ Note that R1 (0) = R2 (0) = I, so x̃1 = x1 , x̃2 = x2 ]return the pair of strategies (x̃1 , x̃2 )12.3.5Polynomial-time computation in zero-sum games

We now show that Algorithm 12.1 requires polynomial time when the game is zero sum.
Theorem 12.3. The problem of computing an EFPE of a zero-sum two-player game Γ can besolved in polynomial time in the size of the input game.

Proof. Like in Theorem 12.2, we can easily extract an EFPE by looking at the feasible matrix B
which solves the (numerical) LCP P (ϵ∗ ). However, in the zero-sum setting, we do not need touse Lemke’s algorithm. Indeed, notice that in zero-sum two-player games, matrix M(ϵ) as
defined in Lemma 12.2 is such that M(ϵ) + M(ϵ)⊤ = 0 for all ϵ, because U2 = −U⊤
1 . Therefore,
the complementarity condition can be rewritten as
0 = z ⊤ (M(ϵ)z + b(ϵ)) =
1 ⊤z M(ϵ) + M(ϵ)⊤ z + z ⊤ b(ϵ) = z ⊤ b(ϵ),
2§12. POSITIVE COMPLEXITY RESULTS FOR TREMBLING-HAND PERFECT EQUILIBRIA

253

a linear condition instead of a quadratic one. This shows that when the game is zero-sum,the LCP is actually an LP. As such, a basis for the LCP of Algorithm 12.1 can be computedin polynomial time, leading to an overall polynomial time algorithm. This completes theproof.
We remark that the technique used to establish the above result yields a polynomial-time
method for computing EFPE. However, the method scales unfavorably in practice due to the needto operate with extremely small rational numbers. A significantly more scalable approach will bepresented in Chapter 13.

12.A Appendix: Undomination does not prevent sequential
irrationality
One might believe that the problem of sequential irrationality is that of picking dominatedstrategies. So, one might be inclined to look into the problem of finding a Nash equilibrium
whose support does not include any (weakly) dominated strategy (the concept is not immediately
well defined, but for the purposes of this discussion let’s restrict ourselves to Nash equilibria in
deterministic strategies).
Example 12.5. Figure 12.3 shows a modification of the classic Guess-the-Ace game introducedby Miltersen and Sørensen (2006).

(A♠ on top)

1/52

51/52

(A♠ not on top)

A
quit

bet

bet

quit

B

(0, 0)¬A♠C¬gift gift(0, 0)A♠(−$, $)(−$, $)¬A♠(−$, $)(0, 0)

A♠

D¬gift gift(0, 0)(−$, $)

Figure 12.3: Modified Guess-the-Ace games. The highlighted equilibrium is undominated, and yet
sequentially irrational.

254§12.A. APPENDIX: UNDOMINATION DOES NOT PREVENT SEQUENTIAL IRRATIONALITY

In regular Guess-the-Ace, at the start a standard 52-card deck is perfectly shuffled, face
down, by a dealer. Then, Player 1 can decide whether to immediately end the game, atwhich point no money is transferred between the players, or offer $1000 to Player 2 if they
can correctly guess whether the top card of the shuffled deck is the ace of spaces or not. If
Player 2 guesses correctly, the $1000 get transferred from Player 1 to Player 2; if not, no money
is transferred.
In Figure 12.3, again due to Miltersen and Sørensen (2006), the Guess-the-Ace game isslightly modified in that, when Player 2 guesses wrong, Player 1 can decide whether they
still want to give $1000 to Player 2 out of the kindness of their heart or not. By introducingthat possibility, action ‘¬A♠’ is not strictly dominating anymore, because Player 2 might still
hope that the second gift of $1000 is given only when the insensible guess ‘A♠’ is made.
The example above implies the following general principle.
Observation 12.1. Undomination does not prevent a player from playing risky actions,“hoping” for an opponent’s mistake.

Chapter 13

Computing exact trembling-handrefinements in two-player zero-sum
games at scale
In this chapter we develop the first technique for computing exact trembling-hand equilibriumrefinements—including extensive-form perfect (Chapter 12) and quasi-perfect equilibria—in large
two-player zero-sum games. For the first time, it enables computing exact, refined, sequentiallyrational Nash equilibrium strategies in games with up to hundreds of millions of terminal states,
4-5 orders of magnitude larger than what prior techniques could handle.

13.1

Related work

An alternative algorithm to compute an exact quasi-perfect equilibrium (QPE) is a simplex
algorithm variant that deals symbolically with the perturbation using the lexico-minimum ratio
test (Miltersen and Sørensen, 2010). It was not known if, in practice, it can scale up to large
instances. Our experiments show that it does not. While in principle also an extensive-form
perfect equilibrium (EFPE) can be computed using a simplex algorithm that deals with theperturbation symbolically, it is not even clear whether it can run in polynomial space. In summary,although there is agreement that Nash equilibrium refinements can play an important role evenin two-player zero-sum games, prior algorithms do not scale in practice.
Some algorithms have been proposed for computing approximate trembling-hand equilibria
resorting to regret-minimization techniques (Farina, Kroer, and Sandholm, 2017) or to smoothing
methods paired with bilinear saddle-point techniques (Kroer, Farina, and Sandholm, 2017). Thosealgorithms do not provide any guarantee of finding or approximating actual QPEs or EFPEs.

256§13.2. REFINED STRATEGIES AS SOLUTIONS TO TREMBLING LINEAR PROGRAMS

Rather, they provide approximate solutions to approximate solution concepts.
Finally, we mention proper equilibria (PEs), proposed by Myerson (1978). PEs form a non-empty
subset of EFPEs, with the additional requirement that the worse an action is for a player, the lower
the agent’s tremble probability on that action must be. Therefore, the trembles are a function of
the strategies of the players. This potentially complicates equilibrium finding. It is unknown
whether PEs can be efficiently found in imperfect-information extensive-form games, and thetechniques in this chapter do not apply to PE.

13.2 Refined strategies as solutions to trembling linear programs
In this chapter we will focus on three notions of trembling-hand refinements: extensive-form
perfect equilibria (EFPE, already seen in Chapter 12), quasi-perfect equilibrium (QPE, definedin Section 13.2.2), and one-sided quasi-perfect equilibrium (OS-QPE, defined in Section 13.2.3).
As we will show, in two-player zero-sum games all these equilibria are similar, in the sense thatthey can be expressed as limit points, as ϵ → 0+ , of sequences of optimal solutions to appropriate
linear programs (LPs) shows entries are polynomials of ϵ > 0. In this section, we formalize thistype of problems into a general framework and develop a series of results that apply not only to
QPEs, OS-QPEs, and EFPEs, but to any other computational problem with that general structure
as well.
Definition 13.1 (Trembling linear program). A trembling linear program (TLP) is a linear
program parametric in ϵ > 0,


 max
P (ϵ) :s.t.
c(ϵ)⊤ xA(ϵ) x = b(ϵ)
x ⩾ 0,

where A(ϵ), b(ϵ), c(ϵ) are polynomials in the variable ϵ with rational coefficients. Furthermore,
we require that the set of all feasible solutions for P (ϵ) be non-empty for all sufficiently smallϵ, and that the set of all feasible solutions for P (0) be non-empty and bounded.

In the case of QPEs (Section 13.2.2) the perturbation variable ϵ only appears in b, whereas in thecase of EFPEs (Section 13.2.1) the perturbation ϵ only appears in A.[13.a][13.a] In principle, one could also study the case where ϵ appears only in c.

To our knowledge, that does not haveapplications to games, and has not been studied in that context. However, it has been studied, for example, for⊤⊤lexicographic multi-objective LPs Isermann, 1982. In particular, given the set of objectives c⊤
1 x, c2 x, . . . , cn x, one cann−1⊤define a new objective (c1 + ϵc2 + · · · + ϵcn ) x. That formulation is fundamentally equivalent to the grossone-based
approach Cococcioni, Pappalardo, and Sergeyev, 2018. Even for the grossone-based approach, our TLP-based approachovercomes some of the key limitations of the work by Cococcioni, Pappalardo, and Sergeyev (2018). In particular, ouralgorithm for finding a TLP limit solution (Section 13.4) is purely numerical (yet provably correct)—thus avoiding the time§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE257

We now formalize the concept of limit solution for a TLP:
Definition 13.2 (Limit solution of a TLP). A limit solution of a TLP P (ϵ) is a limit point of
optimal solutions for P (ϵ) as ϵ → 0+ .

In the rest of the section, we show that EFPE, QPE, and OS-QPE strategies can all be expressed
as limit solutions of appropriately defined trembling LPs. To do so, we will assume that a
two-player zero-sum game has been fixed. Analogously to the previous chapter, for each playeri ∈ {1, 2}, we let Fi ∈ ℝJi ×Σi and f i ∈ ℝJi define the player’s sequence-form polytopeiQi = {x ∈ ℝΣ
⩾0 : Fi x = f i , x ⩾ 0}.

Furthermore, we let Ui be the payoff matrix of Player i.

13.2.1

Extensive-form perfect equilibria as trembling linear programs

As defined in Definition 12.1 in Chapter 12, an EFPE is a limit point, as ϵ → 0+ , of Nash equilibriaof the restricted game in which players need to select each action with probability at least ϵ. Using
the fact that in two-player zero-sum games Nash equilibrium strategies are solutions to a linear
program, we obtain the following result.
Proposition 13.1. Any limit point of solutions to the trembling linear program


max


x̃i ,v −i



s.t.







f⊤
−i v −i
1

−⊤
−⊤
⊤ −1
R−i
(ϵ)F⊤
−i v −i ⩽ R−i (ϵ)Ui Ri (ϵ)x̃i

2

Fi Ri−1 (ϵ)x̃i = f i3x̃i ⩾ 0.

as the trembling magnitude ϵ → 0+ is an extensive-form perfect equilibrium strategy forPlayer i ∈ {1, 2}.

Proof. A Nash equilibrium strategy for Player i ∈ {1, 2} in a two-player zero-sum game isa max-min strategy. Incorporating the constraint that the strategy xi pick each action withprobability at least ϵ, captured via the behavioral perturbation matrix Ri (ϵ) as Ri (ϵ)xi ⩾ 0, we
obtain the optimization problem

and space penalty for storing the symbolic powers of the grossone unit—and not tied to the use of the simplex algorithm,
thereby guaranteeing a polynomial run time.

258§13.2. REFINED STRATEGIES AS SOLUTIONS TO TREMBLING LINEAR PROGRAMS








 max

xi




s.t.







min

 x−i

F−i x−i = f −i2
R−i (ϵ) x−i ⩾ 0
Fi x i = f i
Ri (ϵ) xi ⩾ 0.

s.t.



3
4

x⊤
i Ui x−i1(13.1)

Since the perturbation matrices Ri and R−i are invertible (Lemma 12.1), we can substitutex̃i = Ri (ϵ)xi , x̃−i = R−i (ϵ)x−i , obtaining








maxx̃i






s.t.







minx̃−i


s.t.




−⊤
−1
x̃⊤
i Ri (ϵ)Ui R−i (ϵ)x̃−i1−1
F−i R−i(ϵ)x̃−i = f −i2x̃−i ⩾ 0

3

Fi Ri−1 (ϵ) x̃i = f i4x̃i ⩾ 0.

Taking the dual of the inner problem introducing the vector of dual variables v−i for constraint
1 , we obtain the LP in the statement.

13.2.2

Quasi-perfect equilibria: Definition and formulation

Quasi-perfection, introduced by van Damme (1984), is significantly more intricate to define
than extensive-form perfection. Instead of giving an explicit lower bound on the probability
with which each action needs to be selected, the definition of a quasi-perfect equilibrium (QPE)
relies on a refined notion of best response. We now give one of the multiple known equivalentdefinitions, and we present it for the special case of two-player games only. Several equivalentdefinitions that apply to more general games can be found in the original work by van Damme,
as well as in the work by Miltersen and Sørensen (2010) and Gatti, Gilli, and Marchesi (2020).
Definition 13.3 (I-local purification). Let i ∈ {1, 2} be a player, x be a strategy for Player i,and let I ∈ Ii be an information set. We say that a strategy x′ for Player i is an I-local
purification of x if x′ is deterministic at any information set I ′ ≽ I, and coincides with x at
any other information set. When x′ is an I-local purification of x, we further say that• x′ is ϵ-consistent with x if, for all I ′ ≽ I, x′ assigns probability 1 only to actions that haveprobability ⩾ ϵ in x;• π ′ is optimal against a given strategy of the opponent if no other I-local purification of xachieves (strictly) higher expected utility against said strategy of the opponent.

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
259

Definition 13.4 (ϵ-quasi-perfect best response). A strategy xi is an ϵ-quasi-perfect best responseto the opponent strategy x−i if(i) x assigns strictly positive probability to all actions of Player i; and(ii) for all information sets I ∈ Ii of Player i, every ϵ-consistent I-local purifications of xi(Definition 13.3) is optimal for x−i .
A strategy profile (x1 , x2 ) where each strategy is an ϵ-quasi-perfect best response to the
opponent’s strategy is called an ϵ-quasi-perfect strategy profile.

Definition 13.5 (Quasi-perfect equilibrium). A quasi-perfect equilibrium is any limit point of
ϵ-quasi-perfect strategy profiles as ϵ → 0+ .

It is known since the work by Miltersen and Sørensen (2010) that some QPEs (we call them
Miltersen-Sørensen QPEs) can be computed in any two-player game as the limit point of Nashequilibria of perturbed games Γ(ϵ), akin to EFPE. The subtlety is that while in EFPE each perturbed
game Γ(ϵ) mandates a lower bound of ϵ on the probability of playing each action, in the case of a
Miltersen-Sørensen QPE the lower bounds are given on the probability of each sequence of actions.
iSpecifically, for any ϵ ⩾ 0 and for each player i ∈ {1, 2}, let ℓi : ϵ → ℝΣ
>0 denote the vector whose
entries are defined asℓi (ϵ)[σ] = ϵdepth(σ)∀σ ∈ Σi ,(13.2)

where depth(σ) denotes the number of Player i’s actions on the path from the root of theplayer’s tree-form decision process down to the sequence σ (the empty sequence has a depth of0). Miltersen and Sørensen (2010) prove that any limit point of the solution to the perturbed
optimization problem
max

min

F 1 x1 =f 1 F 2 x2 =f 2

x⊤
1 U1 x 2 .

(13.3)

x1 ⩾ℓ1 (ϵ) x2 ⩾ℓ2 (ϵ)

is a Miltersen-Sørensen QPE.[13.b][13.b] Recently, Gatti, Gilli, and Marchesi (2020) took this construction further, and showed that any QPE can be expressed

as a limit point of solutions to (13.3), as long as more general vectors of polynomials ℓ1 , ℓ2 are used than in (13.2). In thischapter, we will solely focus on Miltersen-Sørensen-style perturbation as defined in (13.2).

260§13.2. REFINED STRATEGIES AS SOLUTIONS TO TREMBLING LINEAR PROGRAMS

Proposition 13.2. Any limit point of solutions to the trembling linear program x̃i ,vmax−i ,z −i



s.t.







⊤
f⊤
−i v −i + ℓ−i (ϵ) z −i

1

⊤
⊤
F⊤
−i v −i − Ui x̃i ⩽ Ui ℓi (ϵ) − z −i2

Fi x̃i = f i − Fi ℓi (ϵ)

3x̃i , z −i ⩾ 0

as the trembling magnitude ϵ → 0+ is a Miltersen-Sørensen QPE strategy for Player i ∈ {1, 2}.

Proof. Similarly to the proof of Proposition 13.1, we start from the problem of finding a maxmin
strategy for Player i in the game subject to the perturbation constraints,







 maxxi




s.t.






 minx−i
s.t.


3
4

x⊤
i Ui x−i12

F−i x−i = f −ix−i ⩾ ℓ−i (ϵ)(13.4)

Fi xi = f ixi ⩾ ℓi (ϵ).

Taking the dual of the inner minimization problem, we obtain







maxxi






 s.t.
maxv−i ,z−i


s.t.



3

Fi x i = f i

4xi ⩾ ℓi (ϵ).

⊤
f⊤−i v−i + ℓ−i (ϵ) z1⊤F⊤−i v−i + z ⩽ Ui xi

2z −i ⩾ 0

Finally, by substituting x̃i = xi − ℓi (ϵ), x̃−i = x−i − ℓ−i (ϵ) we obtain the formulation in thestatement.

13.2.3One-sided quasi-perfect equilibrium: Definition and formulation

Both EFPE and QPE define “two sided” notions of refinements, in the sense that both players
are trembling. That two-sidedness comes at a computational cost: both the domain of the
maximization and minimization problem in the saddle-point formulations (for example, Equation (13.3) in the case of QPE) of the refinements are perturbed, making the computation of alimit point computationally expensive. Yet, in many strategic interactions of interest, a player
might be concerned about being able to capitalize on the opponent’s mistakes, but not about§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
261

making mistakes of her own. After all, in the age of machines, that player might well be a bot
interacting strategically (for example, playing a poker tournament) against imperfect opponents.
In that situation, the player in question might therefore seek, in the interest of lowering thecomputational requirement of computing a robust strategy, to find equilibrium points that arerobust to perturbations of the opponent’s strategy only, thereby breaking the two-sidedness of all
known trembling-hand equilibrium refinements.
We introduce a one-sided trembling-hand refinement, which we coin one-sided quasi-perfectequilibrium (OS-QPE, Farina and Sandholm, 2021a). Because of the asymmetric role of the players,
from now on we stop referring to the players as Player 1 and 2, and adopt the terms machineplayer and imperfect player to highlight their asymmetric role. The machine player is assumed tonever make mistakes: lower bounds on the probability of play (the “trembling hands”) are only
introduced for the imperfect player. Accordingly, from now on we will drop subscripts 1 and 2 todenote quantities that belong to the players, and will use m and h for quantities belonging to themachine and the imperfect player, respectively.
Definition 13.6 (One-sided quasi-perfect equilibrium). We call a strategy profile (π m , π h ) a
one-sided ϵ-quasi-perfect strategy profile if π h is an ϵ-quasi-perfect best response (Definition 13.4)
to π m , and π m is a best response to π h . We say that (π m , π h ) is an one-sided quasi-perfectequilibrium if it is the limit point of one-sided ϵ-quasi-perfect strategy profiles, as ϵ → 0+ .
In two-player zero-sum games, a OS-QPE is any limit point as ϵ → 0+ of solutions of thebilinear saddle-point problem
max

min

xm ⩾0

xh ⩾ℓh (ϵ)

Fm xm =f m Fh xh =f hx⊤m Um xh(13.5)

where ℓh (ϵ) is as in Equation (13.2). Then, for any ϵ > 0 for which the domain of the minimization
problem is nonempty, any solution to (13.5) is a one-sided ϵ-quasi-perfect strategy profile.
Following steps similar to those we took for QPE in Proposition 13.2, we obtain the following
trembling LP formulation.
Proposition 13.3. Any limit point of solutions to the trembling linear program

P (ϵ) :=

arg max (Um ℓh (ϵ))⊤xm + (f h − Fh ℓh (ϵ))⊤v



xm



s.t. 1 U⊤ x − F v ⩾ 0
m








m

h

2

Fm xm

= fm

3

xm ⩾ 0, v free.

262§13.2. REFINED STRATEGIES AS SOLUTIONS TO TREMBLING LINEAR PROGRAMS

as the trembling magnitude ϵ → 0+ is a one-sided quasi-perfect equilibrium strategy for themachine player.

13.2.4Formulations with sparsified payoff matrices

As we discuss in Section 13.4, our algorithm for solving trembling linear programs relies onsolving the linear program P (ϵ) for different numerical instantiations of the value of ϵ > 0. Since
the solution of the linear programs is the bottleneck of our algorithm, generally speaking thesparser the formulation of the linear programs P (ϵ), the better. The use of sparsified payoff
matrices was recently shown to help speed up the solution of linear programs representing Nashequilibrium computations (B. H. Zhang and Sandholm, 2020). Farina and Sandholm (2022) show
games with a strong combinatorial structure such as poker can be sparsified efficiently, andpropose a more general framework than that of B. H. Zhang and Sandholm (2020). In particular,a sparsification of Player i’s payoff matrix Ui is a decomposition of the form
Ui = Ûi + Si K−1 Vi⊤ ,(13.6)

such that the combined number of nonzeros in Ûi , KSi , and Vi is significantly smaller than thenumber of nonzeros in Ui , and K is a square invertible matrix.
All formulations of refinements (Propositions 13.1 to 13.3) can benefit from a sparsified payoff
matrix directly. As an illustration, we consider the case of OS-QPE, but the technique applies to allother equilibria. By plugging in (13.6) into the constraints of the formulation of Proposition 13.3,
we can express OS-QPE as limit solutions of the trembling LP

P (ϵ) :=

arg max (Um ℓh (ϵ))⊤xm + (f h − Fh ℓh (ϵ))⊤v


 xm



s.t. 1 (Û + S K−1 V⊤ )⊤ x − F v ⩾ 0
m








m

m

h

Fm x m2
3

m

xm ⩾ 0, v free.

Hence, by introducing the variable⊤⊤y m := K−⊤ S⊤
m xm ⇐⇒ K y m = Sm xm ,we can rewrite the formulation into sparsified form as= fm

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE263


arg max (Um ℓh (ϵ))⊤xm + (f h − Fh ℓh (ϵ))⊤v



xm




− K⊤ y m = 0
s.t. ⋆ S⊤m xm
P (ϵ) :=1 Û⊤m xm − Fh v + Vm y m ⩾ 0





2 Fm xm
= fm




3 xm ⩾ 0, v free,where the dense Um matrix in constraint 1 has been replaced with a much smaller combined
number of nonzeros due to the presence of K, Vm , and Sm in the sparsified formulation. Dueto the reduced number of nonzero entries in the LP, solving the LP typically requires lesscomputational resources (time and memory).
We will use the sparsified formulation of EFPE, QPE, and OS-QPE when testing on real poker
endgames in Section 13.5.

13.3

Basis stability

We now introduce the concept of basis stability for TLPs. As we will show, there is a tightconnection between a stable basis and a TLP limit solution. In particular, given a stable basis, one
can find a TLP limit solution in polynomial time.
First, recall that a basis of an LP is a subset of the program’s variables such that the submatrix
B obtained considering the columns of A corresponding to variables in the basis is square andinvertible. Now we define our notion of basis stability.
Definition 13.7 (Stable basis). Let P (ϵ) be a TLP. The LP basis B is called stable if there existsϵ̄ > 0 such that B is optimal for P (ϵ) for all ϵ : 0 < ϵ ⩽ ϵ̄.

Theorem 13.1 states a connection between stable bases and TLP limit solutions. Informally,
Theorem 13.1 guarantees that once we know a stable basis B, we can recover a limit solution of
the TLP by simply taking the limit of the solutions to the underlying perturbed LP induced by
B.
Theorem 13.1. Let P (ϵ) be a TLP, and let B be a stable basis for P , optimal for all ϵ : 0 < ϵ ⩽ ϵ̄.
Furthermore, let x(ϵ) := B(ϵ)−1 (ϵ) b(ϵ) be the optimal basic solution of P (ϵ) corresponding
to B. Then,
1. x̃ = limϵ→0+ x(ϵ) exists, and2. x̃ is a limit solution to the TLP P .

264§13.3. BASIS STABILITY

Proof. The fact that x̃ is a solution to P follows directly from Definition 13.2. Therefore, it isenough to show the existence of x̃.
For all ϵ ⩾ 0, let B(ϵ) be the basis matrix in P (ϵ), corresponding to the given basis B. By
hypothesis, B is optimal for P (ϵ) for all ϵ : 0 < ϵ ⩽ ϵ̄; hence, B(ϵ) is invertible for all ϵ : 0 < ϵ ⩽ ϵ̄and we conclude that det B(ϵ) is not identically zero over that range. This implies thatx(ϵ) = B−1 (ϵ) b(ϵ)
is well defined for all 0 < ϵ ⩽ ϵ̄ and is a vector of rational functions. This, together with theboundedness assumption of the feasible set of P (0), is enough to conclude that limϵ→0+ x(ϵ)
exists.
The matrix of (symbolic) rational functions B−1 (ϵ) in Theorem 13.1 can be computed inpolynomial time in the size of the TLP starting from B(ϵ). Hence, we can compute all the rational
function entries of x(ϵ) in polynomial time, and therefore also limϵ→0+ x. Thus we have thefollowing result.
Theorem 13.2. Let P be a TLP, and let B be a stable basis for P . A limit solution to P can becomputed in polynomial time in the size of the input TLP.

13.3.1Analytic basis stability condition and existence of stable bases

Theorem 13.2 above shows that the problem of solving a TLP is not harder than the problem offinding a stable basis for it. In this subsection, we focus on this latter problem, showing that astable basis always exists and can be computed in polynomial time given access to an efficient LP
oracle.
Given a TLP and a stable basis B for it, let B(ϵ) be the basis matrix corresponding to B in theunderlying perturbed LP P (ϵ). From the theory of LPs, we know that B is optimal for P (ϵ) if andonly if (see, for instance, the book by Dantzig and Thapa (2006)):• it is primal-feasible, that is,B−1 (ϵ) b(ϵ) ⩾ 0.
(In practice, sometimes we have non-negativity constraint on only some of the variables,
but not all. In this case, we can simply check that B−1 (ϵ)b(ϵ) is non-negative only for what
concerns the relevant entries.)• the reduced costs of all nonbasic columns are nonpositive, that is,§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
265

−1
c⊤
− c⊤
(ϵ)B̄(ϵ) ⩽ 0,BB
B̄

where cB is the part of c corresponding to the basic variables, cB̄ is the part of c corresponding
to the nonbasic variables, and B̄(ϵ) is the matrix formed by all nonbasic columns.
Therefore, we can define the following analytical notion of a stability certificate. It collects theconditions above into a vector tB (ϵ), which is nonnegative if and only if B is an optimal basis for
the LP P (ϵ). Therefore, by the definition of basis stability, a basis is stable if tB (ϵ) is nonnegative
for all sufficiently small values of ϵ. Formally, we have the following.
Theorem 13.3. Given a TLP P (ϵ), a basis B is stable if and only if there exists ϵ̄ > 0 such thattB (ϵ) :=B−1 (ϵ) b(ϵ)
B̄⊤ (ϵ)B−⊤ (ϵ)cB − cB̄!
⩾0∀ ϵ : 0 ⩽ ϵ ⩽ ϵ̄.

The vector tB (ϵ) is called the stability certificate for B.

13.3.2

Existence of stable bases

We use Theorem 13.3 as an important building block to prove the following. The proof is presented
in the appendix.
Theorem 13.4. Given as input a TLP P (ϵ), there exists ϵ∗ > 0 such that any optimal basis forthe numerical LP P (ϵ∗ ) is stable. Furthermore, such a value ϵ∗ can be computed in polynomial
time in the input size.a
aWe assume that a polynomial of degree d requires Ω(d) space to represent in the input. If this were not thecase, evaluating a polynomial in an integer n would not be an efficient operation, since it requires Ω(d log n) bits torepresent the output.

Before showing the proof of Theorem 13.4 we recall a couple of simple facts from Chapter 12.
Lemma 12.6 (Restated). Letp(ϵ) :=

a0 + a1 ϵ1 + · · · + an ϵnb0 + b1 ϵ1 + · · · + bm ϵm

be a rational function such that a0 , b0 ̸= 0, and let µa := maxi |ai |, µb := maxi |bi |. Then p(ϵ)
has the same sign of a0 /b0 for all 0 ⩽ ϵ ⩽ ϵ∗ , where266§13.3. BASIS STABILITY∗ϵ := min


|a0 |
|b0 |
,
.
µ + |a0 | µ + |b0 |Lemma 12.7 (Restated). Letp(ϵ) :=

a0 + a1 ϵ1 + · · · + an ϵnb0 + b1 ϵ1 + · · · + bm ϵm

be a rational function with integer coefficients, where the denominator is not identically zero;let µa := maxi |ai |, µb := maxi |bi |, µ := max{µa , µb } and ϵ∗ := 1/(2µ). Then exactly one of
the following holds:• p(ϵ∗ ) = 0 for all 0 < ϵ ⩽ ϵ∗ ,• p(ϵ∗ ) > 0 for all 0 < ϵ ⩽ ϵ∗ ,• p(ϵ∗ ) < 0 for all 0 < ϵ ⩽ ϵ∗ .
We now proceed with the proof of Theorem 13.4.
Proof of Theorem 13.4. Let B ⋆ be the set of all bases for P (ϵ) that are optimal for at least one
ϵ ∈ ℝ>0 . For any B ∈ B ⋆ , we let tB (δ) be the stability certificate for B (Theorem 13.3). All
entries of tB (δ) are rational functions in δ; hence, by Lemma 12.6, we can find a value δB∗ > 0,such that all entries of tB (δ) keep the same sign on the domain 0 < δ ⩽ δB∗ . We now introduce
the function f : B ⋆ → ℝ>0 that maps every B ∈ B ⋆ to the corresponding value of δB∗ . Since B ⋆is finite, min f exists and is (strictly) positive; this means that any optimal basis for P (min f ) is
optimal for all P (ϵ) where 0 < ϵ ⩽ min f , and is therefore stable.
In light of the above, we only need to prove that we can compute a lower bound for min fin polynomial time. We will assume without loss of generality that the objective function is
not perturbed. Furthermore, we will assume without loss of generality that A(ϵ), b(ϵ) and c(ϵ)
only contain integer entries (if not, it is enough to multiply all the entries in the LP by the least
common multiple of all denominators to satisfy this assumptions). As long as we can provethat the maximum coefficient appearing in tB is polynomially large (in the size of the input
TLP), the result follows from the bound in Lemma 12.6.
The entries of the stability certificate are obtained by composing sums and products ofentries from three vectors: the LP matrix A(ϵ), the inverse of the basis matrix B−1 (ϵ), the vector
of constants b(ϵ) and the objective function coefficients c. Let M be the largest coefficient thatappears in A(ϵ), b(ϵ) and c, and let m be the largest polynomial degree appearing in A(ϵ) andb(ϵ). We now study the magnitude of the maximum coefficient and the maximum polynomial
degree that can appear in B(ϵ)−1 .

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
267

Introducing C(ϵ) = cof B(ϵ), the cofactor matrix of B(ϵ), we can write the well-known
identityB−1 (ϵ) =C(ϵ)⊤
det B(ϵ)

Denominator. We now give an upper bound on the coefficients of the denominator of theentries in B−1 (ϵ). By using Hadamard’s inequality, we can write
coeff(det B(ϵ)) ⩽ nn/2 M n coeff((1 + ϵ + · · · + ϵmA )n ),where coeff(·) is the largest coefficient of its polynomial argument. Since
coeff((1 + ϵ + · · · + ϵmA )n ) ⩽ (mA + 1)n ,we havecoeff(det B(ϵ)) ⩽ nn/2 ((mA + 1)M )n ,deg(det B(ϵ)) ⩽ n · mA .

Notice that this bound is valid for all possible basis matrices B(ϵ).
Numerator. It is easy to see that the bounds on coeff(det B(ϵ)) hold for the cofactor matrix as
well:coeff(det C(ϵ)⊤ ) ⩽ nn/2 ((mA + 1)M )n ,deg(det C(ϵ)⊤ ) ⩽ n · mA .

Again, it is worthwhile to notice that this bound is valid for all possible basis matrices B(ϵ).
Stability certificate.

We havecoeff(C⊤ b(ϵ)) ⩽ coeff(B̄⊤ (ϵ) C(ϵ)⊤ cB ) ⩽ nn/2 ((mA + 1)M )n · mA · Mc M,
coeff(det B(ϵ) cB̄ ) ⩽ nn/2 ((mA + 1)M )n · Mc .
Hence,
coeff(tB (ϵ)) ⩽ nn/2 ((mA + 1)M )n · Mc (mA M + 1)
⩽ nn/2 ((mA + 1)M )n+1 · Mc .
Therefore, all coefficients involved require a polynomial number of bits to be represented,
concluding the proof.

268§13.4. A PRACTICAL ALGORITHM FOR FINDING A TLP LIMIT SOLUTION

The existence of polynomial-time algorithms for solving LPs (Karmarkar, 1984), as well as forfinding optimal basic solutions (Megiddo, 1991), taken together with Theorems 13.2 and 13.4,
immediately imply the following corollary.
Corollary 13.1. A limit solution to a TLP can be found in polynomial time by means of thefollowing algorithm.

Algorithm 13.1: Naïve algorithm for finding a limit solution to a TLP P (ϵ).
Compute the value ϵ∗ as described in the proof of Theorem 13.4;∗
2 Extract an optimal basis B for P (ϵ );3 Extract the (symbolic) basis matrix B(ϵ) corresponding to B;−1
4 Compute the symbolic vector x(ϵ) = B(ϵ) b(ϵ);
5 return x̃ = limϵ→0+ x(ϵ) ;[▷ See also Theorem 13.2]
1

When this algorithm is applied to the EFPE or QPE TLPs, it essentially specializes into thealgorithm proposed by Miltersen and Sørensen (2010) for QPE and the one proposed in Chapter 12
for EFPE. In practice, as we show in the experiments later in this dissertation, this algorithm isextremely inefficient, because it involves finding an exact solution to an LP whose numericalconstants require a large number of bits. Therefore, we devote the next section to developing apractical algorithm for finding limit solutions to general TLPs.

13.4

A practical algorithm for finding a TLP limit solution

We now develop a practical algorithm for finding a limit solution in a TLP P (ϵ). It avoids thepessimistically small numerical perturbation ϵ∗ of Theorem 13.4 by using an efficient stabilitychecking oracle for checking if a basis is stable or not. It enables an iterative algorithm that
repeatedly picks a numerical perturbation ξ, computes an optimal basis for the perturbed LP
P (ξ), and queries the basis-stability oracle. If the basis is not stable, the algorithm concludes
that the perturbation value ξ was too optimistic, and a new iteration is performed with a smaller
perturbation (for example, ξ/2). On the other hand, if the basis is stable, the algorithm takes
the limit of the LP solution and returns it as the limit solution of the TLP (by Theorem 13.1, thisis guaranteed to provide a limit solution). Termination of the algorithm is guaranteed by thefollowing observation.
Observation 13.1. Let ϵ∗ be as in Theorem 13.4. Any value of ξ in the range (0, ϵ∗ ] guarantees
termination of the algorithm. Indeed, by Theorem 13.4, any optimal basis for P (ξ) is stable§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
269

and makes our iterative algorithm terminate. Furthermore, if after every negative stability
test the value of ξ is reduced by a constant multiplicative factor (e.g., halved), then sinceϵ∗ only has a polynomial number of bits, the algorithm terminates after trying at most apolynomial number of different values for ξ.
This algorithm is summarized pictorially in Figure 13.1.
P, ξ

Solve P (ξ)
B
Reduce ξ

Not stable

Check basis
stability

StableEvaluate limit
of solutionx∗

Figure 13.1: High-level overview of the steps of our practical algorithm for finding a TLP limit solution.

Two practical considerations contribute to making our algorithm appealing. First, as long as ξis sufficiently large, we can use any finite-precision LP solver to solve the numerically-perturbed
linear program. However, as soon as ξ becomes small, a finite-precision solver is doomed to failbecause of numerical instability, and an implementation of the simplex algorithm supporting
arbitrary-precision rational numbers is required. Second, the optimal basis found in the previous
iteration can be used to warm-start the next iteration. This greatly reduces the runtime of themethod, as iterations of the arbitrary-precision simplex method are extremely expensive.
In the next section, we formally state the purpose of the basis-stability oracle.

13.4.1

Basis-stability oracle

We now formally state the purpose of the basis-stability oracle.
Definition 13.8. Given a TLP P (ϵ) and a basis B optimal for some numeric instance P (ϵ̄), abasis-stability oracle determines whether B is stable or not.

It is not necessary—and in general not true—that the inverse of B(0) exist. We start from thesimpler case in which B−1 (0) exists (thus ruling out the possibility that the stability certificate tB
is not defined in 0) and later move to the general case.
However, the existence of B−1 (0) allows further numerical optimizations making the overall
algorithm fast. In what follows we separately analyze these two cases (singular versus non-singular
B(0)).

270

13.4.2§13.4. A PRACTICAL ALGORITHM FOR FINDING A TLP LIMIT SOLUTION

Oracle for non-singular basis matrices

If B(0) is non-singular, then B−1 (ϵ) b(ϵ) and B̄⊤ (ϵ) B−⊤ (ϵ) cB − cB̄ are analytic functions of ϵ at
ϵ = 0. Thus, tB (ϵ) is analytic at ϵ = 0. In other words, each entry ti (ϵ) of tB (ϵ) is equal to its Taylor
expansionti (ϵ) = αi0 +αi1
αi2 2 αi3 3ϵ+ϵ +ϵ + ···
1!
2!
3!

where αij = (dj ti (ϵ)/dϵj )(0) is the j-th derivative of ti (ϵ) evaluated at ϵ = 0.[13.c] The sign of ti (ϵ)
in positive proximity[13.d] of 0 is the same as the first (i.e., relative to the lowest degree monomial)
non-zero coefficient of the expansion of ti (ϵ) around 0. In other words, there exists a ϵ̄ > 0 suchthat ti (ϵ) has the same sign as the first non-zero derivative of ti evaluated in 0 for all 0 < ϵ < ϵ̄. If
all derivatives are 0, then we conclude that ti (ϵ) is identically zero around ϵ = 0.
This suggests a simple algorithm for determining whether B is stable: we compute its stability
certificate tB (ϵ) and repeatedly differentiate each row until we either determine the sign of thatrow in positive proximity of 0 or we establish that the row is identically zero. If all the rows
happen to be non-negative in positive proximity of 0, then the basis is stable; otherwise, it is not.
In order to make the algorithm fast, we need to be able to quickly evaluate tB (ϵ) and its derivatives
at 0. This fundamentally reduces to our ability to efficiently compute a Taylor expansion of afunction of the form B−1 (ϵ) H(ϵ) around ϵ = 0, where H is a matrix or vector whose entries arepolynomial in ϵ. This part of the algorithm assumes that a sparse LU factorization of the numericalbasis matrix B(0) is available; one is easy to compute in polynomial time. Below, we will break
the presentation of the algorithm into multiple steps. Since the algorithm described below can beapplied to any square matrix M(ϵ)—not only to a basis matrix B(ϵ)—with polynomial entries
and with nonzero determinant at ϵ = 0, we will use the symbol M(ϵ) in place of B(ϵ).
Derivatives of M−1 (ϵ) H. We start by showing how to efficiently and inductively evaluate
derivatives of M−1 (ϵ) H in 0, where H is a constant matrix or vector. We start with a simple
lemma.
Lemma 13.1. For all n ⩾ 1,n   iX
n d M(ϵ) dn−i M−1 (ϵ)
i=0

i

dϵi

dϵn−i

= 0.

[13.c] Throughout this dissertation, we define the zeroth derivative d0 f /dϵ0 of f to be f itself.
[13.d]We say that a property parametrized by ϵ is true in positive proximity of 0 to mean that there exists a ϵ̄ > 0 such thatthe property holds for all ϵ : 0 < ϵ < ϵ̄. We say that the property is true in proximity of 0 if there exists a ϵ̄ > 0 such thatthe property holds for all ϵ : 0 < |ϵ| < ϵ̄.

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
271

Proof. The statement is equivalent to the expansion of the identitydn(M(ϵ) M−1 (ϵ)) = 0,dϵnwhich holds true for all n ⩾ 1, by means of the product rule of derivatives.
Lemma 13.1 implies thatM(0)n   iXdn−i M−1
dn M−1
n dM
(0)
=
−
(0)
(0).
n
i
dϵ
i dϵ
dϵn−i
i=1
n

−1

M
Multiplying by H and introducing the symbol Dn := d dϵ(0) H, we obtainnM(0) Dn = −n   i
X
n dM
i=1

i

dϵi

(0) Dn−i .

The right hand side is relatively inexpensive to compute, especially when n is small. Indeed,
computing (di M/dϵi )(0) amounts to extracting the coefficients of the monomials of degree iof the polynomial entries in M. This can be done extremely efficiently by reading directlyfrom the perturbed LP constraint matrix A. Therefore, if we inductively assume knowledge ofD0 , D1 , . . . , Dn−1 , we can easily compute Dn using the precomputed LU factorization of M(0).
Derivatives of M−1 (ϵ) H(ϵ). We now turn our attention to the computation of the derivativesof M−1 (ϵ) H(ϵ), where H(ϵ) can be any matrix or vector with polynomial entries. This case isparticularly relevant, as it applies to both the primal-feasibility conditions and the reduced costs.
We introduce the formal symbol ⟨i, j⟩ defined over pairs (i, j) ∈ ℕ2 as⟨i, j⟩ :=di M−1
dj H(0) j (0).
idϵdϵBy means of the product rule, we have thatn  Xdn (M−1 H)n(0) =⟨i, n − i⟩.
n
dϵ
i
i=0

(13.7)

From the previous section, we know how to compute ⟨i + 1, j⟩ having access to ⟨0, j⟩, ⟨1, j⟩, . . .,⟨i, j⟩. On the other hand, ⟨0, j⟩ = M(0)−1 dj /dϵj H(0) is easy to compute having access to the LU
factorization of M(0). Therefore, Equation (13.7) gives an efficient way of expanding M−1 (ϵ)H(ϵ)
into its power series. Finally, we address the problem of determining, row by row in the derivative
vector in the Taylor series, when it is safe to stop after observing only zero-valued derivatives forsome row for a number of iterations (i.e., a number of terms in the Taylor series).

272§13.4. A PRACTICAL ALGORITHM FOR FINDING A TLP LIMIT SOLUTION

Lemma 13.2. Consider a TLP ϵ 7→ P (ϵ) where P (ϵ) has n rows and let m be the maximum
degree appearing in the polynomial functions defining P . Given any basis B, if the first
2nm + 1 derivatives of the i-th entry of the stability certificate tB (ϵ) are all zero, the entry is
identically zero.

Since 2nm + 1 is a polynomial number in the input size, we conclude that the overall algorithm
runs in polynomial time, since it terminates in a polynomial number of steps and each step takes
polynomial time. There is a more convenient way of determining whether a given row is 0. It issufficient to pick a random number ϵ̃ (for example in (0, 1)), and evaluate the rational function tiat ϵ̃: if ti (ϵ̃) = 0, then ti is identically zero with probability 1 because of the fundamental theoremof algebra. This is the variant that we use in the experiments later in this dissertation.
Finally, in some cases we can take theoretically sound shortcuts to further enhance the speedof the algorithm. For example, in the formulation of QPE given in Section 13.2.2, the LP constraint
matrix A is constant, meaning that the stability certificate b has polynomial entries (as opposed
to ratios of polynomials entries). In this case, we can avoid computing all the derivatives of B−1 ,with large practical savings of time and space.

13.4.3

Oracle for singular basis matrices

We now show how to deal with a singular B(0). An example of this phenomenon is shown in thenext example.
Example 13.1. Consider the TLP of Figure 13.2; there, the (only) basis matrix
B(ϵ) =
is not invertible when ϵ = 0.

2
1 − ϵ22
1 + ϵ − ϵ2!

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
273

x2

max



 s.t.
P (ϵ) :





1/2

1
2

0

OPT

x1

x2
1
2
3

2 x1 + 2 x2 = 1(1 − ϵ2 ) x1 + (1 + ϵ − ϵ2 ) x2 = 1/2
x1 , x2 ⩾ 0

The optimal solution (OPT) is x(ϵ) =

1/2



1−ϵ ϵ
,
, for all ϵ > 0.
22

Figure 13.2: Example of a situation where the basis matrix B(ϵ) is singular at ϵ = 0.

The core idea of our method is to replace the computation of the Taylor expansion of the stability
certificate around ϵ = 0 with a Laurent expansion, that is, a power series in ϵ where negative
exponents are allowed. Lemma 13.3 provides the key result that enables this process.
Lemma 13.3. Let M(ϵ) be a square matrix with polynomial entries, not all of which are
identically zero. Then there exist k ∈ ℕ+ and matrices M̃(ϵ) and T(ϵ) that have polynomials
as entries, with nonsingular M̃(0), such that
M(ϵ) = ϵk T−1 (ϵ) M̃(ϵ),(13.8)

in proximity of ϵ = 0.

Proof. We prove the lemma by induction on the number of roots in 0 of det M(ϵ). This number
corresponds to the maximum integer d ⩾ 0 such that ϵd is a divisor of det M(ϵ).
Base case. When d = 0, det M(0) ̸= 0, and therefore M(0) is nonsingular. The result holds
trivially by letting M̃(ϵ) = M(ϵ) and T(ϵ) = I be the identity function for all ϵ.
Inductive step. Suppose the results holds for all matrices M(ϵ) whose determinants haved ⩽ d¯ − 1 roots in 0, with d¯ ⩾ 1. We will now show that the results holds when d = d¯ as well.
Since d¯ ⩾ 1, det M(0) = 0 and therefore there exists a nonzero vector v such that v ⊤ M(0) = 0.
This necessarily means that ϵ divides all entries of v ⊤ M(ϵ), and therefore ϵ−1 v ⊤ M(ϵ) is avector with polynomial entries. Let i be any index such that vi ̸= 0, and consider the new
matrix M′ (ϵ) obtained by substituting the i-th row in M(ϵ) with ϵ−1 v ⊤ M(ϵ). It is immediate
to verify that we can write this operation compactly asM′ (ϵ) =1
S(ϵ) M(ϵ).
ϵ

where S(ϵ) is a nonzero square matrix with polynomial entries. Hence,274§13.4. A PRACTICAL ALGORITHM FOR FINDING A TLP LIMIT SOLUTION

M(ϵ) = ϵ S−1 (ϵ) M′ (ϵ).

(13.9)M′ (ϵ) is a square matrix with polynomial entries not all of which are identically zero; however,the number of roots in 0 of det M′ (ϵ) is smaller than d¯ since we multiplied one of the rows by
ϵ−1 . Thus, we can apply our inductive hypothesis to M′ (ϵ) and writeM′ (ϵ) = ϵk T ′−1(ϵ) M̃′ (ϵ)

for some integer k ⩾ 0. Substituting into (13.9), we obtainM(ϵ) = ϵk+1 S−1 (ϵ)T ′−1(ϵ) M̃′ (ϵ)= ϵk+1 (T′ (ϵ) S(ϵ))−1 M̃′ (ϵ).
Since T′ (ϵ) S(ϵ) is a square matrix with polynomial entries, concluding the proof.

Example 13.2. For the TLP in Figure 13.2,
2
1 − ϵ2
|

!
!−1
!
2
ϵ
0
2
2
.
=ϵ·
·
1 + ϵ − ϵ2
−1/2 1−ϵ 1 − ϵ
{z
}
|
{z
} |
{z
}
T−1 (ϵ)

M(ϵ)

M̃(ϵ)

We observe that B(ϵ) respects the hypotheses of Lemma 13.3: its entries are not all identicallyzero since its determinant is not identically zero. Inverting Equation (13.8) in proximity of ϵ = 0,
we obtain
M−1 (ϵ) =1
M̃−1 (ϵ) T(ϵ).
ϵkNow, given a matrix or vector with polynomial entries H(ϵ), suppose that we seek to expand
M−1 (ϵ) · H(ϵ) into its Laurent series. Due to Lemma 13.3, we can rewrite this product as
M−1 (ϵ) · H(ϵ) =1M̃−1 (ϵ)(T(ϵ)H(ϵ)) ,kϵwhere the equality holds in proximity of ϵ = 0. Since M̃(ϵ) is a square matrix with polynomial
entries invertible at ϵ = 0 and T(ϵ)H(ϵ) is a vector or matrix with polynomial entries, we canleverage the machinery of Section 13.4.2 to expand M̃−1 (ϵ) · (T(ϵ)H(ϵ)) into its Taylor series
around ϵ = 0. Multiplying this power series by ϵ−k gives a Laurent series for M−1 (ϵ)H(ϵ) at ϵ = 0.
The above shows how to deal with a singular basis matrix. The rest of the algorithm remains
unchanged.

§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
275

13.4.4

Limit of strategy

Together, Sections 13.4.2 and 13.4.3 show that, for every TLP, there exists a polynomial-timebasis-stability oracle. Finally, we deal with the last piece of the algorithm, which is the computationof the limit of optimal solutions limϵ→0+ x(ϵ) = limϵ→0+ B−1 (ϵ)b(ϵ). This task is easy after having
computed the Laurent series expansion of x(ϵ) around ϵ = 0 (see Sections 13.4.2 and 13.4.3).
Theorem 13.1 guarantees that all coefficients of the monomials of negative degree are zero vectors,
so we conclude that computing limϵ→0+ x(ϵ) simply amounts to returning the coefficient of theterm of degree 0 in the expansion.

13.5

Experimental evaluation

We evaluate the algorithm presented in this chapter by computing EFPEs, QPEs, and one-sided
QPEs. The algorithm is single-threaded, was implemented in C++, and was run on a machinewith 32GB of RAM and an Intel processor running at a nominal speed of 2.4GHz per core.
As mentioned in Section 13.4, the algorithm computes, as an intermediate step at every
iteration, an optimal basis of each trembling linear program where the perturbation magnitude ϵhas been set to a numerical value ϵ∗ . We start from the value ϵ∗ = 10−3 and use Gurobi to solve thelinear program. After the first iteration, if the basis is not stable, we halve the magnitude of ϵ∗ andre-solve the linear program using the previous basis as warm start. The process continues untilϵ∗ < 10−6 . If the basis is still not stable, we switch to a modification of Google’s open-source linear
programming solver (GLOP), which we modified so as to use arbitatry-digit precision floating
point numbers via GNU’s MPFR library (Fousse, Hanrot, Lefèvre, Pélissier, and Zimmermann,2007). From there onward, if the basis is not stable we reduce the value of ϵ∗ by a factor 1000 andsolve again with our modified version of GLOP, until a stable basis is found or ϵ∗ < 10−30 . In thelatter case, our implementation was set up to employ a rational-precision implementation (thatis, one that represents all numbers as ratios of integers to achieve an exact “infinite-precision”
solution) of the simplex algorithm, but that case was never hit in practice. The basis stability oracle
is implemented using rational precision, using GNU’s GMP library (Granlund and the GMP
development team, 2012). Therefore, our answer is exact (i.e., infinite-precision) even though theintermediate steps are not.

13.5.1

Experiments on small and medium-sized benchmark games

We experimentally evaluate the performance of our practical algorithm against the following
three algorithms in small and medium-sized benchmark games. A description of the games, as
usual, is available in Appendix A.
Exact Nash equilibrium solver (‘Simplex’) via an implementation of the simplex method using

276§13.5. EXPERIMENTAL EVALUATION

arbitrary-precision arithmetics implemented using the GMP library. We warm start the LP
oracle with a Nash equilibrium found by an LP oracle that uses finite-precision arithmetics.
NPP-based naïve solver (‘NPP’, Algorithm 13.1) using an infinite-precision LP oracle; to improve the efficiency, we warm start the LP oracle with a NE found using an LP oracle withfinite-precision arithmetics.
Symbolic-simplex QPE solver (‘M&S’) proposed by Miltersen and Sørensen (2010) to find a
QPE. It is a modified simplex algorithm, where some entries are kept as polynomials. We
implemented the algorithm as described in the original paper. However, we modified thepivoting rule from the suggested one (pick any nonbasic variable with positive reducedcost) to the greedy one (pick any nonbasic variable with maximum reduced cost). This
greatly improved run time. This method does not apply to EFPE.
Experimental results for QPE and EFPE are given in Table 13.1.
Nash

QPE

EFPE

Simplex
Time

M&S
Time

NPP
Time

This chapter
Time
Final ϵNPPTimeThis chapter
Time
Final ϵ

K23

1ms

78ms

28ms

11ms

1/819200

14ms

26ms

1/819200

L2232
L2252
L2282
L2292

59ms
372ms
3.35s
4.90s

> 6h
> 6h
> 6h
> 6h

1.56s
27.81m
> 6h
> 6h

1.15s
494ms
1s
1s

10−6
10−6
10−6
10−6

> 6h
> 6h
> 6h
> 6h

535ms
1.6s
15s
15s

1/819200
10−6
10−6
10−6

G23
G24

33ms
1.01m

21.64m
> 6h

2.88s
> 6h

203ms
19s

1/819200
1/819200

1.93m
> 6h

489ms
54s

1/819200
10−6

Game

Table 13.1: Comparison between different methods of computing Nash equilibrium refinements in
two-player zero-sum small and medium-sized benchmark games benchmark games.

The largest poker games solvable by the NPP method within 6 hours is L2252 – Leduc poker
with 5 ranks, for QPE. The NPP solver is significantly slower than the NE solver. This is because
1) it requires a larger number of pivoting steps, and 2) each pivoting step has a higher cost. Unlike
the exact NE computation, additional pivoting steps are needed by the rational simplex to find a
QPE or an EFPE, even after warm starting from a Nash equilibrium. These extra pivoting steps
need to manipulate extremely small constants due to the values of ϵ, hence introducing a large
overhead. For instance, already in L2252 the order of magnitude of the ϵ used for QPE is 10−5883 .
In the QPE case, these expensive numerical values only appear in the objective function and in
the right-hand-side constants. In the EFPE case, they appear in the constraint matrix, and there§13. COMPUTING EXACT TREMBLING-HAND REFINEMENTS IN TWO-PLAYER ZERO-SUM GAMESAT SCALE
277

are qualitatively more of them, making the computation even slower. Accordingly, the EFPE NPP
solver scales very poorly.
The M&S solver only applies to QPEs. Empirically, it is significantly slower than the NPP
solver. The reason is two-fold. First, the method is harder to warm start, as the initial basic
solution has to be feasible for all sufficiently small ϵ > 0. We initialize the method according to thesuggestion of the authors, but this initial vertex is empirically farther away from the optimal one
than a NE, which we use to warm start NPP solvers. Second, the pivoting step is more expensive,as the min-ratio test is substituted with a more sophisticated lexicographic test on polynomial
coefficients.
Our solver represents a dramatic improvement over the prior state-of-the-art algorithms. It
finds a QPE/EFPE in few minutes even on the largest game instances. This is a reduction in
runtime by 3–4 orders of magnitude. This breakthrough is mainly due to the fact that, in practice,terminates with an ϵ that is drastically larger than that required by the NPP algorithms.

13.5.2

Experiments on real-world poker endgames

We use our scalable technique to compute sequentially-rational equilibria in real poker endgames
that were encountered as part of the “Brains vs AI” competition played by the Libratus AI (Brown
and Sandholm, 2017c). To increase the scalability of our solver, we employed the Kroneckerproduct-based sparsification technique described in Farina and Sandholm (2022) to bring down
the number of nonzeros of the payoff matrix when solving the linear program at each iteration(see Section 13.2.4). The unsparsified and sparsified dimensions of each endgame are listed inTable 13.2. We remark that unlike other papers (for example, Brown and Sandholm, 2019), we do
not use a simplified abstraction, and rather use the full-sized abstraction used by Libratus.
Game

Decision nodes
Pl. 1
Pl. 2

REL26
REL27
REL28
REL22
REL25

26 55020 80133 93049 470120 744

26 46021 29733 50149 674
121 446

Sequences
Pl. 1
Pl. 2
64 60661 06285 261
146 471360 169

77 61762 51898 786147 075
362 263

Terminal
nodes

Sparsified
NNZ

71 270 327
75 928 256
111 580 420
185 831 684
494 214 830

271 364
279 902395 700
677 886
1 660 170

Table 13.2: Unsparsified and sparsified size of River endgames encountered by Libratus during the“Brains vs AI” competition.

Runtimes for each of the solution concepts are given in Table 13.3. We observe that ouralgorithm is able to compute an exact refinement in all of the game instances. This would havenot been possible for any of the other algorithms known for the problem. Our method pushes
the boundary of what refinement technology can achieve today by several orders of magnitude,

278§13.5. EXPERIMENTAL EVALUATION

hopefully adding a crucial step towards the investigation and adoption of sequentially-rational
equilibrium refinements, which so far had only remained a theoretical remedy to a serious
drawback of the Nash equilibrium, in practice.
Time
Gurobi

GLOP

Stability

Final ϵ

Total

MPFR
precision

Player 1
Player 2

36s
36s

35m 43s
43m 23s

6m 50s
9m 45s

10−15
10−15

43m 11s
53m 45s

200 digits
200 digits

QPE
QPE

Player 1
Player 2

1m 8s
21s

7m 50s12m 44s

25s
19s

10−15
10−12

9m 24s
13m 26s

100 digits
100 digits

OS-QPEOS-QPE

Player 1
Player 2

20s
28s

16m 15s
2m 24s

20s
21s

10−9
10−12

16m 56s
3m 14s50 digits
50 digits

EFPE
EFPE

Player 1
Player 2

46s
3m 37s

1h 6m
1h 10m

5m 12s
9m 12s

10−12
10−12

1h 12m1h 22m200 digits
200 digits

QPE
QPE

Player 1
Player 2

14s
24s

7m 25s4m 22s

20s
19s

10−9
10−9

8m 0s
5m 5s

100 digits
100 digits

OS-QPEOS-QPE

Player 1
Player 2

17s
33s

11m 27s
2m 47s

20s
21s

10−9
10−12

12m 5s
3m 42s50 digits
50 digits

EFPE
EFPE

Player 1
Player 2

2m 52s
4m 28s

1h 3m
1h 52m

58m 7s
29m 52s

10−15
10−15

2h 4m
2h 26m200 digits
200 digits

QPE
QPE

Player 1
Player 2

1m 7s
1m 27s

16m 32s
31m 3s

1m 0s
41s

10−15
10−15

18m 41s
33m 12s

100 digits
100 digits

OS-QPEOS-QPE

Player 1
Player 2

4m 36s2m 26s

14m 20s
14m 31s

1m 1s
38s

10−15
10−12

19m 59s
17m 37s

100 digits
100 digits

EFPE
EFPE

Player 1
Player 2

33m 24s
1h 7m

3h 58m4h 5m

2h 48m
1h 0m

10−15
10−18

7h 20m6h 13m200 digits
200 digits

QPE
QPE

Player 1
Player 2

8m 7s
19m 22s

1h 43m1h 19m

1m 45s
1m 44s

10−12
10−12

1h 53m1h 40m100 digits
100 digits

OS-QPEOS-QPE

Player 1
Player 2

10m 11s
17m 55s

37m 21s
48m 19s

2m 19s
1m 31s

10−12
10−12

49m 51s
1h 7m

100 digits
100 digits

EFPE
EFPE

Player 1
Player 2

—
—

—
—

—
—

—
—

> 72h
> 72h

—
—

QPE
QPE

Player 1
Player 2

39m 37s
1h 16m13h 28m
8h 7m

4m 40s
3m 42s

10−12
10−12

14h 13m
9h 27m200 digits
200 digits

OS-QPEOS-QPE

Player 1
Player 2

23m 40s
53m 7s2h 13m
3h 5m

4m 24s
3m 54s10−910−122h 42m4h 2m200 digits
200 digits

Game

Refinement

REL26

EFPE
EFPE

REL27

REL28

REL22

REL25

Table 13.3: Computation of refined Nash equilibria in real poker endgames using our algorithm.

Chapter 14

Quantal response and regularization
towards human play
All the solution concepts and algorithms we considered so far in this dissertation—Nashequilibrium, coarse correlated equilibrium, correlated equilibrium, team maxmin equilibriumwith or without correlation, and trembling-hand refinements such as extensive-form perfectand quasi-perfect equilibria—implicitly assume a model of game-theoretic rationality in which
players are perfectly utility-maximizing. For example, when faced with a choice between two
actions, one of which produces an expected utility of 1.0, and the other an expected utility of
1.0001, rationality of the player would require that the player pick the latter action 100% of thetimes, and solution concepts learnt via self-play of learning dynamics would evolve to expect(or predict) such a behavior from each player, despite it is arguable that most human decision
makers would instead be rather indifferent between the two actions.
The inability to take into account a model of human play in the computation of gametheoretic behavior is a serious problem for human-computer interactions that involve elementsof cooperation rather than purely competition. In such settings, modeling the other humanparticipants accurately is important for success. For example, it is important for a self-driving
car at a four-way stop sign to conform to existing human conventions rather than an arbitrary
convention derived from self-play (Lerer and Peysakhovich, 2019).

14.1

Contributions and related work

In this chapter, we propose a technique for anchoring no-external-regret dynamics to a given
model of human play. At its core, our approach works by regularizing the utility that eachplayer receives in the game by including a KL term that penalizes selecting strategies that are
far away from the anchor. In the special case in which the anchor strategy is uniformly random,

280§14.1. CONTRIBUTIONS AND RELATED WORK

our approach recovers logit quantal response equilibrium (McKelvey and Palfrey, 1995), whichmodels players that are not perfectly utility-maximizing but rather pick actions with very similar
utilities with roughly equal probability. After developing foundations for how this approach
can be used in imperfect-information extensive-form games, and proposing efficient learningalgorithms, we show that our approach yields strong human-compatible strategies in practice.
In a 200-game no-press Diplomacy tournament involving 62 human participants spanning skill
levels from beginner to expert, two AI bots trained with our algorithm both achieved a higher
average score than all other participants who played more than two games, and ranked first andthird according to an Elo ratings model. We also remark that the same methodology was recently
used in building Cicero, a bot for playing the full version of Diplomacy using natural language
communication, which was recently featured on Science (Bakhtin, Brown, Dinan, Farina, Flaherty,
Fried, Goff, Gray, Hu, Jacob, Komeili, Konath, Kwon, Lerer, Lewis, Alexander H. Miller, Mitts,
Renduchintala, Roller, Rowe, Shi, Spisak, A. Wei, D. Wu, H. Zhang, and Zĳlstra, 2022).
Alternative approaches to human modeling Behavioral cloning (BC) is the standard approach formodeling human behaviors given data. Behavioral cloning learns a strategy that maximizes thelikelihood of the human data by gradient descent on a cross-entropy loss. However, as observed
and discussed in Jacob, David J. Wu, Farina, Lerer, Hu, Bakhtin, Andreas, and Brown, 2022, BC
often falls short of accurately modeling or matching human-level performance, with BC models
underperforming the human players they are trained to imitate in games such as Chess, Go, andDiplomacy. Intuitively, it might seem that initializing self-play with an imitation-learned strategy
would result in an agent that is both strong and human-like. Indeed, Bakhtin, D. Wu, Lerer,
and Brown, 2021 showed improved performance against human-like agents when initializing
the training procedure from a human imitation strategy and value, rather than starting fromscratch. However, such an approach still results in strategies that deviate from human-compatible
equilibria, as shown in an appendix of Bakhtin, David J Wu, Lerer, Gray, Jacob, Farina, Alexander
H Miller, and Brown (2023).
Prior work on Diplomacy As mentioned, we evaluate our imitation-anchored learning dynamics
in the game of Diplomacy. After briefly describing that game specifically, we recall priorapproaches that were investigated for the game, and how they relate to ours.
Diplomacy is a benchmark 7-player mixed cooperative/competitive game featuring simultaneous moves and a heavy emphasis on negotiation and coordination. In the no-press variant of
the game, there is no cheap talk communication. Instead, players only implicitly communicate
through moves. In the full version of the game, every pair of players has a private communication
channel they can use to coordinate in natural language. In the game, seven players compete formajority control of 34 “supply centers” (SCs) on a map. On each turn, players simultaneously
choose actions consisting of an order for each of their units to hold, move, support or convoy

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

281

another unit. If no player controls a majority of SCs and all remaining players agree to a draw
or a turn limit is reached then the game ends in a draw. In this case, we use a common scoringP
system in which the score of Player i is Ci2 / i′ Ci2′ , where Ci is the number of SCs Player i owns.
Most recent successes in no-press Diplomacy use deep learning to imitate human behavior
given a corpus of human games. The first Diplomacy agent to leverage deep imitation learning
was Paquette, Y. Lu, Bocco, Smith, Satya, Kummerfeld, Pineau, Singh, and Courville, 2019.
Subsequent work on no-press Diplomacy have mostly relied on a similar architecture with somemodeling improvements (Gray, Lerer, Bakhtin, and Brown, 2020; Anthony, Eccles, Tacchetti,
Kramár, Gemp, Hudson, Porcel, Lanctot, Perolat, Everett, Singh, Graepel, and Bachrach, 2020;
Bakhtin, D. Wu, Lerer, and Brown, 2021).
Gray, Lerer, Bakhtin, and Brown, 2020 proposed an agent that plays an improved strategy via
one-ply search. It uses strategy and value functions trained on human data to conduct searchusing regret minimization.
Several works explored applying self-play to compute improved strategies. Paquette, Y. Lu,
Bocco, Smith, Satya, Kummerfeld, Pineau, Singh, and Courville (2019) applied an actor-critic
approach and found that while the agent plays stronger in populations of other self-play agents, it
plays worse against a population of human-imitation agents. Anthony, Eccles, Tacchetti, Kramár,
Gemp, Hudson, Porcel, Lanctot, Perolat, Everett, Singh, Graepel, and Bachrach (2020) used a
self-play approach based on a modification of fictitious play in order to reduce drift from humanconventions. The resulting strategy is stronger than pure imitation learning in both 1vs6 and6vs1 settings but weaker than agents that use search. Most recently, Bakhtin, D. Wu, Lerer, andBrown (2021) combined one-ply search based on equilibrium computation with value iteration to
produce an agent called Double Oracle Reinforcement learning for Action exploration (DORA). DORA
achieved superhuman performance in a two-player zero-sum version of Diplomacy without
human data, but in the full 7-player game plays poorly with agents other than itself.
Jacob, David J. Wu, Farina, Lerer, Hu, Bakhtin, Andreas, and Brown (2022) showed that
regularizing inference-time search techniques can produce agents that are not only strong but canalso model human behaviour well. In no-press Diplomacy, they show that regularizing hedge(an equilibrium-finding algorithm) with a KL-divergence penalty towards a human imitation
learning strategy can match or exceed the human action prediction accuracy of imitation learning
while being substantially stronger. KL-regularization toward human behavioral strategies has
previously been proposed in various forms in single- and multiagent RL algorithms (Nair, McGrew,
Andrychowicz, Zaremba, and Abbeel, 2018; Siegel, Springenberg, Berkenkamp, Abdolmaleki,
Neunert, Lampe, Hafner, Heess, and Riedmiller, 2020; Nair, Dalal, Gupta, and Levine, 2021), andwas notably employed in AlphaStar (Vinyals, Babuschkin, Czarnecki, Mathieu, Dudzik, Chung,Choi, Powell, Ewalds, Georgiev, et al., 2019), but this has typically been used to improve sample
efficiency and aid exploration rather than to better model and coordinate with human play.
An alternative line of research has attempted to build human-compatible agents without

282§14.2. LOGIT QUANTAL RESPONSES AND KL-ANCHORED RESPONSES

relying on human data (Hu, Lerer, Peysakhovich, and Foerster, 2020; Hu, Lerer, Cui, Pineda,
D. Wu, Brown, and Foerster, 2021; Strouse, McKee, Botvinick, Hughes, and Everett, 2021). These
techniques have shown some success in simplified settings but have not been shown to becompetitive with humans in large-scale collaborative environments.
Related work on KL-regularized games Other work has studied learning dynamics in games
regularized with a KL or entropic term, and their relationship to quantal response. Ling, Fang, andKolter (2018) establishes the connection between quantal-response equilibria and dilated entropic
regularization in imperfect-information extensive-form games. The work by Farina, Kroer, andSandholm (2019a) shows that a decomposition framework similar to that of regret circuits enables
to decompose the problem of minimizing regret with respect to the regularized utilities into
each decision node, leading to a regularized version of the CFR algorithm. The work by Perolat,
Remi Munos, Lespiau, Omidshafiei, Rowland, Ortega, Burch, Anthony, Balduzzi, De Vylder,
et al. (2021) studies continuous-time learning dynamics with last-iterate convergence. The workby Sokota, D’Orazio, Kolter, Loizou, Lanctot, Mitliagkas, Brown, and Kroer (2023) studies analgorithm that is very similar to a special case of the DiL-piKL algorithm described in this chapter,
proving additional properties related to the last-iterate convergence in the non-Bayesian setting.

14.2

Logit quantal responses and KL-anchored responses

In this section we review some standard idea related to logit quantal responses, and introduce
the more general concept of KL-anchored responses. To better fix ideas, we begin by considering
nonsequential games, that is, games in which each player’s tree-form decision problem only
has a single decision node. Later on in the section, we discuss how the ideas extend to generalimperfect-information extensive-form games. As we have seen in Chapter 4, a suitable no-regret
algorithm leading to logit quantal response equilibria and KL-anchored equilibria in generalimperfect-information extensive-form games can be constructed starting from any no-regret
algorithm for the probability simplex.

14.2.1

Logit quantal responses

As mentioned in the preamble to the chapter, a standard assumption that underpins most of
the material we have seen so far is that players best-respond, that is, when faced with actions
they will tend to deterministically pick the one with the highest value, no matter how similar invalue the other actions are. So, for example, when faced with the choice between an action withexpected utility 1.0 and another action with expected utility 1.0001, it is assumed that rational
players would favor the latter 100% of the times.
The concept of a quantal responses revises this assumption by instead postulating that the283§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

probability with which actions tend to be played by agents rather follows a continuous functionof the value of the actions. Specifically, when faced with a set of actions A each of which has
expected utility u[a], a player responding using a logit quantal response (the most common typeof quantal response) picks actions according to the distribution x ∈ ∆A defined byexp{u[a]/λ},′a′ ∈A exp{u[a ]/λ}x[a] := P(14.1)

for some parameter λ ⩾ 0. When λ = 0, x converges to being a best response, that is, putting
mass 0 to any suboptimal action, no matter how small the suboptimality. For higher values of λ,x puts mass also on suboptimal actions. For example, when λ = 1, the quantal response to two
actions of expected utility 1.0 and 1.0001 puts probability ≈ 49.9975% and ≈ 50.0025% on the two
actions. When λ → ∞, the player would pick every action uniformly at random not matter its
expected utility.
While best responses can be formulated as solutions to the linear optimization problem
arg max ⟨u, x⟩,x∈∆A

it is well-known (see also Chapter 5), that logit quantal responses (14.1) arise as solutions to theentropy-regularized, strongly convex optimization problem
arg max{⟨u, x⟩ − λφent (x)},x∈∆Awhere φent (x) :=X

x[a] log x[a](14.2)a∈A

is the negative entropy function (as is standard, we define 0 log 0 = 0).
Given an n-player nonsequential game, a logit quantal response equilibrium (QRE) relative toanchoring coefficients λi for each player i ∈ [[n]] is an assignment of strategies for each player ina game, such that each player plays the logit quantal response (relative to their own anchoring
coefficient) to everyone else’s strategies. Similarly to Nash equilibrium, in two-player zero-sum
games, in a nonsequential game where the two players have action sets A1 and A2 and anchoring
coefficients λ1 and λ2 , a logit QRE can be expressed as the solution to the strongly convex-concave
saddle-point problem
max

min x⊤ U1 y − λ1 φent (x) + λ2 φent (y),x∈∆A1 y∈∆A2

where U1 is the payoff matrix for Player 1.

14.2.2

Logit quantal response as an instances of KL-anchored response

1
Given the finite set of actions A, the uniform distribution |A|∈ ∆A is such that

(14.3)

284§14.2. LOGIT QUANTAL RESPONSES AND KL-ANCHORED RESPONSES


KL x

1
|A|


= φent (x)∀ x ∈ ∆A .

Hence, we can rewrite (14.2) asarg max ⟨u, x⟩ − λKL x
x∈∆A

1
|A|


.

(14.4)

Since the KL divergence is a notion of distance, the above optimization problem can be interpreted
as interpolating between the objective of maximizing the utility of the response, while at thesame time not picking a response that is too far from the uniform distribution. To highlight theeffect that λ has on the rationality of the player’s choice, we will refer to the parameter λ as theanchoring coefficient.
We call a composite problem of the form (14.4) a KL-anchored response, as formalized next.
Definition 14.1. Given the set of actions A available to a player, an anchor strategy τ ∈ ∆A ,and a utility vector u ∈ ℝA , the τ -anchored response to utility vector u with anchoring coefficientλ > 0 is the solution to the optimization problem
arg max{⟨u, x⟩ − λKL(x ∥ τ )}.
x∈∆A

Like QRE, given an n-player nonsequential game, an anchored response equilibrium relative
to anchoring coefficients λi and anchor strategies τ i for each Player i ∈ [[n]] is an assignmentof strategies for each player in a game, such that each player plays the logit quantal response(relative to their own anchoring coefficient) to everyone else’s strategies. Similarly to (14.3), in
the two-player zero-sum case we can define a KL-anchored equilibrium as the solution to thestrongly convex-concave saddle-point problem
max

min x⊤ U1 y − λ1 KL(x ∥ τ 1 ) + λ2 KL(y ∥ τ 2 ).

x∈∆A1 y∈∆A2

14.2.3

Learning dynamics for KL-anchored equilibria

In order to use learning dynamics to learn a KL-anchored equilibrium, it is necessary to modify
the definition of regret to incorporate the penalty derived from selecting strategies that are far
from the anchors. Specifically, consider a generic player i, and let τ i be his or her anchor policy.
By introducing the regularized utility functionũi (x, u) := ⟨u, x⟩ − λKL(x ∥ τ i )∀x ∈ ∆Ai , u ∈ ℝAi .

285§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

(1)

(2)

we define KL-anchored external regret cumulated by strategies xi , xi , . . . relative to utility vectors(1)(2)
ui , ui , . . . as(T )Regi (x̂) :=

T
X

(t)

(t)

(t)

(T )

ũ(x̂, ui ) − ũ(xi , ui ),(T )RegKL := max RegKL (x̂).
x̂∈∆Ait=1

A no-KL-anchored-external-regret algorithm will be given in Section 14.3 as a special case ofan even more general notion of regret.

14.2.4 Imitation-anchored responses in imperfect-information extensive-form
games
the above construction can be extended naturally to imperfect-information extensive-form games.
In particular, Ling, Fang, and Kolter (2018) shows that in imperfect-information extensive-form
games, a logit quantal response is the solution to the optimization problem over sequence-form
strategies defined as



X X
x[ja] arg max ⟨u, x⟩ − λx[ja] logx[pj ] x∈Q j∈J a∈Aj

!




X
X
x[ja]
= arg max ⟨u, x⟩ − λ.
x[pj ]
φent
x[pj ] a∈Aj 
x∈Q 
j∈J

a∈Aj

(We remark that the convex regularizer being subtracted is an instantiation of the dilated entropy
DGF discussed in Chapter 5.) Correspondingly, the KL-anchored response is naturally generalized
as the solution to

!






X
x[ja]
τ [ja]
arg max ⟨u, x⟩ − λx[pj ]KL.
x[pj ] a∈Aj
τ [pj ] a∈Aj 
x∈Q 
j∈J

We remark that the regret circuit formalism (Chapter 4) can be extended in this case toshow that a no-KL-anchored-external-regret algorithm for the sequence-form polytope of ageneric imperfect-information extensive-form game can be constructed starting from any noKL-anchored-external-regret algorithm for probability simplexes (Farina, Kroer, and Sandholm,
2019a).

286

14.3

§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTSModeling uncertainty on the anchoring coefficients

The previous chapter lays the foundations for a solution concept that trades off utility maximizationand playing close to an anchor policy. However, a major obstacle on the path to operationalizing
KL-anchored equilibria is that in practice it is not easy to define exactly what anchoring coefficientλi the equilibrium notion should be configured with. In this section we propose a possible
solution to this problem in nonsequential games, by modeling the uncertainty via a Bayesian
game—a special type of imperfect-information extensive-form game with a strong combinatorial
structure. In particular, for each player i, we let βi be a distribution over the set Λi of anchoring
coefficients (the “types”) that we think might describe the player. In the Bayesian game, at each(t)time t each player i produces strategies xi,λi , one for each possible type λi ∈ Λi . The regularized
utility relative to each type λi is then defined asũi,λi (x, u) := ⟨u, x⟩ − λi KL(x ∥ τ i ),(14.5)and each player cares about minimizing (that is, keeping sublinear) the per-type regretmaxx̂∈∆Ai

( T
X

)
(t)
(t)
(t)
ũi,λi (x̂, ui ) − ũi,λi (xi,λi , ui )

,

t=1
(t)

no matter the sequence of utility vectors ui received as feedback. As usual (Chapter 3), the(t)utility vectors ui are the gradient of the player’s expected utility in the underlying game, where
the expectation in this case takes into account the additional fact that the type of each player issampled independently from each distribution βi .
The algorithm we propose to minimize the notion of regret just defined is called DiL-piKL
and can be seen in Algorithm 14.1.
The rest of the section will be focused on the analysis of DiL-piKL, specifically its KL-anchoredexternal regret and its last-iterate convergence properties in two-player zero-sum games. Before
delving into the details of the analysis, we summarize a few key takeaways.
DiL-piKL can be understood as a sampled form of follow-the-regularized-leader (FTRL). When(t)a player i learns using DiL-piKL, the distributions xi,λi for any type λi ∈ Λi are no-regret withrespect to the regularized utilities ũi,λi defined in (14.5). Formally:
Theorem 14.1 (Simplified version of Theorem 14.3). Let W be a bound on the maximumP
absolute value of any payoff in the game, and Qi := n1i a∈Ai log τ i (a). Then, for any playeri, type λi ∈ Λi , and number of iterations T , the regret cumulated can be upper bounded as§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

287

Algorithm 14.1: DiL-piKL algorithm, for a generic player i
Data: Ai set of actions for Player i; Λi a set of λ values to consider for Player i;
βi a belief distribution over λ values for Player i.
(0)θi ← 02 function NextStrategy(_)
3
Sample λ ∼ βi
1

4[▷ Predictions are not used by DiL-piKL ](t)

Let xi,λ ∈ ∆Ai be the policy such that((t)
xi,λ [a] ∝ exp

5
6

(t)

(t−1)

θi

[a] + λ log τ i [a]1/(η(t − 1)) + λ)∀a ∈ A(t)

sample an action ai ∼ xi,λreturn 1a(t) ∈ ∆Aii(t)function ObserveUtility(ui ∈ ℝAi )t − 1 (t−1) 1 (t)
(t)
8
θi ←
θi
+ ui
t
t
7

max

x∈∆Ai

( T
X

)
(t)
(t)
(t)
ũi,λi (x, ui ) − ũi,λi (xi,λi , ui )

t=1



2 log Tlog niW2min,Tη ++ ρi,λi ,⩽4
λiηwhere the game constant ρi,λi is defined as ρi,λi := λi (log ni + Qi ).
We remark that the result holds no matter the choice of learning rate η > 0, thus implying a
OT (log T /(T λi )) regret bound without assumptions on η other than η = Ω(1). Second, in the√cases in which λi is tiny, by choosing η = Θ(1/ T ) we recover a sublinear guarantee (of order√T ) on the regret.
In two-player zero-sum games, the logarithmic regret of Theorem 14.1 immediately implies
PT
(T )
(t)
T
that the average strategy x̄i,λi := T1 t=1 xi,λi of each player i is a C log-approximate Bayes-NashTequilibrium strategy. In fact, a strong guarantee on the last-iterate convergence of the algorithm
can be obtained too:
Theorem 14.2 (Simplified version of Theorem 14.5; Last-iterate convergence of DiL-piKL in
two-player zero-sum games). When both players in a two-player zero-sum game learn using(T )DiL-piKL for T → ∞ iterations, their strategies xi,λi converge almost surely to the unique
Bayes-Nash equilibrium (x∗i,λi ) of the regularized game defined by utilities ũi,λi , that is, thesolution to the strongly convex-concave saddle-point problemhimax min 𝔼λ1 ∼β1 𝔼λ2 ∼β2 x⊤
U
x
−
λ
KL(x
∥
τ
)
+
λ
KL(x
∥
τ
)
,
1
2,λ
1
1,λ
1
2
2,λ
2
1,λ1
2
1
2

x1,λ1 x2,λ2

288§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS

where U1 is the utility matrix of Player 1 in the game.
The last-iterate guarantee stated in Theorem 14.2 crucially relies on the strong convexity of
the regularized utilities, and conceptually belongs with related efforts in showing last-iterate
convergence of online learning methods. However, a key difficulty that sets apart Theorem 14.2 isthe fact that the learning agents observe sampled actions from the opponents, which makes theproof of the result (as well as the obtained convergence rate) different from prior approaches.

14.3.1

A technical lemma needed in the analysis

In this section we study the last-iterate convergence of DiL-piKL, establishing that in two-player
zero-sum games DiL-piKL converges to the (unique) Bayes-Nash equilibrium of the regularized
Bayesian game. As a corollary (in the case in which each player has exactly one type), we conclude
that piKL converges to the Nash equilibrium of the regularized game in two-player zero-sum
games. We start from a technical result.
(t)Lemma 14.1. Fix any player i, λi ∈ Λi , and t ⩾ 1. For all x, x′ ∈ ∆Ai , the iterates xi,λi and(t+1)

xi,λi defined in Line 4 of Algorithm 14.1 satisfy




η
(t)
(t)
(t+1)
(t)
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ) + ∇φent (xi,λi ) − ∇φent (xi,λi ), x − x′
ηλi t + 1


= 0.

(1)

Proof. If t = 1, then the results follows from direct inspection: xi,λi is the uniform strategy(1)(and so ⟨∇φent (xi,λi ), x − x′ ⟩ = 0 for any x, x′ ∈ ∆Ai , and so the statement reduces to the first(2)(1)

order optimality conditions for the problem xi,λi = arg maxx∈∆Ai {−φent (x)/η + ⟨ui , π⟩ −
(t+1)

(t)

λi KL(x ∥ τ i )}. So, we now focus on the case t ⩾ 2. The iterates xi,λi and xi,λi produced by
DiL-piKL are respectively the solutions to the optimization problem

φent (x)(t)= arg max −+ ⟨Ū i , x⟩ − λi KL(x ∥ τ i ) ,
ηt
x∈∆Ai


φent (x)(t−1)(t)
xi,λi = arg max −+ ⟨Ū i, x⟩ − λi KL(x ∥ τ i ) ,η(t − 1)
x∈∆Ai
(t+1)

xi,λi

where we let the averages utility vectors be
(t−1)

Ū i

:=

t−1
1 X t′
ui ,
t−1 ′
t =1

(t)

Ū i :=t

1 X t′
ui .
t ′
t =1

289

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

(t+1)

Since the regularizing function negative entropy φent is Legendre, the strategies xi,λi(t)and xi,λi are in the relative interior of the probability simplex, and therefore the first-order
(t+1)

(t)

optimality conditions for xi,λi and xi,λi are respectively


(t)

1
(t+1)
∇φent (xi,λi ), x − x′
ηt



1
(t)
∇φent (xi,λi ), x − x′η(t − 1)

(t+1)

−Ū i + λi ∇φent (xi,λi ) − λi ∇φent (τ i ) += 0 ∀x, x′ ∈ ∆Ai ,(14.6)



(t−1)

−Ū i

(t)

+λi ∇φent (xi,λi )−λi ∇φent (τ i )+= 0 ∀x, x′ ∈ ∆Ai .

Taking the difference between the equalities, we find


(t)

(t−1)

−Ū i + Ū i






1
1
(t+1)
(t)
∇φent (xi,λi ) − λi +∇φent (xi,λi ), x − x′ = 0+ λi +ηtη(t − 1)

We now use the fact that
(t)

(t−1)

Ū i − Ū i

=−

1
1
(t)
(t)
Ū i +
u .
t−1
t−1 i

to further write





 
1
1
1  (t)
(t)
(t+1)
(t)
′
−ui + Ū i
+ λi +∇φent (xi,λi ) − λi +∇φent (xi,λi ), x − x = 0
t−1
ηtη(t − 1)(14.7)
From Equation (14.6) we find(t)⟨Ū i , x − x′ ⟩ =



(t+1)
λi ∇φent (xi,λi ) − λi ∇φent (τ i ) +1(t+1)∇φent (xi,λi ), x − x′ηt

and so, plugging back the previous relationship in Equation (14.7) we can write, for allx, x′ ∈ ∆Ai ,




1
1
(t)
(t+1)
(t+1)
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ) + ∇φent (xi,λi )
t−1
ηt





1
1
(t+1)
(t)
′
+ λi +∇φent (xi,λi ) − λi +∇φent (xi,λi ), x − xηtη(t − 1) 
1  (t)
1
(t+1)
(t+1)
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ) + λi +∇φent (xi,λi )t−1
η(t − 1)

0=

=

290

§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS





1
(t)
′
− λi +∇φent (xi,λi ), x − xη(t − 1) ηλ t + 1
1  (t)i(t)(t+1)−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ) +∇φent (xi,λi )=t−1
η(t − 1)
ηλi t + 1(t)−
∇φent (xi,λi ), x − x′ .
η(t − 1)Dividing by (ηλi t + 1)/(η(t − 1)) yields the statement.

(t)Corollary 14.1. Fix any player i, λi ∈ Λi , and t ⩾ 1. For all x ∈ ∆Ai , the iterates xi,λi and(t+1)

xi,λi defined in Line 4 of Algorithm 14.1 satisfy
D
E
(t)
(t)
(t+1)
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), x − xi,λi=

λi t +



1 
(t)
(t+1)
(t)
(t+1)
KL(x ∥ xi,λi ) − KL(x ∥ xi,λi ) + KL(xi,λi ∥ xi,λi ) .
η

(t+1)

Proof. Since Lemma 14.1 holds for all x, x′ ∈ ∆Ai , we can in particular set x′ = xi,λi , andobtain
D
E
η
(t)
(t)
(t+1)
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), x − xi,λi
ηλi t + 1
D
E
(t+1)
(t)
(t+1)
+ ∇φent (xi,λi ) − ∇φent (xi,λi ), x − xi,λi= 0. (14.8)
Using the three-point identity
E
D
(t)
(t+1)
(t+1)
(t)
(t+1)
(t)
(t+1)
= KL(x ∥ xi,λi )−KL(x ∥ xi,λi )−KL(xi,λi ∥ xi,λi )∇φent (xi,λi )−∇φent (xi,λi ), x−xi,λi
in Equation (14.8) yields
(t+1)

(t)

(t+1)

(t)

KL(x ∥ xi,λi ) = KL(x ∥ xi,λi ) − KL(xi,λi ∥ xi,λi )
D
E
η
(t)
(t)
(t+1)
+
−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), x − xi,λi .
ηλi t + 1
Multiplying by λi t + 1/η yields the statement.

14.3.2(t)Regret analysis

Let ũi,λi be the regularized utility of agent type λi ∈ Λi

291

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

(t)

(t)

ũi,λi : ∆Ai ∋ x 7→ ⟨ui , x⟩ − λi KL(x ∥ τ i ).

Observation 14.1. We note the following:(t)• For any i ∈ {1, 2} and λi ∈ Λi , the function ũi,λi satisfies
(t)

(t)

(t)

ũi,λi (x) = ũi,λi (x′ ) + ⟨∇ũi,λi (x′ ), x − x′ ⟩ − λi KL(x ∥ x′ )∀ x, x′ ∈ ∆Ai .

• Furthermore,
(t)

(t)

(t)

−∇ũi,λi (xi,λi ) = −ui + λi ∇φent (x(t) ) − λi ∇φent (τ i ).

Using Corollary 14.1 we have the following
Lemma 14.2. For any player i and type λi ∈ Λi ,
(t)

(t)

(t)

ũi,λi (x) − ũi,λi (xi,λi ) ⩽(t)∥ui ∥2∞(t)(t+1)+ λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i )4λi t + 4/η




1
1
(t+1)
(t)
− λi t +KL(x ∥ xi,λi ) + λi (t − 1) +KL(x ∥ xi,λi ).
ηηProof. From Lemma 14.1,

0=



1 
(t)
(t+1)
(t)
(t+1)
λi t +− KL(x ∥ xi,λi ) + KL(x ∥ xi,λi ) − KL(xi,λi ∥ xi,λi )
η
(t+1)

(t)

(t)

+ ⟨−∇ũi,λi (xi,λi ), x − xi,λi ⟩



1
(t+1)
(t)
(t+1)
(t)
= λi t +− KL(x ∥ xi,λi ) + KL(x ∥ xi,λi ) − KL(xi,λi ∥ xi,λi )
η
(t)


=

λi t +

1
η



(t)

(t)

(t+1)

(t)

(t)

(t)

+ ⟨∇ũi,λi (xi,λi ), xi,λi + xi,λi ⟩ + ⟨−∇ũi,λi (xi,λi ), x − xi,λi ⟩

(t+1)
(t)
(t+1)
(t)
− KL(x ∥ xi,λi ) + KL(x ∥ xi,λi ) − KL(xi,λi ∥ xi,λi )
(t)

(t)

(t)

(t+1)

(t)

(t)

(t)

(t)

+ ⟨−∇ũi,λi (xi,λi ), xi,λi − xi,λi ⟩ − ũi,λi (x) + ũi,λi (xi,λi ) − λi KL(x ∥ xi,λi ).
Rearranging, we find




1
1
(t)
(t)
(t)
(t+1)
(t)
ũi,λi (x) − ũi,λi (xi,λi ) = − λi t +KL(x ∥ xi,λi ) + λi (t − 1) +KL(x ∥ xi,λi )
η
η

292

§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS




1
(t+1)
(t)
(t)
(t)
(t)
(t+1)
− λi t +KL(xi,λi ∥ xi,λi ) + ⟨−∇ũi,λi (xi,λi ), xi,λi − xi,λi ⟩ .
η{z}|(14.9)(14.10)
We now upper bound the term in (14.9) using convexity of the function x 7→ KL(x ∥ τ i ), as
follows:
(t)

(t)

(t)

(t+1)

⟨−∇ũi,λi (xi,λi ), xi,λi − xi,λi ⟩
(t)

(t)

(t+1)

(t)

(t+1)

(t)

= ⟨−ui , xi,λi − xi,λi ⟩ + λi ⟨∇φent (xi,λi ) − ∇φent (τ i ), xi,λi − xi,λi ⟩


(t)
(t)
(t+1)
(t+1)
(t)
⩽ ⟨−ui , xi,λi − xi,λi ⟩ + λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i ) .
Substituting the above bound into (14.10) yields


1
(t)
(t)
(t)
(t)
(t)
(t+1)
(t+1)
(t)
ũi,λi (x) − ũi,λi (xi,λi ) ⩽ ⟨−ui , xi,λi − xi,λi ⟩ − λi t +KL(xi,λi ∥ xi,λi )
η


(t+1)
(t)
+ λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i )




1
1
(t+1)
(t)
− λi t +KL(x ∥ xi,λi ) + λi (t − 1) +KL(x ∥ xi,λi )
η
η


(t)
∥ui ∥2∞
1
(t)
(t+1)
⩽
+ λi t +∥xi,λi − xi,λi ∥21
4λi t + 4/η
η


1
(t+1)
(t)
KL(xi,λi ∥ xi,λi )− λi t +
η


(t)
(t+1)
+ λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i )




1
1
(t+1)
(t)
− λi t +KL(x ∥ xi,λi ) + λi (t − 1) +KL(x ∥ xi,λi ),ηηwhere the second inequality follows from Young’s inequality. Finally, by using the strong
(t)
(t+1)
convexity of the KL divergence between points xi,λi and xi,λi , that is,
(t+1)

(t)

(t+1)

(t)

KL(xi,λi ∥ xi,λi ) ⩾ ∥xi,λi − xi,λi ∥21 ,yields the statement.
Noting that the right-hand side of Lemma 14.2 is telescopic, we immediately have thefollowing.

293§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

Theorem 14.3. For any player i and type λi ∈ Λi , and strategy x ∈ ∆Ai , the following regretbound holds at all times T :
T
X

W
(t)
(t)
(t)
ũi,λi (x) − ũi,λi (xi,λi ) ⩽2

2 log T
min
,Tη
4
λi

t=1


+

log ni+ λi (log ni + Qi ).
η

Proof. From Lemma 14.2 we have that
T
X

!
(1)
T
KL(x ∥ xi,λi )
W2 X
1
(1)
+ λi KL(xi,λi ∥ τ i ) +4 t=1 λi t + 1/η
η
!


T
1
log ni
W2 Xmin,η
+ λi (log ni + Qi ) +
⩽
4 t=1
λi t
η


W2
2 log Tlog ni⩽min,, ηT + λi (log ni + Qi ) +
4
λi
η

(t)
(t)
(t)
ũi,λi (x) − ũi,λi (xi,λi ) ⩽t=1

where the second inequality follows from the fact that λi t + 1/η ⩾ max{λi t, 1/η} and the fact
(1)that xi,λi is the uniform strategy.

14.3.3

Last-Iterate Convergence in Two-Player Zero-Sum GamesIn two-player game with payoff matrix U for Player 1, a Bayes-Nash equilibrium to the regularizedgame is a collection of strategies (x∗i,λi ) such that for any supported type λi of Player i ∈ {1, 2},
the strategy x∗i,λi is a best response to the average strategy of the opponent. In symbols,nox∗1,λ1 ∈ arg max ⟨U 𝔼λ2 ∼β2 x∗2,λ2 , x⟩ + λ1 KL(x ∥ τ 1 )∀ λ1 ∈ Λ1 ,nox∗2,λ2 ∈ arg max ⟨−U⊤ 𝔼λ1 ∼β1 x∗1,λ1 , x⟩ + λ2 KL(x ∥ τ 2 )∀ λ2 ∈ Λ2 .

x∈∆A1

x∈∆A2

h
i
h
i
Denoting x̄∗1 := 𝔼λ1 ∼β1 x∗1,λ1 , x̄∗2 := 𝔼λ2 ∼β2 x∗2,λ2 , the first-order optimality conditions forthe best response problems above are⟨U x̄∗2 + λ1 ∇φent (x∗1,λ1 ) − λ1 ∇φent (τ 1 ), x∗1,λ1 − x′1,λ1 ⟩ ⩾ 0∀ x′1,λ1 ∈ ∆A1 ,⟨−U⊤ x̄∗1 + λ2 ∇φent (x∗2,λ2 ) − λ2 ∇φent (τ 2 ), x∗2,λ2 − x′2,λ2 ⟩ ⩾ 0∀ x′2,λ2 ∈ ∆A2 .

We also mention the following standard lemma.

294§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS

Lemma 14.3. Let (x∗i,λi )i∈{1,2},λ1 ∈Λi be the unique Bayes-Nash equilibrium of the regularized
game. Let strategies x′i,λi be arbitrary, and let:hihi
• x̄′1 := 𝔼λ1 ∼β1 x′1,λ1 , x̄′2 := 𝔼λ2 ∼β2 x′2,λ2 ;• α := 𝔼λ1 ∼β1 ⟨−Ux̄′2 + λ1 ∇φent (x′1,λ1 ) − λ1 ∇φent (τ 1 ), x∗1,λ1 − x′1,λ1 ⟩ ;• β := 𝔼λ2 ∼β2 ⟨U⊤ x̄′1 + λ2 ∇φent (x′2,λ2 ) − λ2 ∇φent (τ 2 ), x∗2,λ2 − x′2,λ2 ⟩ .
Then,Xα+β ⩽−𝔼λi ∼βi λi KL(x′i,λi ∥ x∗i,λi ) + λi KL(x∗i,λi ∥ x′i,λi ) .

i∈{1,2}

The following potential function will be key in the analysis:
Ψ

(t)

X

:=


𝔼λi ∼βii∈{1,2}



1
(t)
(t)
∗
KL(xi,λi ∥ xi,λi ) + λi KL(xi,λi ∥ τ i ) ,λi (t − 1) +ηt ∈ {1, 2, . . . }.

Proposition 14.1. At all times t ∈ {1, 2, . . . }, lethix̄t−i := 𝔼λ−i ∼β−i xt−i,λ−i .
The potential Ψ(t) satisfies the inequalityΨ(t+1) ⩽ Ψ(t) +X𝔼λi ∼βi i∈{1,2}

(t)

ui

2
∞

4λi t + 4/η


D
E
(t)
(t)
(t) 
+ Ui x̄−i − ui , x∗i,λi − xi,λi .

Proof. By multiplying both sides of Corollary 14.1 for the choice x = x∗i,λi , taking expectations
over λi ∼ βi , and summing over the player i ∈ {1, 2}, we findXi∈{1,2}𝔼λi ∼βi







X
1
1
(t+1)
(t)
λi t +KL(x∗i,λi ∥ xi,λi ) =𝔼λi ∼βi λi t +KL(x∗i,λi ∥ xi,λi )ηηi∈{1,2}−Xi∈{1,2}𝔼λi ∼βi

λi t +



1
(t+1)
(t)
KL(xi,λi ∥ xi,λi )η295§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAYX+

𝔼λi ∼βi

hD

(t)

(t)

(t+1)

−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), x∗i,λi − xi,λi

Ei.

i∈{1,2}

|

{z

}

(♣)

(14.11)
We now proceed to analyze the last summation on the right-hand side. First,
X

(♣) =

𝔼λi ∼βi

hD

(t)

(t)

(t)

−Ui x̄−i + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), x∗i,λi − xi,λi

Ei

i∈{1,2}

|

{z

}

(♠)

X

+

𝔼λi ∼βi

hD

(t)

(t)

(t)

(t+1)

−ui + λi ∇φent (xi,λi ) − λi ∇φent (τ i ), xi,λi − xi,λi

Ei

i∈{1,2}

|

{z

X

+

}

(♡)

𝔼λi ∼βi

hD

(t)

(t)

(t)

Ui x̄−i − ui , x∗i,λi − xi,λi

Ei.

(14.12)i∈{1,2}Using Lemma 14.3 we can immediately write
X

(♠) ⩽

h
i
(t)
𝔼λi ∼βi −λi KL(x∗i,λi ∥ xi,λi ) .

i∈{1,2}

By manipulating the inner product in (♡), we have
(♡) =

X

h
D
Ei
(t)
(t)
(t+1)
(t)
(t+1)
(t+1)
(t)
𝔼λi ∼βi ⟨−ui , xi,λi − xi,λi ⟩ − λi ∇φent (xi,λi ) − φent (xi,λi ), xi,λi − xi,λii∈{1,2}

⩽

X

h

i
(t)
(t)
(t+1)
(t+1)
(t)
𝔼λi ∼βi ⟨−ui , xi,λi − xi,λi ⟩ + λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i )i∈{1,2}⩽X𝔼λi ∼βi i∈{1,2}

(t)

ui

2
∞

4λi t + 4/η++ λi t +

X

1
η



(t)

(t+1)

xi,λi − xi,λi

2
1




h 
i
(t+1)
(t)
𝔼λi ∼βi λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i ) ,i∈{1,2}where the last inequality follow from the fact that ab ⩽ a2 /(4ρ) + ρb2 for all choices of a, b ⩾ 0
and ρ > 0. Substituting the individual bounds into (14.12) yields(♣) ⩽X
i∈{1,2}

h 
i
(t+1)
(t)
𝔼λi ∼βi λi KL(xi,λi ∥ τ i ) − KL(xi,λi ∥ τ i )296§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS


X

+


𝔼λi ∼βi i∈{1,2}X+

𝔼λi ∼βi

2

(t)

ui




∞

4λi t + 4/η

hD

(t)

+ λi t +
(t)

1
η



(t)

(t+1)

xi,λi − xi,λi(t)

Ui x̄−i − ui , x∗i,λi − xi,λi

Ei

2
1




.

i∈{1,2}

Finally, plugging the above bound into (14.11) and rearranging terms yields
Ψ

(t+1)

⩽Ψ

(t)



 
1
(t+1)
(t)
KL(xi,λi ∥ xi,λi )+𝔼λi ∼βi − λi t +ηi∈{1,2}


2
(t)


ui
X
2
1


(t)
(t+1)
∞
+
𝔼λi ∼βi + λi t +xi,λi − xi,λi

4λi t + 4/ηη1Xi∈{1,2}X+

𝔼λi ∼βi

hD

(t)

(t)

(t)

Ui x̄−i − ui , x∗i,λi − xi,λi

Ei.

i∈{1,2}

⩽Ψ

(t)

 


2
1
(t+1)
(t)
+
𝔼λi ∼βi − λi t +xi,λi − xi,λiη1
i∈{1,2}


2
(t)


ui
X
2
1


(t)
(t+1)
∞
+
𝔼λi ∼βi + λi t +xi,λi − xi,λi

4λi t + 4/ηη1Xi∈{1,2}X+

𝔼λi ∼βi

hD

(t)

(t)

(t)

Ui x̄−i − ui , x∗i,λi − xi,λi

Ei.

i∈{1,2}⩽ Ψ(t) +X
𝔼λi ∼βi i∈{1,2}

(t)

ui

2
∞

4λi t + 4/η


D
E
(t)
(t)
(t) 
+ Ui x̄−i − ui , x∗i,λi − xi,λi ,

as we wanted to show.

Theorem 14.4. As in Proposition 14.1, lethix̄t−i := 𝔼λ−i ∼β−i xt−i,λ−i .
TLet D̃KL
be the notion of distance defined as
T
:=
D̃KL

X
i∈{1,2}hi(T )𝔼λi ∼βi (λi + 1/(η(T − 1)))KL(x∗i,λi ∥ xi,λi ) .

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

297

At all times T = 2, 3, . . . ,

T
D̃KL
⩽

log ni
1
W
ρ+
+
T
η
2

2

X
i∈{1,2}
2 log T𝔼λi ∼βi min, ηT 
λi
+

T
hD
Ei
2X X
(t)
(t)
(t)
𝔼λi ∼βi Ui x̄−i − ui , x∗i,λi − xi,λi ,T t=1
i∈{1,2}whereX

ρ := 2𝔼λi ∼βi [λi ] (log ni + Qi ).

i∈{1,2}

Proof. Using the bound on Ψ(t+1) − Ψ(t) given by Proposition 14.1 we obtain
Ψ(T ) − Ψ(1) =T−1X(Ψ(t+1) − Ψ(t) )

t=1

⩽

T
−1
X


X

t=1 i∈{1,2}
𝔼λi ∼βi 

(t)

ui

2
∞

4λi t + 4/η


E
(t)
(t)
(t) 
+ Ui x̄−i − ui , x∗i,λi − xi,λi 
D


2 
(t)
T
T
hD
Ei
u
X
X
i
1 X
X
(t)
(t)
(t)
∞ 
=
𝔼λi ∼βi 
𝔼λi ∼βi Ui x̄−i − ui , x∗i,λi − xi,λi
+4
λ t + 1/ηt=1 i
t=1i∈{1,2}i∈{1,2}⩽1 X
𝔼λi ∼βi
4i∈{1,2}

"T −1
X

#

T
hD
Ei
X
X
W2
(t)
(t)
(t)
+
𝔼λi ∼βi Ui x̄−i − ui , x∗i,λi − xi,λi .
λ t + 1/ηt=1 i
t=1i∈{1,2}

We can now bound
T
X



T
X
W2
1
2
⩽W
min
,η
λ t + 1/η
λi t
t=1 i
t=1
( T
)
T
X 1 X
2
⩽ W min,η
λt
t=1 i t=1

2 log T
⩽ W 2 min,Tη .
λi

On the other hand, note that298

§14.3. MODELING UNCERTAINTY ON THE ANCHORING COEFFICIENTS

Ψ

(T )

−Ψ

(1)



1
(T )
(T )
∗
= −Ψ +
𝔼λi ∼βi λi (T − 1) +KL(xi,λi ∥ xi,λi ) + λi KL(xi,λi ∥ τ i )η
i∈{1,2}hiX(T )⩾ −Ψ(1) +(T − 1)𝔼λi ∼βi (λi + 1/(η(T − 1)))KL(x∗i,λi ∥ xi,λi )
(1)



X

i∈{1,2}"T= (T − 1)D̃KL−

X

𝔼λi ∼βiηi∈{1,2}T⩾ (T − 1)D̃KL−

X
i∈{1,2}𝔼λi ∼βi(1)KL(x∗i,λi ∥ xi,λi )#(1)− λi KL(xi,λi ∥ τ i )
log ni+ λi (log ni + Qi )ηT
= (T − 1)D̃KL− ρ,

where the last inequality follows from expanding the definition of the KL divergence and using(1)
the fact that xi,λi is the uniform strategy. Combining the inequalities and dividing by T − 1yields
W
T
D̃KL
⩽

4

2

X
i∈{1,2}ρ
2 log TT
𝔼λi ∼βi minη+,(T − 1)λi T − 1
T −1
+

T
Ei
hD
1 X X
(t)
(t)
(t)
𝔼λi ∼βi Ui x̄−i − ui , x∗i,λi − xi,λi .
T − 1 t=1i∈{1,2}

Finally, using the fact that 2(T − 1) ⩾ T yields the statement.

Theorem 14.5 (Last-iterate convergence of DiL-piKL in two-player zero-sum games). Let ρ
be as in the statement of Theorem 14.4. When both players in a zero-sum game learn using
DiL-piKL for T iterations, their strategies converge to the unique Bayes-Nash equilibrium(x∗1 , x∗2 ) of the regularized game defined by utilities (14.5), in the following senses:(a) In expectation: for all i ∈ {1, 2} and λi ∈ Λi , at a rate of roughly log T /(λi T )





i
2 X
1 
log niW
2 log T(T )𝔼 KL(x∗i,λi ∥ xi,λi ) ⩽ρ++𝔼λj ∼βj min, ηT .
λi T
η
2
λj
h

j∈{1,2}√√(We remark that for η = 1/ T the convergence is never slower than 1/ T ).
√(b) With high probability, at a rate of roughly 1/ T : for any δ ∈ (0, 1) and Player i ∈ {1, 2},299§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

"
ℙ

h

(T )
(T )
∀ λi ∈ Λi : KL(x∗i,λi ∥ xi,λi ) ⩽ 𝔼 KL(x∗i,λi ∥ xi,λi )

i

#
r
√
8 2W
|Λi |
√
+
log
⩾ 1 − δ.
δ
λi T

h
i
(T )
A n upper bound on 𝔼 KL(x∗i,λi ∥ xi,λi ) was given in the previous point.
(c) Almost surely in the limit:
h
i
T →+∞
(T )
ℙ ∀ λi ∈ Λi : KL(x∗i,λi ∥ xi,λi ) −−−−−→ 0 = 1∀i ∈ {1, 2}.

Proof. We prove the three statements incrementally.
(t′ )(a) Let Ft be the σ-algebra generated by {ui : t′ = 1, . . . , t − 1, i ∈ {1, 2}}. We let(t)𝔼t [ · ] := 𝔼[ · | Ft ]. Since piKL is a deterministic algorithm, xi,λi is Ft -measurable. Hence,(t)

given that uti is an unbiased estimator of Ui x̄−i we have that at all times t
𝔼thD
Ei D h
i
E
(t)
(t)
(t)
(t)
(t)
(t)
Ui x̄−i − ui , x∗i,λi − xi,λi = 𝔼t Ui x̄−i − ui , x∗i,λi − xi,λi = 0.

(14.13)T
Note that from the definition of D̃KL
given in Theorem 14.4
(T )KL(x∗i,λi ∥ xi,λi ) ⩽

1 T
D̃ .
λi KL

(14.14)Hence, taking expectations and using (14.13) yields the statement.
(b) To prove high-probability convergence, we use the Azuma-Hoeffding concentration
inequality. In particular, (14.13) shows that the stochastic process



h
i
(t)
(t)
(t)
𝔼λj ∼βj ⟨Uj x̄−j − uj , x∗j − xj ⟩ Xj∈{1,2}t=1,2,...

is a martingale difference sequence adapted to the filtration Ft . Furthermore, note that
X

h
i
(t)
(t)
(t)
𝔼λj ∼βj ⟨Uj x̄−j − uj , x∗j,λj − xj,λj ⟩ ⩽ 4W

j∈{1,2}for all t. Hence, using the Azuma-Hoeffding inequality for martingale difference
sequences we obtain that for all δ ∈ (0, 1),

300§14.4. EXPERIMENTAL EVALUATION IN NO-PRESS DIPLOMACY



r
T
h
i
X
X
1
(t)
(t)
(t)
ℙ
𝔼λj ∼βj ⟨Uj x̄−j − uj , x∗j − xj ⟩ ⩽ 4W 2T log  ⩾ 1 − δ.
δ
t=1j∈{1,2}Plugging the above probability bound in the statement of Theorem 14.4 and using theunion bound over λi ∈ Λi yields the statement.
(c) follows from (b) via a standard application of the Borel-Cantelli lemma.

14.4

Experimental evaluation in no-press Diplomacy

We empirically investigate using DiL-piKL as the planning algorithm used at each decision-point
within the DORA framework, which we now recall.

14.4.1 Background on Double Oracle Reinforcement learning for Action
exploration (DORA)
DORA (Bakhtin, D. Wu, Lerer, and Brown, 2021) is an algorithm similar to past model-based
reinforcement-learning methods such as AlphaZero (Silver, Hubert, Schrittwieser, Antonoglou,Lai, Guez, Lanctot, Sifre, Kumaran, Graepel, et al., 2018), except that in place of Monte Carlo tree
search—which is unsound in simultaneous-action games such as Diplomacy or other imperfectinformation extensive-form games—it instead uses an equilibrium-finding algorithm such ashedge or RM to iteratively approximate a Nash equilibrium for the current state (i.e., one-step
lookahead search). A deep neural net trained to predict the strategy is used to sample plausible
actions for all players to reduce the large action space in Diplomacy down to a tractable subset for
the equilibrium-finding procedure, and a deep neural net trained to predict state values is used
to evaluate the results of joint actions sampled by this procedure. Beginning with a strategy andvalue network randomly initialized from scratch, a large number of self-play games are played
and the resulting equilibrium strategies and the improved 1-step value estimates computed on
every turn from equilibrium-finding are added to a replay buffer used for subsequently improving
the strategy and value. Additionally, a double-oracle (McMahan, G. Gordon, and Blum, 2003)
method was used to allow the strategy to explore and discover additional actions, and the sameequilibrium-finding procedure was also used at test time.

14.4.2Training of our bot Diplodocus

Our training algorithm closely follows that of DORA, described above. However, the key
difference is that in place of RM to compute the equilibrium strategy σ on each turn of a game§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

301

during self-play, we use DiL-piKL with a λ distribution and human imitation anchor strategy τthat is fixed for all of training. We call this self-play algorithm RL-DiL-piKL. We use RL-DiL-piKL
to train value and strategy proposal networks and use DiL-piKL during test-time search. The call
final trained agent is called Diplodocus.
Following Bakhtin, D. Wu, Lerer, and Brown, 2021, at evaluation time we perform 1-ply
lookahead where on each turn we sample up to 30 of the most likely actions for each player fromthe RL strategy proposal network. However, rather than using RM to compute the equilibrium σ,
we apply DiL-piKL.
As also mentioned previously in Section 14.3, while our agent samples λi from the probability
distribution βi when computing the DiL-piKL equilibrium, the agent chooses its own action toactually play using a fixed low λ. For all experiments, including all ablations, the agent uses thesame BC anchor strategy. For DiL-piKL experiments for each player i we set βi to be uniform over
{10−4 , 10−3 , 10−2 , 10−1 } and play according to λ = 10−4 , except for the first turn of the game. On
the first turn we instead sample from {10−2 , 10−1.5 , 10−1 , 10−0.5 } and play according to λ = 10−2 ,so that the agent plays more diverse openings, which more closely resemble those that humansplay.

14.4.3

Experimental setup

We compare the performance of two variants of Diplodocus both in a population of prior agentsand other baseline agents, and in a tournament with humans.
In order to measure the ability of agents to play well against a diverse set of opponents, weplay many games between AI agents where each of the seven players are sampled randomly froma population of baselines or the agent to be tested. We report scores for each of the following
algorithms against the baseline population.
• Diplodocus-Low and Diplodocus-High are the proposed agents that use RL-DiL-piKL
during training with player types {10−4 , 10−1 } and {10−2 , 10−1 }, respectively.
• DORA is an agent that is trained via self-play and uses RM as the search algorithm during
training and test-time. Both the strategy and the value function are randomly initialized at
the start of training.
• DNVI is similar to DORA, but the strategy proposal and value networks are initialized fromhuman behavioral-cloning pretraining.
• DNVI-NPU is similar to DNVI, but during training only the RL value network is updated.
The strategy proposal network is trained but never fed back to self-play workers, to limit
drift from human conventions. The trained strategy proposal is only used at test time (alongwith the RL value network).

302§14.4. EXPERIMENTAL EVALUATION IN NO-PRESS DIPLOMACY• BRBot is an approximate best response to the behavioral-cloning strategy. It was trained thesame as Diplodocus, except that during training the agent plays one distinguished player
each game with λ = 0 while all other players use λ ≈ ∞.
• SearchBot is a one-step lookahead equilibrium search agent from the paper by Gray, Lerer,
Bakhtin, and Brown (2020), evaluated using their published model.
• HedgeBot is an agent similar to SearchBot (Gray, Lerer, Bakhtin, and Brown, 2020) but using
our latest architecture and using hedge rather than RM as the equilibrium-finding algorithm.
• FPPI-2 and SL are two agents from the work by Anthony, Eccles, Tacchetti, Kramár, Gemp,
Hudson, Porcel, Lanctot, Perolat, Everett, Singh, Graepel, and Bachrach (2020), evaluatedusing their published model.[14.a]

14.4.4

Performance compared to prior algorithms

We report results for Diplodocus against prior algorithms for the problem. The results, shown in
Table 14.1, show Diplodocus-Low and Diplodocus-High perform the best by a wide margin.

Agent
Diplodocus-Low
Diplodocus-High
DNVI-NPU (retrained)
BRBot
DNVI (retrained)
HedgeBot (retrained)

Reference

Bakhtin, D. Wu, Lerer, and Brown, 2021

Bakhtin, D. Wu, Lerer, and Brown, 2021
Jacob, David J. Wu, Farina, Lerer, Hu, Bakhtin, Andreas,and Brown, 2022

Score against
population
29% ± 1%28% ± 1%20% ± 1%18% ± 1%15% ± 1%14% ± 1%

DORA (retrained)

Bakhtin, D. Wu, Lerer, and Brown, 2021

13% ± 1%FPPI-2

Anthony, Eccles, Tacchetti, Kramár, Gemp, Hudson,
Porcel, Lanctot, Perolat, Everett, Singh, Graepel, andBachrach, 2020

9% ± 1%

SearchBot
SL

Gray, Lerer, Bakhtin, and Brown, 2020
Anthony, Eccles, Tacchetti, Kramár, Gemp, Hudson,
Porcel, Lanctot, Perolat, Everett, Singh, Graepel, andBachrach, 20207% ± 1%6% ± 1%

Table 14.1: Performance of different algorithms. Agents above the line were retrained. Agents below
the line were evaluated using the models and the parameters provided by the authors. The ±shows one standard error.

[14.a] https://github.com/deepmind/diplomacy

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

14.4.5

303

Experiments against human players

we organized a tournament where we evaluated four agents for 50 games each in a population of
online human participants. We evaluated two baseline agents, BRBot and DORA, and our new
agents, Diplodocus-Low and Diplodocus-High.
In order to limit the duration of games to only a few hours, these games used a time limit of 5
minutes per turn and a stochastic game-end rule where at the beginning of each game year between
1909 and 1912 the game ends immediately with 20% chance per year, increasing in 1913 to a 40%
chance. Players were not told which turn the game would end on for a specific game, but were told
the distribution it was sampled from. Our agents were also trained based on this distribution.[14.b]Players were recruited from Diplomacy mailing lists and from webdiplomacy.net. In order tomitigate the risk of cheating by collusion, players were paid hourly rather than based on in-gameperformance. Each game had exactly one agent and six humans. The players were informed that
there was an AI agent in each game, but did not know which player was the bot in each particulargame. In total 62 human participants played 200 games with 44 human participants playing morethan two games and 39 human participants playing at least 5 games.
We report results for the human tournament in Table 14.2. For each listed player, we reporttheir average score, Elo rating, and rank within the tournament based on Elo among players
who played at least 5 games. Elo ratings were computed using a standard generalization ofBayesElo (Coulom, 2005) to multiple players (Hunter, 2004).This gives similar rankings as averagescore, but also attempts to correct for both the average strength of the opponents, since somegames may have stronger or weaker opposition, as well as for which of the seven European
powers a player was assigned in each game, since some starting positions in Diplomacy areadvantaged over others. To regularize the model, a weak Bayesian prior was applied such thateach player’s rating was normally distributed around 0 with a standard deviation of around 350
Elo.
The results show that Diplodocus-High performed best among all the humans by both Elo andaverage score. Diplodocus-Low followed closely behind, ranking second according to averagescore and third by Elo. BRBot performed relatively well, but ended ranked below that of both
DiL-piKL agents and several humans. DORA performed relatively poorly.
Two participants achieved a higher average score than the Diplodocus agents, a player
averaging 35% but who only played two games, and a player with a score of 29% who played
only one game.
We note that given the large statistical error margins, the results in Table 14.2 do not conclusively
demonstrate that Diplodocus outperforms the best human players, nor do they alone demonstrate
an unambiguous separation between Diplodocus and BRBot. However, the results do indicate[14.b] Games were run by a third-party contractor. In contradiction of the criteria we specified, the contractor ended games

artificially early for the first ∼80 games played in the tournament, with end dates of 1909-1911 being more common thanthey should have been. We immediately corrected this problem once it was identified.

304§14.4. EXPERIMENTAL EVALUATION IN NO-PRESS DIPLOMACY

Diplodocus-High
Human
Diplodocus-Low
Human
Human
BRBot
Human
Human
···
DORA
···
Human

Rank

Elo

Average score# Games

1
2
3
4
5
6
7
8
···
32
···
43

181
162
152
138
136
119
102
96
···
-20
···
-187

27% ± 4%25% ± 6%26% ± 4%22% ± 9%22% ± 3%23% ± 4%18% ± 8%17% ± 3%···13% ± 3%···
1% ± 1%

50
13
50
7
57
50
8
51
···
50
···
7

Table 14.2: Performance of four different agents in a population of human players, ranked by Elo,among all 43 participants who played at least 5 games. The ± shows one standard error.

that Diplodocus performs at least at the level of expert players in this population of players withdiverse skill levels. Additionally, the superior performance of both Diplodocus agents comparedto BRBot is consistent with the results from the agent population experiments in Table 14.1.
In addition to the tournament, we asked three expert human players to evaluate the strength of
the agents in the tournament games based on the quality of their actions. Games were presented tothese experts with anonymized labels so that the experts were not aware of which agent was whichin each game when judging that agent’s strategy. All the experts picked a Diplodocus agent as
the strongest agent, though they disagreed about whether Diplodocus-High or Diplodocus-Low
was best. Additionally, all experts indicated one of the Diplodocus agents as the one they would
most like to cooperate with in a game.

Conclusions and future work
This dissertation sought to provide solid theoretical and algorithmic foundations for strategic,
game-theoretic decision-making in imperfect-information extensive-form games.
In the first part of the dissertation, we focused on learning dynamics, procedures each player canuse to iteratively refine their strategy. Learning dynamics are a fascinating technique, which canrecover global notions of game-theoretic optimality from local notions of strategy improvements.
In this dissertation, we gave several new fundamental results about learning dynamics inimperfect-information extensive-form games. In Chapter 4 we presented a methodology forconstructing predictive dynamics with state-of-the-art practical performance in many gameinstances. In Chapter 5 we investigated several fundamental questions about the metric structureof strategy spaces in imperfect-information extensive-form games, establishing the first notion of
distance that enables linear-time projections while at the same time guaranteeing polynomial
distance between any two strategies. In Chapters 6 and 7 we settled for the positive the questionof whether near-optimal no-regret learning can be achieved in general imperfect-information
extensive-form games, all while improving bounds for normal-form games as well. Finally,in Chapter 8 we provided the first uncoupled, polynomial-time learning dynamics leading
to extensive-form correlated equilibrium (the natural generalization of the classic correlated
equilibrium in normal-form games), closing a longstanding open problem in the literature. As
we pointed out along the way, learning in imperfect-information games is significantly morechallenging (and fascinating) than learning in normal-form (nonsequential) games, and often
requires different techniques. However, surprisingly many positive results can be given. I
hope these results will be useful foundations for future investigations of learning in imperfectinformation games.
Several questions about learning in extensive-form games remain open. We mention just afew. First, what are lower bounds for no-external-regret dynamics, both in terms of dependence
on the number of repetitions T of the game, and as a function of the dimensions of the game?
More concretely, is the state-of-the-art dependence on the game size we developed in Chapter 7
optimal? Can learning dynamics in games with provable OT (1/T ) convergence to coarsecorrelated equilibrium be given? And is there an inherent tradeoff between rate of convergence306§14.4. EXPERIMENTAL EVALUATION IN NO-PRESS DIPLOMACY

and complexity of iterations?[14.c]
Another set of questions relate to alternative feedback models. Is it possible to extend
most (or all) of the no-regret learning dynamics described so far to online learning models
such as bandit and semi-bandit settings? To achieve that, we need to construct several tools of
independent interest, such as efficient loss estimation techniques and the introduction of certain
combinatorially-structured local norms to study convergence of the learning algorithm. Whilesome work has already been done (see, e.g., Farina, Schmucker, and Sandholm (2021), Farina andSandholm (2021b), Kozuno, Ménard, Rémi Munos, and Valko (2021), Bai, C. Jin, Mei, Song, andYu (2022), Song, Mei, and Bai (2022), McAleer, Farina, Lanctot, and Sandholm (2023), and Fiegel,
Ménard, Kozuno, Rémi Munos, Perchet, and Valko (2023)), significant work remains to be done.
In the second part of the dissertation, we focused on the geometry of correlated strategies inimperfect-information games. This topic does not have a counterpart in the theory of normal-form
games, as the difficulty in characterizing correlated strategies lies in the asymmetric informationthat each player observes while playing the imperfect-information game. By making progress on
the fundamental problem of characterizing the polytope of correlated strategies in Chapter 9, wewere able to provide positive complexity results and new algorithms for several domains, including
welfare-maximizing solution concepts (Chapter 10) and TMECor equilibria in adversarial teamgames (Chapter 11). This rippling effect shows that much value is to be gained from attacking
the fundamental question of the structure of correlation in imperfect-information games. In the
future, it would be interesting to further expand the class of games for which the set of correlated
strategies can be characterized (and optimized over) efficiently, and further push the scalability ofexisting methods.
The special combinatorial structure of the polytope of correlated strategies has also enabled
us to develop the first learning algorithms for optimal correlated solution concepts and TMECor
strategies. These techniques are the current practical state of the art. As of now, these learningalgorithms require that correlating players update their strategies in a rather coupled manner. By
shedding even more light on the structure of correlation, it would be interesting to understandwhether this requirement could be somehow relaxed.
Finally, in the last part of the dissertation we focused on learning and equilibrium computationin the presence of imperfect players. As we argued in Chapter 14, it is important to train artificial
intelligence by taking into account a model of human play, while at the same time avoiding theinherent strategic weaknesses and potential for manipulation of imitation learning. This areaof the literature seems to still be in its infancy, both in terms of solution concepts and scalable
computational techniques for computing those concepts. In Chapters 12 and 13 we focused on
trembling-hand refinements in two-player games, standard refinements of the Nash equilibriumthat provably avoid sequentially-irrational behavior and that are therefore more robust to mistakes[14.c] For example, can variance reduction methods, such as those of Carmon, Y. Jin, Sidford, and Tian (2019) for normal-form

games, be meaningfully extended to imperfect-information extensive-form games?

§14. QUANTAL RESPONSE AND REGULARIZATION TOWARDS HUMAN PLAY

307

of the players. These solution concepts had remained a theoretical technique, and before the workin this dissertation, it was an open question how to compute them. After settling the recognizedopen problem of the complexity of computing extensive-form perfect equilibria in Chapter 12, in
Chapter 13 we gave the first practical algorithm for computing exact trembling-hand equilibriumrefinements in real (non-synthetic) games with up to half a billion terminal nodes. These games
are 4-5 orders of magnitude larger than what could be handled with prior techniques. Using our
results, it would be interesting to assess empirically trembling-hand refinements deliver on their
theoretical promises, for example, by evaluating the performance of artificial intelligence trained
to converge to refinements against humans, and comparing with prior techniques for unrefined
equilibria.
In addition to the above questions, furthering our understanding of the computational aspectsof equilibrium computation and learning in other classes of structured games, beyond imperfectinformation extensive-form games, is a natural and important direction of research. For example,
some strategic interactions might be better conceptualized as taking place on a DAG or on a genericdirected graph, rather than a tree. In other applications, the specific class of games has such astrong combinatorial structure that it might be advantageous to resort to different (for example,
factorized) representations of the interaction. This might apply to sequential auctions, security
games, and other games whose action space has a strong combinatorial flavor. More generally, one
could also try to design algorithms that are able to automatically exploit symmetries, structures,and invariants of the games, constructing an appropriate representation on the fly. Infinite games
might also be considered. These could arise from reasoning about infinite repetitions of a finite
game, or from continuous action spaces. Orthogonally to these models, fundamental questions
remain regarding what solution concepts are even meaningful in these settings.
With the rapid advancements in machine learning, game theory, and artificial intelligence,
we have the potential to create systems that can make strategic, sound, intentional, optimal
decisions in ways and at scales that were once thought impossible. I have no doubt that as we
continue to rely more and more on technology and artificial intelligence in our daily lives, thesesystems will help us make better and more efficient decisions. In fact, systems developed using
technology covered in this dissertation are already able, today, to make strategic decisions thatare often superior to human ones in certain settings (including some involving natural language
and cooperation, such as Diplomacy). It is my hope that the work presented in this dissertation
will inspire others to continue pushing the boundaries of what is possible. As we move forward, I
am eager to be a part of the ongoing effort to develop machines that can make strategic decisions,with both old and new colleagues, and to explore the many exciting open questions that lie ahead.

308§14.4. EXPERIMENTAL EVALUATION IN NO-PRESS DIPLOMACYBibliography
Abe, Kenshi, Mitsuki Sakamoto, and Atsushi Iwasaki (2022). “Mutation-driven follow theregularized leader for last-iterate convergence in zero-sum games”. In: Proceedings of theConference on Uncertainty in Artificial Intelligence (UAI).
Abernethy, Jacob and Alexander Rakhlin (2009). Beating the Adaptive Bandit with High Probability.
Tech. rep. UCB/EECS-2009-10. EECS Department, University of California, Berkeley. url:
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-10.html.
Anagnostides, Ioannis, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah
Golowich, and Tuomas Sandholm (2022). “Near-Optimal No-Regret Learning for Correlated
Equilibria in Multi-Player General-Sum Games”. In: Proceedings of the Annual Symposium on
Theory of Computing (STOC).
Anagnostides, Ioannis, Gabriele Farina, Christian Kroer, Andrea Celli, and Tuomas Sandholm(2022). “Faster No-Regret Learning Dynamics for Extensive-Form Correlated and Coarse
Correlated Equilibrium”. In: Proceedings of the ACM Conference on Economics and Computation(EC).
Anagnostides, Ioannis, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, andTuomas Sandholm (2022). “Uncoupled Learning Dynamics with O(log T ) Swap Regret in
Multiplayer Games”. In: Proceedings of the Annual Conference on Neural Information Processing
Systems (NeurIPS).
Anagnostides, Ioannis, Gabriele Farina, and Tuomas Sandholm (2023). “Near-Optimal Φ-Regret
Learning in Extensive-Form Games”. In: Proceedings of the International Conference on Machine
Learning (ICML).
Anagnostides, Ioannis, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm (2022). “On LastIterate Convergence Beyond Zero-Sum Games”. In: Proceedings of the International Conference
on Machine Learning (ICML).

310

BIBLIOGRAPHY

Anthony, Thomas, Tom Eccles, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas Hudson,
Nicolas Porcel, Marc Lanctot, Julien Perolat, Richard Everett, Satinder Singh, Thore Graepel,
and Yoram Bachrach (2020). “Learning to Play No-Press Diplomacy with Best Response Policy
Iteration”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Audibert, Jean-Yves, Sébastien Bubeck, and Gábor Lugosi (2014). “Regret in online combinatorial
optimization”. In: Mathematics of Operations Research 39.1, pp. 31–45.
Azizian, Waiss, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos (2021). “The LastIterate Convergence Rate of Optimistic Mirror Descent in Stochastic Variational Inequalities”.
In: Proceedings of the Conference on Learning Theory (COLT).
Babichenko, Yakov and Aviad Rubinstein (2022). “Communication complexity of approximate
Nash equilibria”. In: Games Econ. Behav. 134, pp. 376–398.
Bai, Yu, Chi Jin, Song Mei, Ziang Song, and Tiancheng Yu (2022). In: Proceedings of the Annual
Conference on Neural Information Processing Systems (NeurIPS).
Bakhtin, Anton, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried,
Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik
Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya
Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, DavidWu, Hugh Zhang, and Markus Zĳlstra (2022). “Human-level play in the game of Diplomacy
by combining language models with strategic reasoning”. In: Science 378.6624. url: https://www.science.org/doi/pdf/10.1126/science.ade9097.
Bakhtin, Anton, David Wu, Adam Lerer, and Noam Brown (2021). “No-Press Diplomacy from
Scratch”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Bakhtin, Anton, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina,
Alexander H Miller, and Noam Brown (2023). “Mastering the Game of No-Press Diplomacy via
Human-Regularized Reinforcement Learning and Planning”. In: Proceedings of the International
Conference on Learning Representations (ICLR).
Basilico, Nicola, Andrea Celli, Giuseppe De Nittis, and Nicola Gatti (2017). “Team-maxmin
equilibrium: efficiency bounds and algorithms”. In: Proceedings of the AAAI Conference on
Artificial Intelligence (AAAI).
Beck, Amir and Marc Teboulle (2003). “Mirror descent and nonlinear projected subgradientmethods for convex optimization”. In: Operations Research Letters 31.3, pp. 167–175.

BIBLIOGRAPHY

311

Ben-Tal, Aharon and Arkadi Nemirovski (2005). “Non-Euclidean restricted memory level method
for large-scale convex optimization”. In: Mathematical Programming 102.3, pp. 407–456.
Ben-Tal, Ahron and Arkadi Nemirovski (2001). Lectures on modern convex optimization: analysis,
algorithms, and engineering applications. Vol. 2. Siam.
Bošanskỳ, B, Christopher Kiekintveld, V Lisý, and Michal Pěchouček (2014). “An Exact DoubleOracle Algorithm for Zero-Sum Extensive-Form Games with Imperfect Information”. In:
Journal of Artificial Intelligence Research, pp. 829–866.
Bošanskỳ, Branislav and Jiři Čermák (2015). “Sequence-form algorithm for computing Stackelberg
equilibria in extensive-form games”. In: Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).
Bowling, Michael, Neil Burch, Michael Johanson, and Oskari Tammelin (2015). “Heads-up Limit
Hold’em Poker is Solved”. In: Science 347.6218.
Boyd, Stephen and Lieven Vandenberghe (2004). Convex Optimization. Cambridge University
Press.
Brown, Noam, Christian Kroer, and Tuomas Sandholm (2017). “Dynamic Thresholding andPruning for Regret Minimization”. In: Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).
Brown, Noam and Tuomas Sandholm (2014). “Regret Transfer and Parameter Optimization”. In:
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Brown, Noam and Tuomas Sandholm (2015). “Regret-Based Pruning in Extensive-Form Games”.
In: Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
Brown, Noam and Tuomas Sandholm (2017a). “Reduced Space and Faster Convergence inImperfect-Information Games via Pruning”. In: Proceedings of the International Conference on
Machine Learning (ICML).
Brown, Noam and Tuomas Sandholm (2017b). “Safe and nested subgame solving for imperfectinformation games”. In: Proceedings of the Annual Conference on Neural Information Processing
Systems (NIPS).
Brown, Noam and Tuomas Sandholm (2017c). “Superhuman AI for heads-up no-limit poker:
Libratus beats top professionals”. In: Science 359, pp. 418–424.
Brown, Noam and Tuomas Sandholm (2019). “Solving imperfect-information games via discounted
regret minimization”. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).

312

BIBLIOGRAPHY

Burch, Neil, Michael Johanson, and Michael Bowling (2014). “Solving Imperfect Information
Games Using Decomposition”. In: Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).
Cai, Yang, Argyris Oikonomou, and Weiqiang Zheng (2022). “Tight Last-Iterate Convergenceof the Extragradient Method for Constrained Monotone Variational Inequalities”. In: CoRR
abs/2204.09228.
Carminati, Luca, Federico Cacciamani, Marco Ciccone, and Nicola Gatti (2022). “Public Information Representation for Adversarial Team Games”. In: Proceedings of the International Conference
on Machine Learning (ICML).
Carmon, Yair, Yujia Jin, Aaron Sidford, and Kevin Tian (2019). “Variance reduction for matrix
games”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Celli, Andrea and Nicola Gatti (2018). “Computational Results for Extensive-Form Adversarial
Team Games”. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Celli, Andrea, Alberto Marchesi, Gabriele Farina, and Nicola Gatti (2020). “No-Regret Learning
Dynamics for Extensive-Form Correlated Equilibrium”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NeurIPS).
Cesa-Bianchi, Nicolo and Gábor Lugosi (2012). “Combinatorial bandits”. In: Journal of Computerand System Sciences 78.5, pp. 1404–1422.
Chen, Xi, Xiaotie Deng, and Shang-Hua Teng (2009). “Settling the Complexity of Computing
Two-Player Nash Equilibria”. In: Journal of the ACM.
Chen, Xi and Binghui Peng (2020). “Hedging in games: Faster convergence of external andswap regrets”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Chiang, Chao-Kai, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, andShenghuo Zhu (2012). “Online optimization with gradual variations”. In: Proceedings of theConference on Learning Theory (COLT).
Chu, Francis and Joseph Halpern (2001). “On the NP-completeness of finding an optimal strategy
in games with common payoffs”. In: International Journal of Game Theory.
Cococcioni, Marco, Massimo Pappalardo, and Yaroslav D. Sergeyev (2018). “Lexicographic
multi-objective linear programming using grossone methodology: Theory and algorithm”. In:
Applied Mathematics and Computation 318, pp. 298–311.

BIBLIOGRAPHY

313

Condat, Laurent (2016). “Fast projection onto the simplex and the ℓ1 ball”. In: Mathematical
Programming 158.1, pp. 575–585.
Conitzer, Vincent and Tuomas Sandholm (2008). “New Complexity Results about Nash Equilibria”.
In: Games and Economic Behavior 63.2. Early version in ĲCAI-03, pp. 621–641.
Coulom, Rémi (2005). BayesElo. Software. url: https : / / www . remi - coulom . fr / Bayesian Elo/#theory.
Dantzig, George B and Mukund N Thapa (2006). Linear programming 2: theory and extensions.
Springer Science & Business Media.
Daskalakis, Constantinos, Alan Deckelbaum, and Anthony Kim (2011). “Near-optimal no-regret
algorithms for zero-sum games”. In: Proceedings of the Annual ACM-SIAM Symposium on
Discrete Algorithms (SODA).
Daskalakis, Constantinos, Maxwell Fishelson, and Noah Golowich (2021). “Near-Optimal NoRegret Learning in General Games”. In: CoRR abs/2108.06924.
Daskalakis, Constantinos, Paul W. Goldberg, and Christos H. Papadimitriou (2009). “The Complexity of Computing a Nash Equilibrium”. In: Commun. ACM 52.2, pp. 89–97.
Daskalakis, Constantinos and Noah Golowich (2022). “Fast rates for nonparametric online learning:from realizability to learning in games”. In: Proceedings of the Annual Symposium on Theory ofComputing (STOC).
Daskalakis, Constantinos and Ioannis Panageas (2019). “Last-Iterate Convergence: Zero-Sum
Games and Constrained Min-Max Optimization”. In: 10th Innovations in Theoretical Computer
Science Conference, ITCS 2019.
Dudik, Miroslav and Geoffrey J Gordon (2009). “A sampling-based approach to computing
equilibria in succinct extensive-form games”. In: Proceedings of the Conference on Uncertainty in
Artificial Intelligence (UAI).
Farina, Gabriele, Tommaso Bianchi, and Tuomas Sandholm (2020). “Coarse Correlation in
Extensive-Form Games”. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Farina, Gabriele, Andrea Celli, Nicola Gatti, and Tuomas Sandholm (2018). “Ex Ante Coordination
and Collusion in Zero-Sum Multi-Player Extensive-Form Games”. In: Proceedings of the Annual
Conference on Neural Information Processing Systems (NeurIPS).
Farina, Gabriele, Andrea Celli, Nicola Gatti, and Tuomas Sandholm (2021). “Connecting Optimal
Ex-Ante Collusion in Teams to Extensive-Form Correlation: Faster Algorithms and Positive
Complexity Results”. In: Proceedings of the International Conference on Machine Learning (ICML).

314

BIBLIOGRAPHY

Farina, Gabriele, Andrea Celli, Alberto Marchesi, and Nicola Gatti (2022). “Simple Uncoupled
No-regret Learning Dynamics for Extensive-form Correlated Equilibrium”. In: Journal of theACM 69.6. url: https://dl.acm.org/doi/10.1145/3563772.
Farina, Gabriele and Nicola Gatti (2017). “Extensive-Form Perfect Equilibrium Computation in
Two-Player Games”. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Farina, Gabriele, Nicola Gatti, and Tuomas Sandholm (2018). “Practical Exact Algorithm for
Trembling-Hand Equilibrium Refinements in Games”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NeurIPS).
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2017). “Regret Minimization in
Behaviorally-Constrained Zero-Sum Games”. In: Proceedings of the International Conference on
Machine Learning (ICML).
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019a). “Online Convex Optimizationfor Sequential Decision Processes and Extensive-Form Games”. In: Proceedings of the AAAI
Conference on Artificial Intelligence (AAAI).
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019b). “Optimistic Regret Minimization
for Extensive-Form Games via Dilated Distance-Generating Functions”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2019c). “Regret Circuits: Composabilityof Regret Minimizers”. In: Proceedings of the International Conference on Machine Learning (ICML).
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2021a). “Better Regularization for
Sequential Decision Spaces: Fast Convergence Rates for Nash, Correlated, and Team Equilibria”.
In: ACM Conference on Economics and Computation.
Farina, Gabriele, Christian Kroer, and Tuomas Sandholm (2021b). “Faster Game Solving via
Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descent”. In:
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Farina, Gabriele, Chung-Wei Lee, Haipeng Luo, and Christian Kroer (2022). “Kernelized Multiplicative Weights for 0/1-Polyhedral Games: Bridging the Gap Between Learning in Extensive-Formand Normal-Form Games”. In: Proceedings of the International Conference on Machine Learning(ICML).
Farina, Gabriele, Chun Kai Ling, Fei Fang, and Tuomas Sandholm (2019a). “Correlation in
Extensive-Form Games: Saddle-Point Formulation and Benchmarks”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).

BIBLIOGRAPHY

315

Farina, Gabriele, Chun Kai Ling, Fei Fang, and Tuomas Sandholm (2019b). “Efficient Regret
Minimization Algorithm for Extensive-Form Correlated Equilibrium”. In: Proceedings of theAnnual Conference on Neural Information Processing Systems (NeurIPS).
Farina, Gabriele and Tuomas Sandholm (2020). “Polynomial-Time Computation of Optimal
Correlated Equilibria in Two-Player Extensive-Form Games with Public Chance Moves andBeyond”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NeurIPS).
Farina, Gabriele and Tuomas Sandholm (2021a). “Equilibrium Refinement for the Age of Machines:
The One-Sided Quasi-Perfect Equilibrium”. In: Proceedings of the Annual Conference on Neural
Information Processing Systems (NeurIPS).
Farina, Gabriele and Tuomas Sandholm (2021b). “Model-Free Online Learning in Unknown
Sequential Decision Making Problems and Games”. In: Proceedings of the AAAI Conference on
Artificial Intelligence (AAAI).
Farina, Gabriele and Tuomas Sandholm (2022). “Fast Payoff Matrix Sparsification Techniques
for Structured Extensive-Form Games”. In: Proceedings of the AAAI Conference on Artificial
Intelligence (AAAI).
Farina, Gabriele, Robin Schmucker, and Tuomas Sandholm (2021). “Bandit Linear Optimizationfor Sequential Decision Making and Extensive-Form Games”. In: Proceedings of the AAAI
Conference on Artificial Intelligence (AAAI).
Fiegel, Côme, Pierre Ménard, Tadashi Kozuno, Rémi Munos, Vianney Perchet, and Michal Valko(2023). Adapting to game trees in zero-sum imperfect information games. arXiv: 2212.12567.
Foster, Dean and Rakesh Vohra (1997). “Calibrated Learning and Correlated Equilibrium”. In:
Games and Economic Behavior 21, pp. 40–55.
Foster, Dylan J., Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Éva Tardos (2016).
“Learning in Games: Robustness of Fast Convergence”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NIPS).
Fousse, Laurent, Guillaume Hanrot, Vincent Lefèvre, Patrick Pélissier, and Paul Zimmermann(2007). “MPFR: A Multiple-Precision Binary Floating-Point Library with Correct Rounding”.
In: ACM Trans. Math. Softw. 33.2. doi: 10.1145/1236463.1236468.
Ganzfried, Sam and Tuomas Sandholm (2015). “Endgame Solving in Large Imperfect-Information
Games”. In: Proceedings of the International Conference on Autonomous Agents and Multiagent
Systems (AAMAS).

316

BIBLIOGRAPHY

Gatti, Nicola, Mario Gilli, and Alberto Marchesi (2020). “A characterization of quasi-perfectequilibria”. In: Games and Economic Behavior 122, pp. 240–255.
Gilboa, Itzhak and Eitan Zemel (1989). “Nash and Correlated Equilibria: Some Complexity
Considerations”. In: Games and Economic Behavior 1, pp. 80–93.
Gilpin, Andrew (2009). “Algorithms for abstracting and solving imperfect information games”.
PhD thesis. Computer Science Department, Carnegie Mellon University.
Gilpin, Andrew, Javier Peña, and Tuomas Sandholm (2012). “First-Order Algorithm with O(ln(1/ϵ))
Convergence for ϵ-Equilibrium in Two-Person Zero-Sum Games”. In: Mathematical Programming
133.1–2. Conference version appeared in AAAI-08, pp. 279–298.
Golowich, Noah, Sarath Pattathil, and Constantinos Daskalakis (2020). “Tight last-iterate convergence rates for no-regret learning in multi-player games”. In: Advances in Neural Information
Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS
2020.
Gorbunov, Eduard, Hugo Berard, Gauthier Gidel, and Nicolas Loizou (2022). “Stochastic Extragradient: General Analysis and Improved Rates”. In: International Conference on Artificial
Intelligence and Statistics.
Gorbunov, Eduard, Nicolas Loizou, and Gauthier Gidel (2022). “Extragradient Method: O(1/K)
Last-Iterate Convergence for Monotone Variational Inequalities and Connections With Cocoercivity”. In: International Conference on Artificial Intelligence and Statistics.
Gordon, Geoffrey J, Amy Greenwald, and Casey Marks (2008). “No-regret learning in convex
games”. In: Proceedings of the International Conference on Machine Learning (ICML), pp. 360–367.
Granlund, Torbjörn and the GMP development team (2012). GNU MP: The GNU Multiple Precision
Arithmetic Library. 5.0.5. http://gmplib.org/.
Grant, Michael, Stephen Boyd, and Yinyu Ye (2006). “Disciplined convex programming”. In:
Global optimization. Springer, pp. 155–210.
Gray, Jonathan, Adam Lerer, Anton Bakhtin, and Noam Brown (2020). “Human-Level Performancein No-Press Diplomacy via Equilibrium Search”. In: Proceedings of the International Conference
on Learning Representations (ICLR).
Greenwald, Amy and Amir Jafari (2003). “A General Class of No-Regret Learning Algorithms
and Game-Theoretic Equilibria”. In: Proceedings of the Conference on Learning Theory (COLT).
Hart, Sergiu and Yishay Mansour (2010). “How long to equilibrium? The communication
complexity of uncoupled equilibrium procedures”. In: Games Econ. Behav. 69.1, pp. 107–126.

BIBLIOGRAPHY

317

Hart, Sergiu and Andreu Mas-Colell (2000). “A Simple Adaptive Procedure Leading to Correlated
Equilibrium”. In: Econometrica 68, pp. 1127–1150.
Hart, Sergiu and Andreu Mas-Colell (2003). “Uncoupled dynamics do not lead to Nash equilibrium”. In: American Economic Review 93, pp. 1830–1836.
Held, Michael, Philip Wolfe, and Harlan P Crowder (1974). “Validation of subgradient optimization”. In: Mathematical programming 6.1, pp. 62–88.
Hirsch, Michael D., Christos H. Papadimitriou, and Stephen A. Vavasis (1989). “Exponential lower
bounds for finding Brouwer fix points”. In: J. Complex. 5.4, pp. 379–416.
Hoda, Samid, Andrew Gilpin, Javier Peña, and Tuomas Sandholm (2010). “Smoothing Techniques
for Computing Nash Equilibria of Sequential Games”. In: Mathematics of Operations Research
35.2.
Hsieh, Yu-Guan, Kimon Antonakopoulos, and Panayotis Mertikopoulos (2021). “Adaptive Learning in Continuous Games: Optimal Regret Bounds and Convergence to Nash Equilibrium”.
In: Proceedings of the Conference on Learning Theory (COLT).
Hu, Hengyuan, Adam Lerer, Brandon Cui, Luis Pineda, David Wu, Noam Brown, and Jakob
Foerster (2021). “Off-Belief Learning”. In: Proceedings of the International Conference on Machine
Learning (ICML).
Hu, Hengyuan, Adam Lerer, Alexander Peysakhovich, and Jakob Foerster (2020). ““Other-Play”
for Zero-Shot Coordination”. In: Proceedings of the International Conference on Machine Learning(ICML).
Hunter, David R. (2004). “MM algorithms for generalized Bradley-Terry models.” In: The annals of
statistics 32, pp. 384–406.
Isermann, H. (1982). “Linear lexicographic optimization”. In: Operations-Research-Spektrum 4.4,
pp. 223–228.
Jacob, Athul Paul, David J. Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin,
Jacob Andreas, and Noam Brown (2022). “Modeling Strong and Human-Like Gameplay with
KL-Regularized Search”. In: Proceedings of the International Conference on Machine Learning(ICML).
Jaggi, Martin (2013). “Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization”. In:
Proceedings of the International Conference on Machine Learning (ICML).
Karmarkar, Narendra (1984). “A new polynomial-time algorithm for linear programming”. In:
Proceedings of the Annual Symposium on Theory of Computing (STOC), pp. 302–311.

318

BIBLIOGRAPHY

Koller, Daphne, Nimrod Megiddo, and Bernhard von Stengel (1996). “Efficient Computation of
Equilibria for Extensive Two-Person Games”. In: Games and Economic Behavior 14.2.
Koo, Terry, Amir Globerson, Xavier Carreras Pérez, and Michael Collins (2007). “Structured
prediction models via the matrix-tree theorem”. In: Joint Conference on Empirical Methods in
Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL),
pp. 141–150.
Kozuno, Tadashi, Pierre Ménard, Rémi Munos, and Michal Valko (2021). “Model-Free Learning for
Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall”. In: Proceedingsof the Annual Conference on Neural Information Processing Systems (NeurIPS).
Kreps, David M. and Robert Wilson (1982). “Sequential Equilibria”. In: Econometrica 50.4, pp. 863–
894.
Kroer, Christian, Gabriele Farina, and Tuomas Sandholm (2017). “Smoothing Method for Approximate Extensive-Form Perfect Equilibrium”. In: Proceedings of the International Joint Conferenceon Artificial Intelligence (ĲCAI).
Kroer, Christian, Gabriele Farina, and Tuomas Sandholm (2018a). “Robust Stackelberg Equilibria
in Extensive-Form Games and Extension to Limited Lookahead”. In: Proceedings of the AAAI
Conference on Artificial Intelligence (AAAI).
Kroer, Christian, Gabriele Farina, and Tuomas Sandholm (2018b). “Solving Large Sequential
Games with the Excessive Gap Technique”. In: Proceedings of the Annual Conference on Neural
Information Processing Systems (NeurIPS).
Kroer, Christian, Kevin Waugh, Fatma Kılınç-Karzan, and Tuomas Sandholm (2020). “Faster algorithms for extensive-form game solving via improved smoothing functions”. In: Mathematical
Programming.
Kuhn, H. W. (1950). “A Simplified Two-Person Poker”. In: Contributions to the Theory of Games.
Vol. 1. Annals of Mathematics Studies, 24. Princeton University Press, pp. 97–103.
Kuhn, H. W. (1953). “Extensive Games and the Problem of Information”. In: Contributions to
the Theory of Games. Vol. 2. Annals of Mathematics Studies, 28. Princeton University Press,
pp. 193–216.
Lanctot, Marc, Kevin Waugh, Martin Zinkevich, and Michael Bowling (2009). “Monte Carlo
Sampling for Regret Minimization in Extensive Games”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NIPS).
Lee, Chung-Wei, Christian Kroer, and Haipeng Luo (2021). “Last-iterate convergence in extensiveform games”. In: Advances in Neural Information Processing Systems 34.

BIBLIOGRAPHY

319

Lemke, Carlton E. (1970). “Recent results on complementarity problems”. In: Nonlinear programming, pp. 349–384.
Lerer, Adam and Alexander Peysakhovich (2019). “Learning Existing Social Conventions via
Observationally Augmented Self-Play”. In: Proceedings of the AAAI/ACM Conference on AI,
Ethics, and Society (AIES).
Lin, Tianyi, Zhengyuan Zhou, Panayotis Mertikopoulos, and Michael I. Jordan (2020). “Finite-Time
Last-Iterate Convergence for Multi-Agent Learning in Games”. In: Proceedings of the International
Conference on Machine Learning (ICML).
Ling, Chun Kai, Fei Fang, and J. Zico Kolter (2018). “What game are we playing? End-to-end
learning in normal and extensive form games”. In: Proceedings of the International Joint Conferenceon Artificial Intelligence (ĲCAI).
Lisỳ, Viliam, Marc Lanctot, and Michael Bowling (2015). “Online Monte Carlo counterfactual regret
minimization for search in imperfect information games”. In: Proceedings of the International
Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 27–36.
Liu, Deyi, Volkan Cevher, and Quoc Tran-Dinh (2020). “A Newton Frank-Wolfe Method forConstrained Self-Concordant Minimization”. In: CoRR abs/2002.07003.
McAleer, Stephen, Gabriele Farina, Marc Lanctot, and Tuomas Sandholm (2023). “ESCHER:
Eschewing Importance Sampling in Games by Computing a History Value Function to
Estimate Regret”. In: Proceedings of the International Conference on Learning Representations(ICLR).
McKelvey, Richard D. and Thomas R. Palfrey (1995). “Quantal Response Equilibria for Normal
Form Games”. In: Games and Economic Behavior 10.1.
McMahan, Brendan, Geoffrey Gordon, and Avrim Blum (2003). “Planning in the Presence of
Cost Functions Controlled by an Adversary”. In: Proceedings of the International Conference on
Machine Learning (ICML), pp. 536–543.
Megiddo, Nimrod (1991). “On finding primal-and dual-optimal bases”. In: ORSA Journal on
Computing 3.1, pp. 63–65.
Milionis, Jason, Christos H. Papadimitriou, Georgios Piliouras, and Kelly Spendlove (2022). “Nash,
Conley, and Computation: Impossibility and Incompleteness in Game Dynamics”. In: CoRR
abs/2203.14129.
Miltersen, Peter Bro and Troels Bjerre Sørensen (2006). “Computing Proper Equilibria of Zero-Sum
Games”. In: Computers and Games.

320

BIBLIOGRAPHY

Miltersen, Peter Bro and Troels Bjerre Sørensen (2010). “Computing a Quasi-Perfect Equilibriumof a Two-Player Game”. In: Economic Theory 42.1.
Moravcik, Matej, Martin Schmid, Karel Ha, Milan Hladik, and Stephen Gaukrodger (2016).
“Refining Subgames in Large Imperfect Information Games”. In: Proceedings of the AAAI
Conference on Artificial Intelligence (AAAI).
Moravčík, Matej, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor
Davis, Kevin Waugh, Michael Johanson, and Michael Bowling (2017). “DeepStack: Expert-levelartificial intelligence in heads-up no-limit poker”. In: Science.
Morrill, Dustin, Ryan D’Orazio, Marc Lanctot, James R Wright, Michael Bowling, and Amy R
Greenwald (2021). “Efficient Deviation Types and Learning for Hindsight Rationality in
Extensive-Form Games”. In: Proceedings of the International Conference on Machine Learning(ICML).
Morrill, Dustin, Ryan D’Orazio, Reca Sarfati, Marc Lanctot, James Wright, Amy Greenwald,and Michael Bowling (2020). “Hindsight and Sequential Rationality of Correlated Play”. In:
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Myerson, Roger B. (1978). “Refinements of the Nash equilibrium concept”. In: International Journalof Game Theory 15, pp. 133–154.
Nair, Ashvin, Murtaza Dalal, Abhishek Gupta, and Sergey Levine (2021). “AWAC: Accelerating
online reinforcement learning with offline datasets”. In: Proceedings of the International Conference
on Learning Representations (ICLR).
Nair, Ashvin, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel (2018).
“Overcoming exploration in reinforcement learning with demonstrations”. In: IEEE international
conference on robotics and automation (ICRA).
Nash, John (1950). “Equilibrium points in N-person games”. In: Proceedings of the National Academyof Sciences 36, pp. 48–49.
Nayyar, Ashutosh, Aditya Mahajan, and Demosthenis Teneketzis (2013). “Decentralized stochastic
control with partial history sharing: A common information approach”. In: IEEE Transactions
on Automatic Control 58.7, pp. 1644–1658.
Nemirovski, Arkadi (2004). “Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point
problems”. In: SIAM Journal on Optimization 15.1.
Nesterov, Yurii (2005a). “Excessive Gap Technique in Nonsmooth Convex Minimization”. In:
SIAM Journal of Optimization 16.1.

BIBLIOGRAPHY

321

Nesterov, Yurii (2005b). “Smooth Minimization of Non-Smooth Functions”. In: Mathematical
Programming. A 103.
Paige, C.C., George P.H. Styan, and Peter G. Wachter (1975). “Computation of the stationary
distribution of a markov chain”. In: Journal of Statistical Computation and Simulation 4.3, pp. 173–
186.
Paquette, Philip, Yuchen Lu, Seton Steven Bocco, Max Smith, O-G Satya, Jonathan K Kummerfeld,
Joelle Pineau, Satinder Singh, and Aaron C Courville (2019). “No-press diplomacy: Modelingmulti-agent gameplay”. In: Proceedings of the Annual Conference on Neural Information Processing
Systems (NeurIPS).
Perolat, Julien, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro
Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. (2021). “FromPoincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium
via Regularization”. In: Proceedings of the International Conference on Machine Learning (ICML).
Piliouras, Georgios, Ryann Sim, and Stratis Skoulakis (2021). “Optimal No-Regret Learning in
General Games: Bounded Regret with Unbounded Step-Sizes via Clairvoyant MWU”. In:
arXiv preprint arXiv:2111.14737.
Piliouras, Georgios, Ryann Sim, and Stratis Skoulakis (2022). “Beyond Time-Average Convergence:
Near-Optimal Uncoupled Online Learning via Clairvoyant Multiplicative Weights Update”.
In: Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
Rakhlin, Alexander and Karthik Sridharan (2013). “Online Learning with Predictable Sequences”.
In: Proceedings of the Conference on Learning Theory (COLT).
Rakhlin, Sasha and Karthik Sridharan (2013). “Optimization, learning, and games with predictablesequences”. In: Proceedings of the Annual Conference on Neural Information Processing Systems(NIPS).
Romanovskii, I. (1962). “Reduction of a Game with Complete Memory to a Matrix Game”. In:
Soviet Mathematics 3.
Ross, Sheldon M (1971). “Goofspiel—the game of pure strategy”. In: Journal of Applied Probability
8.3, pp. 621–625.
Roughgarden, Tim and Omri Weinstein (2016). “On the Communication Complexity of Approximate Fixed Points”. In: 2016 IEEE 57th Annual Symposium on Foundations of Computer Science(FOCS), pp. 229–238.
Selten, Reinhard (1975). “Reexamination of the perfectness concept for equilibrium points inextensive games”. In: International Journal of Game Theory.

322

BIBLIOGRAPHY

Shalev-Shwartz, Shai (2012). “Online Learning and Online Convex Optimization”. In: Foundations
and Trends in Machine Learning 4.2.
Shalev-Shwartz, Shai and Yoram Singer (2007). “A primal-dual perspective of online learningalgorithms”. In: Machine Learning 69.2-3, pp. 115–142.
Siegel, Noah Y, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert,
Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller (2020). “Keep doing
what worked: Behavioral modelling priors for offline reinforcement learning”. In: Proceedingsof the International Conference on Learning Representations (ICLR).
Silver, David, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur
Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. (2018). “A generalreinforcement learning algorithm that masters chess, shogi, and Go through self-play”. In:
Science 362.6419, pp. 1140–1144.
Sion, Maurice (1958). “On general minimax theorems.” In: Pacific J. Math. 8.4, pp. 171–176.
Sokota, Samuel, Ryan D’Orazio, J. Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas,
Noam Brown, and Christian Kroer (2023). “A unified approach to reinforcement learning,quantal response equilibria, and two-player zero-sum games”. In: Proceedings of the International
Conference on Learning Representations (ICLR).
Sokota, Samuel, Edward Lockhart, Finbarr Timbers, Elnaz Davoodi, Ryan D’Orazio, Neil Burch,
Martin Schmid, Michael Bowling, and Marc Lanctot (2021). “Solving Common-Payoff Games
with Approximate Policy Iteration”. In: Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).
Song, Ziang, Song Mei, and Yu Bai (2022). In: Proceedings of the Annual Conference on Neural
Information Processing Systems (NeurIPS).
Southey, Finnegan, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings,
and Chris Rayner (2005). “Bayes’ Bluff: Opponent Modelling in Poker”. In: Proceedings of theConference on Uncertainty in Artificial Intelligence (UAI).
Stoltz, Gilles and Gábor Lugosi (2007). “Learning correlated equilibria in games with compact
sets of strategies”. In: Games and Economic Behavior 59.1, pp. 187–208.
Strouse, DJ, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett (2021). “Collaborating with humans without human data”. In: Proceedings of the Annual Conference on Neural
Information Processing Systems (NeurIPS).
Syrgkanis, Vasilis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire (2015). “Fast convergenceof regularized learning in games”. In: Advances in Neural Information Processing Systems.

BIBLIOGRAPHY

323

Takimoto, Eĳi and Manfred K Warmuth (2003). “Path kernels and multiplicative updates”. In:
Journal of Machine Learning Research 4, pp. 773–818.
Tammelin, Oskari (2014). Solving large imperfect information games using CFR+. arXiv: 1407.5042.
Tammelin, Oskari, Neil Burch, Michael Johanson, and Michael Bowling (2015). “Solving Heads-up
Limit Texas Hold’em”. In: Proceedings of the International Joint Conference on Artificial Intelligence(ĲCAI).
Tran-Dinh, Quoc, Anastasios Kyrillidis, and Volkan Cevher (2015). “Composite self-concordant
minimization”. In: Journal of Machine Learning Research 16, pp. 371–416.
van Damme, Eric (1984). “A relation between perfect equilibria in extensive form games andproper equilibria in normal form games”. In: International Journal of Game Theory.
Vinyals, Oriol, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik,
Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. (2019).
“Grandmaster level in StarCraft II using multi-agent reinforcement learning”. In: Nature
575.7782.
von Stengel, Bernhard (1996). “Efficient Computation of Behavior Strategies”. In: Games andEconomic Behavior 14.2, pp. 220–246.
von Stengel, Bernhard (2002). “Computing Equilibria for Two-Person Games”. In: Handbook ofgame theory. Ed. by Robert Aumann and Sergiu Hart. Vol. 3.
Von Stengel, Bernhard and Françoise Forges (2008). “Extensive-form correlated equilibrium:
Definition and computational complexity”. In: Mathematics of Operations Research 33.4, pp. 1002–
1022.
Von Stengel, Bernhard and Daphne Koller (1997). “Team-maxmin equilibria”. In: Games andEconomic Behavior 21.1-2, pp. 309–321.
Warmuth, Manfred K and Dima Kuzmin (2008). “Randomized online PCA algorithms with regretbounds that are logarithmic in the dimension”. In: Journal of Machine Learning Research 9.10,
pp. 2287–2320.
Wei, Chen-Yu, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo (2021). “Linear Last-iterate
Convergence in Constrained Saddle-point Optimization”. In: Proceedings of the International
Conference on Learning Representations (ICLR).
Zhang, Brian Hu, Gabriele Farina, Ioannis Anagnostides, Federico Cacciamani, Stephen McAleer,
Andreas Haup, Andrea Celli, Nicola Gatti, Vincent Conitzer, and Tuomas Sandholm (2023).
Learning and Steering toward Optimal Equilibria and Mechanisms.

324

BIBLIOGRAPHY

Zhang, Brian Hu, Gabriele Farina, Andrea Celli, and Tuomas Sandholm (2022). “Optimal
Correlated Equilibria in General-Sum Extensive-Form Games: Fixed-Parameter Algorithms,
Hardness, and Two-Sided Column-Generation”. In: Proceedings of the ACM Conference on
Economics and Computation (EC).
Zhang, Brian Hu, Gabriele Farina, and Tuomas Sandholm (2023). “Team Belief DAG Form: A
Concise Representation for Team-Correlated Game-Theoretic Decision Making”. In: Proceedingsof the International Conference on Machine Learning (ICML).
Zhang, Brian Hu and Tuomas Sandholm (2020). “Sparsified Linear Programming for Zero-Sum
Equilibrium Finding”. In: Proceedings of the International Conference on Machine Learning (ICML).
Zhang, Brian Hu and Tuomas Sandholm (2022a). “Team Correlated Equilibria in Zero-Sum
Extensive-Form Games via Tree Decompositions”. In: Proceedings of the AAAI Conference on
Artificial Intelligence (AAAI).
Zhang, Brian Hu and Tuomas Sandholm (2022b). “Team Correlated Equilibria in Zero-Sum
Extensive-Form Games via Tree Decompositions”. In: Proceedings of the AAAI Conference on
Artificial Intelligence (AAAI).
Zhang, Hugh (2022). A Simple Adaptive Procedure Converging to Forgiving Correlated Equilibria. arXiv:
2207.06548.
Zhang, Youzhi, Bo An, and Jakub Černỳ (2021). “Computing Ex Ante Coordinated Team-Maxmin
Equilibria in Zero-Sum Multiplayer Extensive-Form Games”. In: Proceedings of the AAAI
Conference on Artificial Intelligence (AAAI).
Zinkevich, Martin, Michael Bowling, Michael Johanson, and Carmelo Piccione (2007). “Regret
Minimization in Games with Incomplete Information”. In: Proceedings of the Annual Conference
on Neural Information Processing Systems (NIPS).

Appendix A

Description of benchmark games
used in experimental evaluations

In this appendix, we provide a detailed description of the game instances used throughout thedifferent empirical evaluations in the dissertation. All games are identified with a mnemonic that
identifies the game and parameters. A table summarizing several key dimensions of the games isavailable in Appendix A.2.

A.1

Description of game instances

A.1.1

Battleship ( B )The game is a general-sum version of the classic game Battleship, where two players take turns
placing ships of varying sizes and values on two separate grids of size h × w, and then take turns
firing at their opponent. Ships which have been hit at all their tiles are considered destroyed.
The game ends when one player loses all their ships, or after each player has fired r shots. Each
player’s payoff is determined by the sum of the value of the opponent’s destroyed ships minusγ ⩾ 1 times the number of their own lost ships. We denote by Bphwr an instance with p players
on a grid of size h × w, one unit-size ship for each player, and r rounds.
The Battleship game is known to possess a rich set of correlated solution concepts and wereintroduced by Farina, Ling, Fang, and Sandholm (2019a) as a benchmark for extensive-form
correlated equilibrium.

326

A.1.2

§A.1. DESCRIPTION OF GAME INSTANCES

Liar’s dice ( D )This is a standard benchmark game in the game-solving literature (see, e.g., Lisỳ, Lanctot, andBowling, 2015). At the start of the game, each of the n players rolls a fair k-sided die privately.
Then, the players take turns making claims about the outcome of their roll. The first player starts
by stating any number from 1 to k and the minimum number of dice they believe are showingthat value among all players. On their turn, each player has the option to make a higher claim
or challenge the previous claim by calling the previous player a “liar”. A claim is higher if
the number rolled is higher or the number of dice showing that number is higher. If a player
challenges the previous claim and the claim is found to be false, the challenger is rewarded +1
and the last bidder receives a penalty of -1. If the claim is true, the last bidder is rewarded +1,
and the challenger receives -1. Game instances are encoded as Dnk .

A.1.3

Goofspiel ( G )This is a standard trick-taking game introduced by Ross (1971). It is an n-player card game,
employing n + 1 identical decks of k cards each whose values range from 1 to k. At the beginningof the game, each player gets dealt a full deck as their hand, and the third deck (the “prize” deck)is shuffled and put face down on the board. In each turn, the topmost card from the prize deck
is revealed. Then, each player privately picks a card from their hand. This card acts as a bid to
win the card that was just revealed from the prize deck. The selected cards are simultaneously
revealed, and the highest one wins the prize card. If the players’ played cards are equal, the prize
card is split. The players’ score are computed as the sum of the values of the prize cards they
have won. Game instances are encoded as Gnk .

A.1.4

Limited-information Goofspiel ( GL )This is a variant of Goofspiel with limited information. In this variation, in each turn the players
do not reveal the cards that they have played. Instead, players show their cards to a neutral
umpire, who then decides the winner of the round by determining which card is the highest.
In the event of a tie, the umpire directs the players to divide the prize equally among the tied
players, similar to the Goofspiel game. Game instances are encoded as GLnk .

A.1.5

Kuhn poker ( K )Three-player Kuhn Poker, an extension of the original two-player version proposed by Kuhn
(1950), is played with n players and r cards. Each player begins by paying one chip to the pot andreceiving a single private card. The first player can check or bet (i.e., putting an additional chip in
the pot). Then, the second player can check or bet after a first player’s check, or fold/call the firstplayer’s bet. The third player can either check or bet if no previous bet was made, otherwise they§A. DESCRIPTION OF BENCHMARK GAMES USED IN EXPERIMENTAL EVALUATIONS

327

must fold or call. At the showdown, the player with the highest card who has not folded wins
all the chips in the pot. We encode Kuhn poker instances with the mnemonic Knr , where r is
encoded in hexadecimal (so, for example, K2c is set up with r = 12).

A.1.6

Leduc poker ( L )This is another standard benchmark introduced by Southey, Bowling, Larson, Piccione, Burch,
Billings, and Rayner (2005). In n-player Leduc poker the deck consists of s suits with r cards each.
Our instances are parametric in the maximum number of bets b, which in limit hold’em is notnecessarily tied to the number of players. The maximum number of raise per betting round canbe either 1, 2 or 3. At the beginning of the game, players each contribute one chip to the pot. The
game proceeds with two rounds of betting. In the first round, each player is dealt a private card,and in the second round, a shared board card is revealed. The minimum raise is set at 2 chips in
the first round and 4 chips in the second round. We denote by Lnbrs an instance with n players,b bets per round, r ranks, and s suits.

A.1.7

Pursuit-evasion ( P )This is a security-inspired pursuit-evasion game played on the graph shown in Figure A.1. It
is a zero-sum variant of the one used by Kroer, Farina, and Sandholm (2018a), and a similar
search game has been considered by Bošanskỳ, Kiekintveld, Lisý, and Pěchouček (2014) andBranislav Bošanskỳ and Čermák (2015).
5
10

S

3
P1

P2

Figure A.1: The graph on which the search game is played.

In each turn, the attacker and the defender act simultaneously. The defender controls two
patrols, one per each respective patrol areas labeled P1 and P2 . Each patrol can move by one
step along the gray dashed lines, or stay in place. The attacker starts from the leftmost node(labeled S) and at each turn can move to any node adjacent to its current position by following
the black directed edges. The attacker can also choose to wait in place for a time step in order tohide all their traces. If a patrol visits a node that was previously visited by the attacker, and theattacker did not wait to clean up their traces, they will see that the attacker was there. The goal of
the attacker is to reach any of the rightmost nodes, whose corresponding payoffs are 5, 10, or 3,

328§A.1. DESCRIPTION OF GAME INSTANCES

respectively, as indicated in Figure A.1. If at any time the attacker and any patrol meet at thesame node, the attacker loses the game, which leads to a payoff of −1 for the attacker and of 1 forthe defender. The game times out after m simultaneous moves, in which case both defenders
receive payoffs 0. We encode instances of this game via the mnemonic P2m .

A.1.8

River Endgame ( REL )The river endgame is structured and parameterized as follows. The game is parameterized by theconditional distribution over hands for each player, current pot size, board state (5 cards dealt tothe board), and a betting abstraction. First, Chance deals out hands to the two players according
to the conditional hand distribution. Then, Libratus has the choice of folding, checking, or betting
by a number of multipliers of the pot size: 0.25x, 0.5x, 1x, 2x, 4x, 8x, and all-in. If Libratus checks
and the other player bets then Libratus has the choice of folding, calling (i.e. matching the bet
and ending the betting), or raising by pot multipliers 0.4x, 0.7x, 1.1x, 2x, and all-in. If Libratus
bets and the other player raises Libratus can fold, call, or raise by 0.4x, 0.7x, 2x, and all-in. Finally,
when facing subsequent raises Libratus can fold, call, or raise by 0.7x and all-in. When facedwith an initial check, the opponent can fold, check, or raise by 0.5x, 0.75x, 1x, and all-in. When
faced with an initial bet the opponent can fold, call, or raise by 0.7x, 1.1x, and all-in. When facedwith subsequent raises the opponent can fold, call, or raise by 0.7x and all-in. The game ends
whenever a player folds (the other player wins all money in the pot), calls (a showdown occurs),
or both players check as their first action of the game (a showdown occurs). In a showdown theplayer with the better hands wins the pot. The pot is split in case of a tie. We denote with REL2i
the instances of the river endgame, where i is a unique identifier of the endgame configuration.

A.1.9

Sheriff ( S )This game is a simplified version of the Sheriff of Nottingham board game, which models theinteraction between a Smuggler—who is trying to smuggle illegal items in their cargo—and theSheriff —whose goal is stopping the Smuggler. First, the Smuggler has to decide the numbern ∈ {0, . . . , N } of illegal items to load on the cargo. Then, the Sheriff decides whether to inspect
the cargo. If they choose to inspect, and find illegal goods, the Smuggler has to pay p · n to
the Sheriff. Otherwise, the Sheriff has to compensate the Smuggler with a reward of s. If theSheriff decides not to inspect the cargo, the Sheriff’s utility is 0, and the Smuggler’s utility isv · n. After the Smuggler has loaded the cargo, and before the Sheriff decides whether to inspect,
the Smuggler can try to bribe the Sheriff to avoid the inspection. In particular, they engage in r
rounds of bargaining and, for each round i, the Smuggler proposes a bribe bi ∈ {0, . . . , B}, andthe Sheriff accepts or declines it. Only the proposal and response from the final round r are
executed. If the Sheriff accepts a bribe br then they get br , while the Smuggler’s utility is vn − br .
Further details on the game can be found in Farina, Ling, Fang, and Sandholm (2019a). An

§A. DESCRIPTION OF BENCHMARK GAMES USED IN EXPERIMENTAL EVALUATIONS

329

instance SpN Br has p players, N illegal items (encoded in hexadecimal notation), a maximum
bribe of B, and r rounds of bargaining. The other parameters are v = 5, p = 1, s = 1 and they are
fixed across all instances.

A.1.10

Double-dummy bridge endgame ( T , TP )The double-dummy bridge endgame is a benchmark introduced by B. H. Zhang, Farina, Celli,and Sandholm (2022) which simulates a bridge endgame scenario. The game uses a fixed deck ofplaying cards that includes three ranks (2, 3, 4) of each of four suits (spades, hearts, diamonds,
clubs). Spades are designated as the trump suit. There are four players involved: two defenders
sitting across from each other, the dummy, and the declarer. The dummy’s actions will becontrolled by the declarer, so there are only three players actively participating. However, forclarity, we will refer to all four players throughout this section.
The entire deck of cards is randomly dealt to the four players. We study the version of thegame that has perfect information, meaning that all players’ cards are revealed to everyone,creating a game in which all information is public (i.e., a double-dummy game). The game is playedin rounds called tricks. The player to the left of the declarer starts the first trick by playing a card.
The suit of this card is known as the lead suit. Going in clockwise order, the other three players
play a card from their hand. Players must play a card of the lead suit if they have one, otherwise,
they can play any card. If a spade is played, the player with the highest spade wins the trick.
Otherwise, the highest card of the lead suit wins the trick. The winner of each trick then leads
the next one. At the end of the game, each player earns as many points as the number of tricks
they won. In this adversarial team game, the two defenders are teammates and play against thedeclarer, who controls the dummy.
The specific instance that we use (i.e., TP3 ) has 3 ranks and perfect information. The dummy’s
hand is fixed as 2♠ 2♥ 3♥.
In the limited deals variant T3[L] , L deals are randomly selected at the beginning of the game,and it is common knowledge that the true deal is among them. This limits the size of the game
tree, as well as the parameters on which the complexity of our algorithms depend. Note thatL = 9!/(3!)3 = 1680 is the full game.

A.1.11

Ridesharing game ( RS )This benchmark was first introduced by B. H. Zhang, Farina, Celli, and Sandholm (2022), and it
models the interaction between two drivers competing to serve requests on a road network. The
network is defined as an undirected graph GU = (V U , E U ), where each vertex v ∈ V U corresponds
to a ride request to be served. Each request has a reward in ℝ⩾0 , and each edge in the network
has some cost. The first driver who arrives at node v ∈ V U serves the corresponding ride, andreceives the corresponding reward. Once a node has been served, it stays clean until the end of

330§A.1. DESCRIPTION OF GAME INSTANCES

the game. The game terminates when all nodes have been cleared, or when a timeout is met (i.e.,there’s a fixed time horizon T ). If the two drivers arrive simultaneously on the same vertex theyboth get reward 0. The final utility of each driver is computed as the sum of the rewards obtained
from the beginning until the end of the game. The initial position of the two drivers is randomly
selected at the beginning of the game. Finally, the two drivers can observe each other’s position
only when they are simultaneously on the same node, or they are in adjacent nodes.
Ridesharing games are particularly well-suited to study the computation of optimal equilibria
because they are not triangle-free (Farina and Sandholm, 2020).
Setup We denote by RS p i T a ridesharing instance with p drivers, network configuration i,and horizon T . Parameter i ∈ {1, 2} specifies the graph configuration. We consider the two
network configurations of B. H. Zhang, Farina, Celli, and Sandholm (2022), their structure isreported in Figure A.2. All edges are given unitary cost. We consider a total of four instances:
RS212 , RS222 , RS213 , RS223 .
{.5}

1
{.5}

{2}

1

5

5 {2.5}
{.5}

2
{1}

{1}

0

{.5}

3

{4.5}

0

4

7 {5}

3 {1.5}

2

6

{1}

{1.5}

{1.5}

4

6 {1.5}Figure A.2: Left: Graph configuration 1, used for RS212 RS213 . Right: Graph configuration 2, used
for RS222 RS223 . In both cases the position of the two drivers is randomly chosen at the
beginning of the game, edge costs are unitary, and the reward for each node is indicated between
curly brackets.

A.1.12

Small matrix ( SM )This is a small 2 × 2 matrix game introduced by Farina, Kroer, and Sandholm (2019b) as a hard
instance for the CFR+ algorithm. Given a mixed strategy x = (x1 , x2 ) ∈ ∆2 for Player 1 and amixed strategy y = (y1 , y2 ) ∈ ∆2 for Player 2, the payoff function for player 1 is defined asu(x, y) := 5x1 y1 − x1 y2 + x2 y2 .
We denote the small matrix with the mnemonic SM2 .

§A. DESCRIPTION OF BENCHMARK GAMES USED IN EXPERIMENTAL EVALUATIONS

A.2

Game dimensions

Game

Players

Decision nodes

Sequences

Terminal nodes

Constant sum?

B2222
B2322
B2323
B2324

2
2
2
2

162
938
15 338144 938506
4730
62 330451 1301072
19 116
191 916969 516

✗
✗
✗
✗

D32
D33
D42

3
3
4

128
1536
12 288255
3069
24 574504
13 797331 695

✓
✓
✓

G33
G24
G25

3
2
2

4383
34 952
4 369 010

4890
42 6585 332 052

1296
13 824
1 728 000

✓
✓
✓

GL24
GL25
GL33
GL44
GL55

2
2
3
4
5

17 432
1 175 330
2511
13 23665 38521 298
1 428 452
2802
14 00867 31013 824
1 728 000
1296
7776
46 656

✓
✓
✓
✓
✓

K23
K2c
K33
K34
K35
K36
K3c
K38
K45

2
2
3
3
3
3
3
3
4

12
5148
36
48
60
72
144
96
160

26
12 014
75
99
123
147
291
195
324

30
98 956
78
312
780
1560
17 160
4368
3960

✓
✓
✓
✓
✓
✓
✓
✓
✓

Table A.1: Dimensions of game instances used in this dissertation.

331

332

§A.2. GAME DIMENSIONS

Game

Players

Decision nodes

Sequences

Terminal nodes

Constant sum?

L2232
L2252
L2282
L2292
L22d2
L3132
L3133
L3151
L3223
L3523

2
2
2
2
2
3
3
3
3
3

288
780
1968
2484
5148
684
684
1500
1890
148 752

674
1822
4594
5798
12 014
1371
1371
3003
4329
369 459

1116
5500
22 93632 72498 9564500
6477
10 0208762
775 148

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

P24
P25
P26

2
2
2

382
2078
11 8882081
11 89969 02915 89861 084
118 514

✓
✓
✓

REL26
REL27
REL28
REL22
REL25

2
2
2
2
2

53 01042 09867 431
99 144242 190

142 223
123 580
184 047
293 546722 432

71 270 327
75 928 256
111 580 420
185 831 684
494 214 830

✓
✓
✓
✓
✓

S2122
S2123
S2133
S2254
S2264

2
2
2
2
2

99
603
1096
50 89682 741

286
1690
3809
260 153475 778

396
2376
5632
435 456806 736

✗
✗
✗
✗
✗

SM2

2

2

6

4

✓

T3[50]

3

14 44824 30410 300✓TP3

3531 728909 059

379 008

✓

RS212
RS213
RS222
RS223

2
2
2
2

68
704
76
672

214
2370
218
1982

400
4356
484
4096

✗
✗
✗
✗

Table A.2: (Continued) Dimensions of game instances used in this dissertation.

334§B.1. TREE-FORM DECISION PROCESSES

Appendix B

Summary of notation
B.1

Tree-form decision processes

SymbolDescriptionJ
Aj

Set of decision nodes
Set of legal actions at decision node j ∈ J

KSk

Set of observation nodes
Set of possible signals at observation node k ∈ KρTransition function:• given j ∈ J and a ∈ Aj , ρ(j, a) returns the next point v ∈ J ∪ K in thedecision tree that is reached after selecting legal action a in j, or ⊥ if thedecision process ends;• given k ∈ K and s ∈ Sk , ρ(k, s) returns the next point v ∈ J ∪ K in thedecision tree that is reached after observing signal s in k, or ⊥ if the decision
process ends

Σ∗
Σ
pj

Cσ

Set of non-empty sequences, defined as Σ∗ := {(j, a) : j ∈ J , a ∈ Aj }Set of sequences, defined as Σ := Σ∗ ∪ {∅} where the special element ∅ iscalled the empty sequence
Parent sequence of decision node j ∈ J , defined as the last sequence (decisionnode-action pair) on the path from the root of the TFDP to decision node j; if
the player does not act before j, pj = ∅Decision nodes j ∈ J with parent sequence σ ∈ Σ: Cσ := {j ∈ J : pj = σ}′j ′ ∈ J is on the path from the root to j ∈ J
Action a′ at j ′ is on the path from the root to action a at j ∈ J
Shorthand for σ = j ′ a′ with j ′ ≽ jΣ≽jSequences at or below j: Σ≽j := {σ ∈ Σ∗ : σ ≽ j}

j ≼j
′ ′
j a ≼ jaσ≽j335

§B. SUMMARY OF NOTATION

B.2

Correlated strategies

SymbolDescription

j1 ⇌ j2

Σ1 ▷◁ Σ2

Connected decision nodes: if j1 ⇌ j2 , it is possible that the trajectory in thedecision process goes through both j1 ∈ J1 and j2 ∈ J2
Relevant sequence pair: σ1 ▷◁ σ2 if at least one between σ1 and σ2 is the empty
sequence ∅, or if σ1 = (j1 , a1 ), σ2 = (j2 , a2 ) with j1 ⇌ j2
Set of all relevant sequence pairs {(σ1 , σ2 ) ∈ Σ1 × Σ2 : σ1 ▷◁ σ2 }j1 ▷◁ σ2
σ1 ▷◁ j2

Shorthand for: (j1 , a1 ) ▷◁ σ2 for all a1 ∈ Aj1
Shorthand for: σ1 ▷◁ (j2 , a2 ) for all a2 ∈ Aj2

σ1 ▷◁ σ2

B.3

Mathematical notation

Vectors and matrices• Vectors and matrices are marked in bold.
• Given a finite set S = {s1 , . . . , sn }, we denote as ℝS (resp., ℝS⩾0 ) the set of real (resp.,
nonnegative real) |S|-dimensional vectors whose entries are denoted as x[s1 ], . . . , x[sn ].
′′• Similarly, given finite sets S, S ′ , we denote as ℝS×S (resp., ℝS×S
⩾0 ) the set of real (resp.,
nonnegative real) S × S ′ square matrices M whose entries are denoted as M[sr , sc ] (sr ∈S, sc ∈ S ′ ), where sr corresponds to the row index and sc to the column index.
Standard sets
• We denote the set of real numbers as ℝ, the set of nonnegative real numbers as ℝ⩾0 , and theset {1, 2, . . . } of positive integers as ℕ.
• The set {1, . . . , n}, where n ∈ ℕ, is compactly denoted as [[n]].
• The empty set is denoted as { }.
• Given a finite set S, we denote by ∆S the simplex ∆S := {x ∈ ℝS⩾0 :symbol ∆n , with n ∈ ℕ, is used to mean ∆[[n]] .

Ps∈S x[s] = 1}. The• Given a finite set S, we use the symbol 𝕊S ⊆ ℝS×S
⩾0 to denote the set of stochastic matrices,that is, nonnegative square matrices whose columns all sum up to 1. The symbol 𝕊n , wheren ∈ ℕ, is used to mean 𝕊[[n]] .

336§B.3. MATHEMATICAL NOTATION

Operations on sets
• Given a set S, we denote its convex hull with the symbol co S. The convex hull of the unionof finitely many sets S1 , . . . , Sm is denoted co{S1 , . . . , Sm }.
• Disjoint union of set is denoted with the symbol ⊔.
Functions• Given two functions f : X → Y and g : Y → Z, we denote by g ◦ f : X → Z theircomposition x 7→ g(f (x)).
• Given a set S and a function f , the image of S via f is denoted as f (S) := {f (s) : s ∈ S}.
• Given a proposition p, we denote with 𝟙p the indicator function of that proposition:

1
𝟙p =
0

if p is true
otherwise.

Partial orders• Given a partially ordered set (S, ≺) and two elements s, s′ ∈ S, we use the standard derived
symbols s ≼ s′ to mean that (s = s′ ) ∨ (s ≺ s′ ), s ≻ s′ to mean that s′ ≺ s, and s ≽ s′ tomean that s′ ≼ s.
Asymptotic notation• We use the symbols O, Ω, Θ to denote the usual asymptotic notation.
• The symbol OT is used to denote that only dependence on the parameter T is highlighted,
treating all other parameters as constants.