Title: Efficient $\Phi$-Regret Minimization with Low-Degree Swap Deviations in Extensive-Form Games
Venue: NeurIPS 2024
Accepted: yes
Keywords: swap regret

extensive-form games

low-degree deviations
Abstract: Recent breakthrough results by Dagan, Daskalakis, Fishelson and Golowich [2023] and Peng and Rubinstein [2023] established an efficient algorithm attaining at most $\epsilon$ swap regret over extensive-form strategy spaces of dimension $N$ in $N^{\tilde O(1/\epsilon)}$ rounds. On the other extreme, Farina and Pipis [2023] developed an efficient algorithm for minimizing the weaker notion of linear-swap regret in $\mathsf{poly}(N)/\epsilon^2$ rounds. In this paper, we develop efficient parameterized algorithms for regimes between these two extremes. We introduce the set of $k$-mediator deviations, which generalize the untimed communication deviations recently introduced by Zhang, Farina and Sandholm [2024] to the case of having multiple mediators, and we develop algorithms for minimizing the regret with respect to this set of deviations in $N^{O(k)}/\epsilon^2$ rounds. Moreover, by relating $k$-mediator deviations to low-degree polynomials, we show that regret minimization against degree-$k$ polynomial swap deviations is achievable in $N^{O(kd)^3}/\epsilon^2$ rounds, where $d$ is the depth of the game, assuming a constant branching factor. For a fixed degree $k$, this is polynomial for Bayesian games and quasipolynomial more broadly when $d = \mathsf{polylog} N$---the usual balancedness assumption on the game tree. The first key ingredient in our approach is a relaxation of the usual notion of a fixed point required in the framework of Gordon, Greenwald and Marks [2008]. Namely, for a given deviation $\phi$, we show that it suffices to compute what we refer to as a fixed point in expectation; that is, a distribution $\pi$ such that $\mathbb{E}_{x \sim \pi} [\phi(x) - x] \approx 0$. Unlike the problem of computing an actual (approximate) fixed point $x \approx \phi(x)$, which we show is \PPAD-hard, there is a simple and efficient algorithm for finding a solution that satisfies our relaxed notion. As a byproduct, we provide, to our knowledge, the fastest algorithm for computing $\epsilon$-correlated equilibria in normal-form games in the medium-precision regime, obviating the need to solve a linear system in every round. Our second main contribution is a characterization of the set of low-degree deviations, made possible through a connection to low-depth decisions trees from Boolean analysis.

Efficient Φ -Regret Minimization with Low-Degree Swap Deviations in Extensive-Form Games

Abstract

Recent breakthrough results by Dagan, Daskalakis, Fishelson and Golowich [2023] and Peng and Rubinstein [2023] established an efficient algorithm attaining at most ϵ swap regret over extensive-form strategy spaces of dimension N in N ˜ O (1 /ϵ ) rounds. On the other extreme, Farina and Pipis [2023] developed an efficient algorithm for minimizing the weaker notion of linear-swap regret in poly ( N ) /ϵ 2 rounds. In this paper, we develop efficient parameterized algorithms for regimes between these two extremes. We introduce the set of k -mediator deviations , which generalize the untimed communication deviations recently introduced by Zhang, Farina and Sandholm [2024] to the case of having multiple mediators, and we develop algorithms for minimizing the regret with respect to this set of deviations in N O ( k ) /ϵ 2 rounds. Moreover, by relating k -mediator deviations to low-degree polynomials, we show that regret minimization against degreek polynomial swap deviations is achievable in N O ( kd ) 3 /ϵ 2 rounds, where d is the depth of the game, assuming a constant branching factor. For a fixed degree k , this is polynomial for Bayesian games and quasipolynomial more broadly when d = polylog N -the usual balancedness assumption on the game tree. The first key ingredient in our approach is a relaxation of the usual notion of a fixed point required in the framework of Gordon, Greenwald and Marks [2008]. Namely, for a given deviation ϕ , we show that it suffices to compute what we refer to as a fixed point in expectation ; that is, a distribution π such that E x ∼ π [ ϕ ( x ) -x ] ≈ 0 . Unlike the problem of computing an actual (approximate) fixed point x ≈ ϕ ( x ) , which we show is PPAD -hard, there is a simple and efficient algorithm for finding a solution that satisfies our relaxed notion. As a byproduct, we provide, to our knowledge, the fastest algorithm for computing ϵ -correlated equilibria in normal-form games in the medium-precision regime, obviating the need to solve a linear system in every round. Our second main contribution is a characterization of the set of low-degree deviations, made possible through a connection to low-depth decisions trees from Boolean analysis.

1 Introduction

Correlated equilibrium (CE) , introduced in a groundbreaking work by Aumann [1974], has emerged as one of the most influential solution concepts in game theory. Often contrasted with Nash equilibrium [Nash, 1950], it is regarded by many as more natural; in the words attributed to another Nobel laureate, Roger Myerson, 'if there is intelligent life on other planets, in a majority of them, they would have discovered correlated equilibrium before Nash equilibrium.' Correlated equilibria also enjoy more favorable computational properties: unlike Nash equilibria, they can be expressed as solutions to a linear program, thereby enabling their computation in polynomial time, at least in normal-form games [Papadimitriou and Roughgarden, 2008, Jiang and Leyton-Brown, 2011]. Further, a correlated equilibrium arises through repeated play from natural no-regret learning dynamics [Hart and Mas-Colell, 2000, Foster and Vohra, 1997].

However, many real-world strategic interactions feature sequential moves and imperfect information. In such scenarios, the so-called extensive form constitutes the canonical game representation [Kuhn, 1953, Shoham and Leyton-Brown, 2009]: a normal-form description of the game would be prohibitively large. It is startling to realize that 50 years after Aumann's original work, the complexity of computing correlated equilibria in extensive-form games-sometimes referred to as normal-form correlated equilibria (NFCE) to disambiguate from other pertinent but weaker solution concepts-remains an outstanding open problem [von Stengel and Forges, 2008, Papadimitriou and Roughgarden, 2008].

The long-standing absence of efficient algorithms for computing an NFCE shifted the focus to natural relaxations thereof, which can be understood through the notion of Φ -regret [Greenwald and Hall, 2003, Stoltz and Lugosi, 2007, Rakhlin et al., 2011]. In particular, Φ represents a set of strategy deviations; the richer the set of deviations, the stronger the induced solution concept. When Φ contains all possible transformations, one recovers the notion of NFCE-corresponding to swap regret . At the other end of the spectrum, coarse correlated equilibria correspond to Φ consisting solely of constant transformations (aka. external regret ). Perhaps the most notable relaxation is the extensive-form correlated equilibrium (EFCE) [von Stengel and Forges, 2008], which can be computed exactly in time polynomial in the representation of the game tree [Huang and von Stengel, 2008]. Considerable interest in the literature has recently been on learning dynamics that minimize Φ -regret ( e.g. , Morrill et al. [2021a,b], Bai et al. [2022], Bernasconi et al. [2023], Noarov et al. [2023], Dud´ ık and Gordon [2009], Gordon et al. [2008], Fujii [2023], Dann et al. [2023], Mansour et al. [2022]). A key reference point in this line of work is the recent construction of Farina and Pipis [2023], an efficient algorithm minimizing linear swap regret -that is, the notion of Φ -regret where Φ contains all linear deviations. Such algorithms lead to an ϵ -equilibrium in time polynomial in the game's description and 1 /ϵ -aka. a fully polynomial-time approximation scheme ( FPTAS ).

Yet, virtually nothing was known beyond those special cases until recent breakthrough results by Dagan et al. [2024] and Peng and Rubinstein [2024], who introduced a new approach for reducing swap regret to external regret; unlike earlier reductions [Gordon et al., 2008, Blum and Mansour, 2007, Stoltz and Lugosi, 2005], their algorithm can be implemented efficiently even in certain settings with an exponential number of pure strategies. For extensive-form games, their reduction implies a polynomial-time approximation scheme ( PTAS ) for computing an ϵ -correlated equilibrium; their algorithm has complexity N ˜ O (1 /ϵ ) for games of size N , which is polynomial only when ϵ is an absolute constant. Unfortunately, it was thereafter shown that in the usual regime of interest, where instead ϵ ≤ poly (1 /N ) , an exponential number of rounds is inevitable even against an oblivious adversary [Daskalakis et al., 2024]. In light of that lower bound, our focus here is on developing algorithms attaining a better complexity bound of poly ( N, 1 /ϵ ) -the typical guarantee one hopes for within the no-regret framework-by considering a more structured but rich class of deviations Φ .

2 Preliminaries

Before we proceed by giving an overview of our results and technical contributions, we first introduce some basic background on tree-form decisions problems and Φ -regret minimization.

2.1 Tree-form decision problems

A tree-form decision problem describes a sequential interaction between a player and a (possibly adversarial) environment . There is a tree of nodes . The root is denoted ∅ . We will use s ∈ S to denote a generic node, and p s (where s = ∅ ) to denote the parent of s . Leaves are called terminal nodes ; a generic terminal node is denoted z ∈ Z . Internal nodes can be one of three types: decision points , where the player plays an action, observation points , where the environment picks the next decision point. A generic decision point will be denoted j , and the set of actions at j will be denoted A j . The child node reached by following action a ∈ A j is denoted ja . We will use N to denote the number of terminal nodes. We will also assume without loss of generality that all decision points have branching factor at least 2, and that decision and observation points alternate. Thus, the total number of nodes in the tree is also O ( N ) . The depth of a decision problem is the largest number of decision points in any root-to-terminal-node path. An example of a tree-form decision problem is depicted below in Figure 1.

̸

A

B

x

2

x

3

C

x

4

x

5

x

1

0

2

3

4

5

1

Figure 1: An example of a tree-form decision problem. Decision points are black squares with white text labels; observataion points are white squares. Edges are labeled with action names, which are numbers. Pure strategies in this decision problem are identified with vectors x = ( x 1 , x 2 , x 3 , x 4 , x 5 ) ∈ { 0 , 1 } 5 satisfying 1 -x 1 = x 2 + x 3 = x 4 + x 5 .

A pure strategy consists of an assignment of one action a j ∈ A j to each decision point j . The tree-form representation of the pure strategy is the vector x ∈ { 0 , 1 } N where x [ z ] = 1 if and only if the player plays all the actions on the ∅ → z path. Although x is a vector indexed only by terminal nodes, we also overload notation to write x [ s ] = 1 if and only if the player plays all actions on the ∅ → s path (In other words, x [ s ] = 1 if there exists some z ⪰ s with x [ z ] = 1 ). Multiple pure strategies can have the same tree-form representation, but in this paper we will only concern ourselves with strategies in tree-form representation, and thus for our purposes such strategies will be treated as identical. We will use X ⊆ { 0 , 1 } N to denote the set of tree-form strategies, and sometimes (when context is clear) we will also use X to denote the tree-form decision problem itself. For a point in the convex hull of X , conv X , we also use the symbol x ∈ conv X . For mixed strategies, we instead use π ∈ ∆( X ) . When it is relevant, we assume that utilities are rational numbers representable with poly ( N ) bits.

2.2 Regret minimization

In the framework of online learning, a learner interacts with an adversary over a sequence of rounds. In each round, the learner selects a strategy, whereupon the adversary constructs a utility function. Throughout this paper, we operate in the full feedback setting, wherein the learner gets to observe the entire utility function produced by the adversary after each round. We allow the adversary to be strongly adaptive , so that the (linear) utility function at the t th round u ( t ) : X ∋ x ↦→⟨ u ( t ) , x ⟩ can depend on the strategy of the learner at that round; this is a standard assumption ( cf. the notion of leaky forecasts in the context of calibration [Foster and Hart, 2018]) that will be used for our lower bound (Theorem 3.3). We assume that utilities belong to U := { u : |⟨ u , x ⟩| ≤ 1 , ∀ x ∈ X} . It will be convenient to use ∥ x ∥ X := max u ∈U ⟨ u , x ⟩ for the induced norm.

We measure the performance of an online learning algorithm as follows. Suppose that Φ ⊆ (conv X ) X is a set of deviations. If the learner outputs in each round a mixed strategy π ( t ) ∈ ∆( X ) ,

its (time-average) Φ -regret [Greenwald and Hall, 2003, Stoltz and Lugosi, 2007] is defined as

In the special case where Φ contains only constant transformations , one recovers the notion of external regret . On the other extreme, swap regret corresponds to Φ containing all functions X → X .

It is sometimes assumed that the learner instead selects in each round a strategy x ( t ) ∈ conv X . To translate (1) in that case, we introduce the extended mapping of a deviation ϕ : X → conv X as ϕ δ := E x ′ ∼ δ ( x ) [ ϕ ( x ′ )] , where δ : conv X → ∆( X ) is a function that is consistent in the sense that E x ′ ∼ δ ( x ) [ x ′ ] = x . A canonical example of such a function δ is the behavioral strategy map β : conv X → ∆( X ) , which returns the unique (ignoring actions at decision points reached with probability zero) mixed strategy whose actions at different decision points are independent and whose expectation is x . We give another example of a consistent map later in Appendix C.2. Accordingly, we let Φ δ denote all extended mappings. In this context, Φ δ -regret is defined as

We are interested in algorithms whose regret is bounded by ϵ after T = poly ( N, 1 /ϵ ) rounds. We refer to such algorithms as fully polynomial no-regret learners .

Remark 2.1 . Weclarify that all the algorithms we consider in this paper are deterministic , even when we allow mixed strategies. The fact that (1) contains an expectation over x ( t ) ∼ π ( t ) is simply how Φ -regret is defined; at no point does the algorithm actually sample from π ( t ) . Using deterministic algorithms is in line with most of the prior work in the full feedback setting.

3 Overview of our results

In this section, we present an overview of our results on parameterized algorithms for minimizing Φ -regret in extensive-form games. We shall first describe our results for the special case of Bayesian games with two actions per player, and we then treat general extensive-form games.

3.1 Bayesian games

For now, we assume that each player's strategy space is a hypercube { 0 , 1 } N . Hypercubes are linear transformations of tree-form decision problems; in particular, for Bayesian games in which each player has exactly two actions, the strategy space of every player is, up to linear transformations, a hypercube. Since our results are particularly clean for the hypercube case, we start with that.

First, we introduce the set of depthk decision tree deviations Φ k DT , which can be described as follows. For each of k ∈ N rounds, the deviator first elects a decision point and receives a recommendation, whereupon the deviator gets to decide which action to follow in that decision point. More formally, the set of deviations Φ k DT is defined as follows:

The deviator observes an index j 0 ∈ [ N ] .

For i = 1 , . . . , k : the deviator selects an index j i ∈ [ N ] , and observes x [ j i ] .

The deviator selects a 0 ∈ { 0 , 1 } .

We call attention to the order of operations. In particular, each query j is allowed to depend on previously observed x [ j ] s. We can assume (WLOG) that the deviator always chooses k distinct indices j . Now, the set of deviations ϕ : { 0 , 1 } N → [0 , 1] N that can be expressed in the above manner is precisely the set of functions representable as (randomized) depthk decision trees on N variables. To connect Φ k DT with the concepts referred to earlier, we clarify that k = 1 corresponds to linear-swap deviations, while k = N captures all possible swap deviations. Our first result is a parameterized online algorithm minimizing regret with respect to deviations in Φ k DT . (All our results are in the full feedback model under a strongly adaptive adversary.)

Theorem 3.1. There is an online algorithm incurring (average) Φ k DT -regret at most ϵ in N O ( k ) /ϵ 2 rounds with a per-round running time of N O ( k ) /ϵ .

Next, we consider the set Φ k poly consisting of all degreek polynomials ϕ : { 0 , 1 } N →{ 0 , 1 } N . Our result for this class of deviations mirrors the one for Φ k DT , but with a worse dependence on k .

Theorem 3.2. There is an online algorithm incurring Φ k poly -regret at most ϵ in N O ( k 3 ) /ϵ 2 rounds with a per-round running time of N O ( k 3 ) /ϵ .

We find those results surprising; we originally surmised that even for quadratic polynomials ( k = 2 ) the underlying online problem would be hard in the regime where ϵ ≤ poly (1 /N ) . We will elaborate on our technical approach for establishing those results in Section 4 coming up.

Hardness in behavioral strategies A salient aspect of the previous results, which was intentionally blurred above, is that the learner is allowed to output a mixed strategy -aprobability distribution over { 0 , 1 } N . In stark contrast, and perhaps surprisingly, when the learner is constrained to output behavioral strategies, that is to say, points in [0 , 1] N , we show that the problem immediately becomes PPAD -hard even for degree k = 2 (Theorem 3.3)-thereby being intractable under standard complexity assumptions. We are not aware of any such hardness results pertaining to a natural online learning problem, necessitating the use of mixed strategies.

The key connection behind our lower bound is an observation by Hazan and Kale [2007], which reveals that any Φ β -regret minimizer is inadvertedly able to compute approximate fixed points of any deviation in Φ β (Proposition B.1). Computing fixed points is in general a well-known (presumably) intractable problem, being PPAD -hard. In our context, the set Φ β does not contain arbitrary (Lipschitz continuous) functions [0 , 1] N → [0 , 1] N , but instead contains multilinear functions from [0 , 1] N to [0 , 1] N . To establish PPAD -hardness for our problem, we start with a generalized circuit (Definition I.3), and we show that all gates can be approximately simulated using exclusively gates involving multilinear operations (Proposition I.7); we defer the formal argument to Appendix I.1. As a result, we arrive at the following hardness result.

Theorem 3.3. If a regret minimizer R outputs strategies in [0 , 1] N , it is PPAD -hard to guarantee Reg Φ β ≤ ϵ/ √ N , even with respect to low-degree deviations and an absolute constant ϵ > 0 .

3.2 Extensive-form games

Wenext expand our scope to arbitrary extensive-form games. We will assume here that the branching factor b of the game is 2-any game can be transformed as such by incurring a log b factor overhead in the depth d of the game tree. Generalizing Φ k DT described above, we introduce the set of k -mediator deviations Φ k med . Informally, the player here has access to k distinct mediators, which the player can query at any time; a formal definition is given in Section 4. Once again, the case k = 1 corresponds to linear-swap deviations. Further, if X denotes the set of pure strategies, we let Φ k poly denote the set of all degreek deviations X → X . We establish similar parameterized results in extensive-form games, but which may now also depend on the depth of the game tree d .

Theorem 3.4. There is an online algorithm incurring at most an ϵ Φ k poly regret in N O ( kd ) 3 /ϵ 2 rounds with a per-round running time of N O ( kd ) 3 /ϵ . For Φ k med both bounds instead scale as N O ( k ) .

We recall that N here denotes the dimension of the strategy space. We further clarify that parameter k appearing in Φ k poly is different than the k in Φ k med : the former refers to the degree of a polynomial, while the latter is the number of mediators. As all k -mediator deviations are degreek polynomials (but not vice versa), it is to be expected that the bound in the theorem above concerning the former is worse. For a fixed degree k and assuming that the game tree is balanced , in the sense that d = polylog N , Theorem 3.4 guarantees a quasipolynomial complexity with respect to Φ k poly , even when ϵ is itself inversely quasipolynomial. The complexity we obtain for Φ k med is more favorable, being polynomial for any extensive-form game. 1 Finally, in light of the connection between noregret learning and convergence to correlated equilibria, our results imply parameterized tractability of the equilibrium concepts induced by Φ k med or Φ k poly (see Appendix F.1 for a formal treatment).

4 Technical contributions

From a technical standpoint, our starting point is the familiar template of Gordon et al. [2008] for minimizing Φ -regret, which consists of two key components. Accordingly, we split our technical overview into two parts.

4.1 Circumventing fixed points

The first key ingredient one requires in the framework of Gordon et al. [2008] is an algorithm for computing an approximate fixed point of any function within the set of deviations. In particular, if X is the set of pure strategies and conv X is the convex hull of X , we now work with functions Φ δ ∋ ϕ δ : conv X → conv X , so that fixed points exist by virtue of Brouwer's theorem. 2 As we discussed earlier, this fixed point computation is-at least in some sense-inherent: Hazan and Kale [2007] observed that minimizing Φ δ -regret is computationally equivalent to computing approximate fixed points of transformations in Φ δ . Specifically, an efficient algorithm minimizing Φ δ -regretwith respect to any sequence of utilities-can be used to compute an approximate fixed point of any transformation in Φ δ (Proposition B.1 in Appendix B). Given that functions in Φ δ are generally nonlinear, this brings us to PPAD -hard territory (Theorem 3.3), seemingly contradicting the recent positive results of Dagan et al. [2024] and Peng and Rubinstein [2024].

As we have alluded to, it turns out that there is a delicate precondition on the reduction of Hazan and Kale [2007] that makes all the difference: computing approximate fixed points is only necessary if the learner outputs points on conv X . In stark contrast, a crucial observation that drives our approach is that a learner who selects a probability distribution over X does not have to compute (approximate) fixed points of functions in Φ . Instead, we show that it is enough to determine what we refer to as an approximate fixed point in expectation . More precisely, for a deviation Φ ∋ ϕ : X → conv X with an efficient representation, it is enough to compute a distribution π ∈ ∆( X ) such that E x ∼ π ϕ ( x ) ≈ E x ∼ π x . It is quite easy to compute an approximate fixed point in expectation: take any x 1 ∈ conv X , and consider the sequence x 1 , . . . , x L ∈ conv X such that x ℓ +1 := E x ′ ℓ ∼ δ ( x ℓ ) ϕ ( x ′ ℓ ) for all ℓ , where δ : conv X → ∆( X ) is a mapping such that E x ′ ∼ δ ( x ) [ x ′ ] = x . 3 Then, for π := E ℓ ∈ [ L ] [ δ ( x ℓ )] , we have

This procedure can replace the fixed point oracle required by the template of Gordon et al. [2008], which is prohibitive when Φ contains nonlinear functions, as we formalize in Appendix C.

Application to faster computation of correlated equilibria In fact, even in normal-form games where considering linear deviations suffices, computing a fixed point is relatively expensive, amounting to solving a linear system, dominating the per-iteration complexity. Leveraging instead our new reduction, we obtain the fastest algorithm for computing an approximate correlated equilibrium in the moderate-precision regime (Corollary 4.1). In particular, let us focus for simplicity on n -player normal-form games with a succinct representation. Here, each player i ∈ [ n ] selects as strategy a probability distribution π i ∈ ∆( A i ) , where we recall that A i is a finite set of available actions. The expected utility of player i is given by u i ( π 1 , . . . , π n ) := E a 1 ∼ π 1 ,...,a n ∼ π n [ u i ( a 1 , . . . , a n )] , where u i : A 1 ×··· × A n → [ -1 , 1] . We assume that there is an expectation oracle that computes the vector

in time bounded by EO ( n, A ) , where A := max i |A i | ; it is known that EO ( n, A ) ≤ poly ( n, A ) for most interesting classes of succinct classes of games [Papadimitriou and Roughgarden, 2008]. Using our framework, we arrive at the following result.

Assuming that the oracle call to (2) ( EO ( n, A ) ) does not dominate the per-iteration running timewhich is indeed the case in, for example, polymatrix games-Corollary 4.1 gives (to our knowledge) the fastest algorithm for computing ϵ -correlated equilibria in the moderate-precision regime 1 /A ω 2 -1 ≤ ϵ ≤ 1 / log A , where ω ≈ 2 . 37 is the exponent of matrix multiplication [Williams et al., 2024]; without fast matrix multiplication, which is widely impractical, the lower bound instead reads ϵ ≥ 1 / √ A . We provide a comparison with previous algorithms in Table 1 and defer the details to Appendix I.3. Finally, we stress that similar improvements can be obtained beyond normal-form games using our template; indeed, virtually all prior Φ -regret minimizers rely on some fixed point operation.

Before moving on, it is worth stressing that the discrepancy that has arisen between operating over ∆( X ) versus conv X is quite singular when it comes to regret minimization in extensive-form games and beyond. Kuhn's theorem [Kuhn, 1953] is often invoked to argue about their equivalence, but in our setting it is the nonlinear nature of deviations in Φ that invalidates that equivalence. 4 To tie up the loose ends, we adapt the reduction of Hazan and Kale [2007] to show that minimizing Φ -regret over ∆( X ) necessitates computing approximate fixed points in expectation (Proposition C.3), and we observe that the reductions of Dagan et al. [2024] and Peng and Rubinstein [2024] are indeed compatible with computing approximate fixed points in expectation; the latter observation is made precise in Appendix F.3.

4.2 Regret minimization over the set of deviations Φ

The second ingredient prescribed by Gordon et al. [2008] is an algorithm minimizing external regret but with respect to the set of deviations Φ . The crux in this second step lies in the fact that, even in normal-form games, Φ contains at least an exponential number of deviations, so black-box reductions are of little use here. Instead, the problem boils down to appropriately leveraging the combinatorial structure of Φ , as we explain below.

Wewill first describe our approach when X = { 0 , 1 } N , and we then proceed with the more technical generalization to extensive-form games. The key observation here is that regret minimization over Φ k DT can be viewed as a tree-form decision problem of size N O ( k ) . Terminal nodes in this decision problem are identified by the original index j 0 ∈ [ N ] , the queries j 1 , . . . , j k ∈ [ N ] , their replies a 1 , . . . , a k ∈ { 0 , 1 } , and finally the action a 0 ∈ { 0 , 1 } that is played. Each tree-form strategy q in this decision problem defines a function ϕ q : X → conv X , which is computed by following the

strategy q through the decision problem. Formally, we have

where x [ j i , a i ] = x [ j i ] if a i = 1 , and 1 -x [ j i ] if a i = 0 . Hence ϕ q is a degreek polynomial in x .

Now, since q ↦→ ϕ q ( x )[ i ] is linear, it follows that q ↦→ ⟨ u , ϕ q ( x ) ⟩ is also linear for any given u ∈ R n . Therefore, a regret minimizer on Φ k DT can be constructed starting from any regret minimizer for tree-form decision problems; for example, counterfactual regret minimization [Zinkevich et al., 2007], or any of its modern variants. This enables us to rely on usual techniques for dealing with such problems, eventually leading to a complexity bound of N O ( k ) , as we formalize in Appendix D.

For the set of low-degree polynomials Φ k poly , we leverage a result from Boolean analysis relating (randomized) low-depth decision trees with low-degree polynomials, stated below.

Theorem 4.2 (Midrijanis, 2004) . Every degreek polynomial f : { 0 , 1 } N →{ 0 , 1 } can be written as a decision tree of depth at most 2 k 3 .

In particular, this implies that Φ k poly ⊆ Φ 2 k 3 DT . Consequently, low-degree polynomials can be reduced to low-depth decision trees, albeit with an overhead in the exponent.

Turning to general extensive-form games, we follow a similar blueprint, although there are now additional technical challenges. In particular, in what follows, to describe the set of deviations it will be convenient to introduce a new formalism related to tree-form decision problems.

Definition 4.3. The dual ¯ X of X is the decision problem identical to X , except that the decision points and observation points have been swapped.

Definition 4.4. The interleaving X ⊗Y is the tree-form decision problem defined as follows. There is a state s = ( s 1 , s 2 ) ∈ S 1 ×S 2 . The root state is the tuple ( ∅ , ∅ ) . The decision problem is defined by the player being able to interact with both decision problems, in the following manner. At each state s = ( s 1 , s 2 ) :

If s 1 and s 2 are both terminal then so is s . Otherwise:

If either of the s i s is an observation point, then so is s . The children are the states ( s ′ i , s -i ) where s ′ i is a child of s i . (If both s i s are observation points, both children s ′ 1 , s ′ 2 are selected simultaneously. This can only happen at the root.)

Otherwise, s is a decision point. The player selects an index i ∈ { 1 , 2 } at which to act, and a child s ′ i to transition to. The next state is ( s ′ i , s -i ) .

In X ⊗ Y , the same state ( s 1 , s 2 ) can be reachable through possibly exponentially many paths, because the learner may choose to interleave actions in X with actions in Y in any order. Thus, each state ( s 1 , s 2 ) corresponds to actually exponentially many histories in X⊗Y . In the discussion below, we will therefore carefully distinguish between histories and states . In light of the above exponential gap between histories and states, it seems wasteful to represent X ⊗Y as a tree. Indeed, Zhang et al. [2023] recently studied DAG -form decision problems, and showed that regret minimization on them is possible so long as the DAG obeys some natural properties.

Using the language we have now introduced, we can define the set of k -mediator deviations Φ k med as the set of reduced strategies in the decision problem X ⊗ ¯ X ⊗ k . That is, the player has access to not one but k mediators, all holding strategy x , which the player can query at any time. This is a significant advantage over having just one mediator since the player can send different queries to each of the k mediators (who must all reply according to x ), and therefore can learn more about the strategy x than it could have otherwise. We will call the responses given by the mediator action recommendations . For a graphical illustration of such deviations, we refer to Figure 2 (in Appendix E).

Reduced strategies q ∈ π ( X ⊗ ¯ X ⊗ k ) , once again, induce functions ϕ q : X → conv X given by

and in particular we have that ϕ q is a degreek polynomial. We define Φ k med as the set of such deviations. For that set, we show that there is a reduction to a particular type of DAG-form decision problem of size N O ( k ) . As we explained, that formulation is more suitable than tree-form decision problems when the number of possible histories far exceeds the number of states, which is precisely the case when the player is gradually querying multiple mediators as the game progresses.

Finally, we establish a reduction from low-degree polynomials to having few mediators; namely, we show that Φ k poly ⊆ Φ O ( kd ) 3 med , where we recall that d is the depth of the game tree. Our basic strategy is to again leverage the connection between low-depth decision trees and low-degree polynomials we described earlier (Theorem 4.2). To do so, we need to cast our problem in terms of functions { 0 , 1 } N →{ 0 , 1 } N instead of X → X . To that end, we first show how to extend a degreek function f : X → { 0 , 1 } to a degreekd function ¯ f : { 0 , 1 } N → { 0 , 1 } ; that is, ¯ f coincides with f on all points in X ⊆ { 0 , 1 } N (Lemma E.7). This step is where the overhead factor d comes from. The final technical piece is to show that if each component of ϕ : X → X can be expressed using K mediators, the same holds for ϕ ; the naive argument here incurs another factor of d , but we show that this is in fact not necessary. The details of the above argument are deferred to Appendix E.

5 Further related research

Akey reference point is the result of Blum and Mansour [2007], and a generalization due to Gordon et al. [2008], which reduces minimizing swap regret to minimizing external regret. Specifically, for the probability simplex ∆( A ) , it maintains a separate external-regret minimizer, one for each action a ∈ A . Both the per-iteration complexity and the number of iterations required is generally polynomial in A := |A| . Therefore, in settings where A is exponentially large in the natural parameters of the problem (such as extensive-form games) it does not appear that the reduction of Blum and Mansour [2007] is of much use. It is tempting to instead rely on the reduction of Stoltz and Lugosi [2005] for minimizing internal regret, a weaker notion than swap regret, which is nonetheless sufficient for (asymptotic) convergence to correlated equilibria. However, one should be careful when relying on internal regret in settings where A is exponentially large; as we point out in Remark A.1, internal regret can be smaller than swap regret by up to a factor of A , so it is only meaningful when ϵ ≤ 1 /A , a regime which is generally out of reach for regret minimization techniques when A is exponentially large.

This gap motivated the new reduction by Dagan et al. [2024] and Peng and Rubinstein [2024], which we discussed earlier. Beyond extensive-form games, those reductions apply whenever it is possible to minimize external regret efficiently. The complexity of computing correlated equilibria beyond the regime where the precision parameter ϵ is an absolute constant remains a major open problem, generally conjectured to be hard [von Stengel and Forges, 2008]; the recent online lower bound in the adversarial setting [Daskalakis et al., 2024] provides further evidence in support of that conjecture.

As a result, most prior work has focused on more permissive equilibrium concepts, understood through the framework of Φ -regret [Morrill et al., 2021a,b, Farina et al., 2022, Bai et al., 2022, Bernasconi et al., 2023, Noarov et al., 2023, Dud´ ık and Gordon, 2009, Gordon et al., 2008, Fujii, 2023, Dann et al., 2023, Mansour et al., 2022, Sharma, 2024, Cai et al., 2024a]). In terms of the most recent developments, Farina and Pipis [2023] established efficient learning dynamics minimizing what is referred to as linear swap regret ( cf. Dann et al. [2023], Fujii [2023] for related results in Bayesian games). The solution concept that arises from linear swap regret was later endowed with a natural mediator-based interpretation by Zhang et al. [2024], which can be viewed as a natural precursor to this work. Convergence to correlated equilibria has also attracted attention in the context of Markov (aka. stochastic) games ( e.g. , [Cai et al., 2024b, Jin et al., 2021, Erez et al., 2023, Liu and Zhang, 2023], and references therein).

Moreover, as we explained earlier, our approach also gives rise to a faster algorithm for computing approximate correlated equilibria in a certain regime. As we discuss further in Appendix I.3, improving the per-iteration complexity of Blum and Mansour [2007] has received interest in prior work [Ito, 2020, Greenwald et al., 2006, Yang and Mohri, 2017] (see also [Huang and Pan, 2023, Huang et al., 2023]). The main bottleneck lies in the (approximate) computation of a stationary distribution of a Markov chain, which can be phrased as a linear system. It is worth noting that solving linear systems faster than matrix multiplication even for a crude approximation is precluded, at least

subject to fine-grained complexity assumptions [Bafna and Vyas, 2021]; we are not aware whether such hardness results are also known for computing the stationary distribution of a Markov chain.

Finally, although we have so far mostly directed our attention to the game-theoretic implication of minimizing swap (or indeed Φ ) regret, namely the celebrated connection with correlated equilibria in repeated games, the notion of swap regret is a fundamental solution concept in its own right more broadly in online learning and learning theory. Compared to the more common notion of external regret, swap regret gives rise to a more appealing notion of hindsight rationality; as such, it is often adopted as a behavioral assumption to model learning agents ( e.g. , [Deng et al., 2019]). It is also fundamentally tied to the notion of calibration [Hu and Wu, 2024], and recently inspired work by Gopalan et al. [2023] in the context of multi-group fairness.

6 Conclusions and future research

We provided a new family of parameterized algorithms for minimizing Φ -regret in extensive-form games. Our results capture perhaps the most natural class of functions interpolating between linearswap and swap deviations, namely degreek deviations. Along the way, we refined the usual template for minimizing Φ -regret-taught in many courses on algorithmic game theory and online learning-which revolves around (approximate) fixed points [Gordon et al., 2008, Blum and Mansour, 2007, Stoltz and Lugosi, 2005]. Instead, we showed that it suffices to rely on a relaxation that we refer to as an approximate fixed point in expectation, which-unlike actual fixed points-can always be computed efficiently. Our refinement of the usual template for minimizing Φ -regret is of independent interest beyond extensive-form games. For example, it can speed up the computation of approximate correlated equilibria even in normal-form games, as it obviates the need to solve a linear system in every round. As in the recent works by Dagan et al. [2024] and Peng and Rubinstein [2024], a crucial feature of our approach is to allow the learner to select a distribution over pure strategies, for otherwise we showed that regret minimization immediately becomes PPAD -hard (under a strongly adaptive adversary).

There are many interesting avenues for future research. First, the complexity of our algorithm pertaining to degreek deviations depends exponentially on the depth of the game tree. We suspect that such a dependency could be superfluous. To show this, it would be enough to refine Lemma E.7 by coming up with an extension whose degree does not depend on the depth of the game tree. It would also be interesting to devise parameterized algorithms for k -mediator deviations that recover as a special case the PTAS of Peng and Rubinstein [2024] and Dagan et al. [2024], so as to smoothly interpolate between existing results for linear-swap regret [Farina and Pipis, 2023] and the aforementioned results for swap regret; is k = ˜ O (1 /ϵ ) enough to capture swap regret?

Finally, perhaps the most important question is to understand the computational complexity of computing Φ -equilibria in extensive-form games. In particular, our results raise the interesting question of whether there is an algorithm (in the centralized model) for computing in polynomial time an exact correlated equilibrium induced by low-degree deviations. Extending the paradigm of Papadimitriou and Roughgarden [2008] in that setting presents several challenges, not least because computing fixed points-which are crucial for implementing the separation oracle [Papadimitriou and Roughgarden, 2008]-is now computationally hard. Relatedly, we suspect that there is an inherent connection between fixed points and correlated equilibria, in the spirit of the equivalence between Φ -regret minimization and fixed points established by Hazan and Kale [2007].

References

Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. Near-optimal no-regret learning for correlated equilibria in

multi-player general-sum games. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, 2022 , pages 736-749. ACM, 2022.

Robert Aumann. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics , 1:67-96, 1974.

Yakov Babichenko, Christos H. Papadimitriou, and Aviad Rubinstein. Can almost everybody be almost happy? In Conference on Innovations in Theoretical Computer Science (ITCS) , 2016.

Mitali Bafna and Nikhil Vyas. Optimal fine-grained hardness of approximation of linear equations. In International Colloquium on Automata, Languages, and Programming, (ICALP) , pages 20:120:19, 2021.

Yu Bai, Chi Jin, Song Mei, Ziang Song, and Tiancheng Yu. Efficient phi-regret minimization in extensive-form games via online mirror descent. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Francesco Trov` o, and Nicola Gatti. Constrained phi-equilibria. In International Conference on Machine Learning (ICML) , 2023.

Avrim Blum and Yishay Mansour. From external to internal regret. J. Mach. Learn. Res. , 8:13071324, 2007.

Yang Cai, Constantinos Daskalakis, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Tractable local equilibria in non-concave games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2024a.

Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Near-optimal policy optimization for correlated equilibrium in general-sum markov games. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2024b.

Nicolo Cesa-Bianchi and G´ abor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player Nash equilibria. Journal of the ACM , 2009.

Xinyi Chen, Angelica Chen, Dean Foster, and Elad Hazan. Ai safety by debate via regret minimization, 2023.

Michael B. Cohen, Jonathan A. Kelner, John Peebles, Richard Peng, Anup B. Rao, Aaron Sidford, and Adrian Vladu. Almost-linear-time algorithms for markov chains and new spectral primitives for directed graphs. In Proceedings of the Annual Symposium on Theory of Computing (STOC) , pages 410-419. ACM, 2017.

Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. From external to swap regret 2.0: An efficient reduction and oblivious adversary for large action spaces. Proceedings of the Annual Symposium on Theory of Computing (STOC) , 2024.

Christoph Dann, Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Pseudonorm approachability and applications to regret minimization. In International Conference on Algorithmic Learning Theory (ALT) , 2023.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 27604-27616, 2021.

Constantinos Daskalakis, Gabriele Farina, Noah Golowich, Tuomas Sandholm, and Brian Hu Zhang. Exponential lower bounds for minimizing swap regret in extensive-form games, 2024.

Argyrios Deligkas, John Fearnley, Alexandros Hollender, and Themistoklis Melissourgos. Purecircuit: Strong inapproximability for PPAD. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , pages 159-170, 2022.

Yuan Deng, Jon Schneider, and Balasubramanian Sivan. Strategizing against no-regret learners. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 1577-1585, 2019.

Miroslav Dud´ ık and Geoffrey J. Gordon. A sampling-based approach to computing equilibria in succinct extensive-form games. In Jeff A. Bilmes and Andrew Y. Ng, editors, UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , pages 151-160. AUAI Press, 2009.

Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning (ICML) , 2023.

Gabriele Farina and Charilaos Pipis. Polynomial-time linear-swap regret minimization in imperfectinformation sequential games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Gabriele Farina, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Simple uncoupled no-regret learning dynamics for extensive-form correlated equilibrium. Journal of the ACM , 69(6), 2022.

Aris Filos-Ratsikas, Yiannis Giannakopoulos, Alexandros Hollender, Philip Lazos, and Diogo Poc ¸as. On the complexity of equilibrium computation in first-price auctions. SIAM Journal on Computing , 52(1):80-131, 2023.

Dean Foster and Rakesh Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior , 21:40-55, 1997.

Dean P. Foster and Sergiu Hart. Smooth calibration, leaky forecasts, finite recall, and nash dynamics. Games and Economic Behavior , 109:271-293, 2018.

Kaito Fujii. Bayes correlated equilibria and no-regret dynamics. CoRR , abs/2304.05005, 2023.

Anat Ganor and Karthik C. S. Communication complexity of correlated equilibrium with small support. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM) , 2018.

Paul W. Goldberg and Aaron Roth. Bounds for the query complexity of approximate equilibria. ACM Trans. Economics and Comput. , 4(4):24:1-24:25, 2016.

Parikshit Gopalan, Michael P. Kim, and Omer Reingold. Swap agnostic learning, or characterizing omniprediction via multicalibration. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Geoffrey J Gordon, Amy Greenwald, and Casey Marks. No-regret learning in convex games. In International Conference on Machine Learning (ICML) , 2008.

Amy Greenwald and Keith Hall. Correlated Q-learning. In International Conference on Machine Learning (ICML) , 2003.

Amy Greenwald, Zheng Li, and Casey Marks. Bounds for regret-matching algorithms. In ISAIM , 2006.

Amy Greenwald, Zheng Li, and Warren Schudy. More efficient internal-regret-minimizing algorithms. In Conference on Learning Theory (COLT) , pages 239-250, 2008.

Martin Gr¨ otschel, L´ aszl´ o Lov´ asz, and Alexander Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica , 1:169-197, 1981.

Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68:1127-1150, 2000.

Elad Hazan and Satyen Kale. Computational equivalence of fixed points and no regret algorithms, and convergence to equilibria. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2007.

Lunjia Hu and Yifan Wu. Predict to minimize swap regret for all payoff-bounded tasks. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , 2024.

Wan Huang and Bernhard von Stengel. Computing an extensive-form correlated equilibrium in polynomial time. In International Workshop On Internet And Network Economics (WINE) , 2008.

Zhiming Huang and Jianping Pan. A near-optimal high-probability swap-regret upper bound for multi-agent bandits in unknown general-sum games. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) , 2023.

Zhiming Huang, Kaiyang Liu, and Jianping Pan. End-to-end congestion control as learning for unknown games with bandit feedback. In International Conference on Distributed Computing Systems (ICDCS) , 2023.

Shinji Ito. A tight lower bound and efficient reduction for swap regret. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Albert Jiang and Kevin Leyton-Brown. Polynomial-time computation of exact correlated equilibrium in compact games. In Proceedings of the ACM Conference on Electronic Commerce (EC) , 2011.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning - A simple, efficient, decentralized algorithm for multiagent RL. CoRR , abs/2110.14555, 2021.

W. Kuhn. Extensive games and the problem of information. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games , volume 2 of Annals of Mathematics Studies, 28 , pages 193-216. Princeton University Press, Princeton, NJ, 1953.

Nicolas S. Lambert, Adrian Marple, and Yoav Shoham. On equilibria in games with imperfect recall. Games and Economic Behavior , 113:164-185, 2019.

Xiangyu Liu and Kaiqing Zhang. Partially observable multi-agent RL with (quasi-)efficiency: The blessing of information sharing. In International Conference on Machine Learning (ICML) , pages 22370-22419, 2023.

Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in bayesian games. In Conference on Learning Theory (COLT) , 2022.

Gatis Midrijanis. Exact quantum query complexity for total boolean functions. arXiv preprint quant-ph/0403168 , 2004.

Dustin Morrill, Ryan D'Orazio, Marc Lanctot, James R. Wright, Michael Bowling, and Amy R. Greenwald. Efficient deviation types and learning for hindsight rationality in extensive-form games. In International Conference on Machine Learning (ICML) , 2021a.

Dustin Morrill, Ryan D'Orazio, Reca Sarfati, Marc Lanctot, James R. Wright, Amy R. Greenwald, and Michael Bowling. Hindsight and sequential rationality of correlated play. In Conference on Artificial Intelligence (AAAI) , 2021b.

John Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences , 36:48-49, 1950.

Georgy Noarov, Ramya Ramalingam, Aaron Roth, and Stephan Xie. High-dimensional prediction for sequential decision making. CoRR , abs/2310.17651, 2023.

Ryan O'Donnell. Analysis of boolean functions . Cambridge University Press, 2014.

Christos H. Papadimitriou and Tim Roughgarden. Computing correlated equilibria in multi-player games. Journal of the ACM , 55(3):14:1-14:29, 2008.

Binghui Peng and Aviad Rubinstein. Fast swap regret minimization and applications to approximate correlated equilibria. Proceedings of the Annual Symposium on Theory of Computing (STOC) , 2024.

Michele Piccione and Ariel Rubinstein. On the interpretation of decision problems with imperfect recall. Games and Economic Behavior , pages 3-24, 1997.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Beyond regret. In Conference on Learning Theory (COLT) , 2011.

Aviad Rubinstein. Settling the complexity of computing approximate two-player nash equilibria. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , 2016.

Dravyansh Sharma. No internal regret with non-convex loss functions. In Conference on Artificial Intelligence (AAAI) , 2024.

Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations . Cambridge University Press, 2009.

Gilles Stoltz and G´ abor Lugosi. Internal regret in on-line portfolio selection. Machine Learning , 59 (1-2):125-159, 2005.

Gilles Stoltz and G´ abor Lugosi. Learning correlated equilibria in games with compact sets of strategies. Games and Economic Behavior , 59(1):187-208, 2007.

Emanuel Tewolde, Caspar Oesterheld, Vincent Conitzer, and Paul W. Goldberg. The computational complexity of single-player imperfect-recall games. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , 2023.

Bernhard von Stengel and Franc ¸oise Forges. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research , 33(4):1002-1022, 2008.

Virginia Vassilevska Williams, Yinzhan Xu, Zixuan Xu, and Renfei Zhou. New bounds for matrix multiplication: from alpha to omega. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2024.

Scott Yang and Mehryar Mohri. Online learning with transductive regret. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , pages 5214-5224, 2017.

Brian Hu Zhang, Gabriele Farina, and Tuomas Sandholm. Team belief DAG: generalizing the sequence form to team games for fast computation of correlated team max-min equilibria via regret minimization. In International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 40996-41018. PMLR, 2023.

Brian Hu Zhang, Gabriele Farina, and Tuomas Sandholm. Mediator interpretation and faster learning algorithms for linear correlated equilibria in general extensive-form games. In International Conference on Learning Representations (ICLR) , 2024.

Martin Zinkevich, Michael Bowling, Michael Johanson, and Carmelo Piccione. Regret minimization in games with incomplete information. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2007.

A Further preliminaries

In this section, we introduce some further preliminaries. For additional background, we refer the interested reader to the excellent books of Cesa-Bianchi and Lugosi [2006] and Shoham and LeytonBrown [2009]. Before we describe more formally the construction of Gordon et al. [2008], we make a remark regarding minimizing internal regret in extensive-form games.

Remark A.1 (Swap versus internal regret) . When it comes to defining correlated equilibria in normal-form games, there are two prevalent definitions appearing in the literature; one is based on internal regret , while the other on swap regret ( e.g. , [Ganor and Karthik C. S., 2018, Goldberg and Roth, 2016]). The key difference is that internal regret only contains deviations that swap a single action -thereby being weaker. Nevertheless, it is not hard to see that swap regret can only be larger by a factor of |X| [Blum and Mansour, 2007], where we recall that X denotes the set of pure strategies. So, in normal-form games those two definitions are polynomially equivalent, and in most applications one can safely switch from one to the other.

However, this is certainly not the case in games with an exponentially large action space, such as extensive-form games. In fact, the definition of internal regret itself is problematic when the action set is exponentially large: the uniform distribution always attains an error of at most 1 / |X| . Consequently, any guarantee for ϵ ≥ 1 / |X| is vacuous. That is, if |X| is exponentially large, an algorithm that requires a number of iterations polynomial in 1 /ϵ -which is what we expect to get from typical no-regret dynamics-would need an exponential number of iterations to yield a nontrivial guarantee; this issue with internal regret was also observed by Fujii [2023]. Nevertheless, internal regret in the context of games with an exponentially large action set was used in a recent work by Chen et al. [2023], who provided oracle-efficient algorithms for minimizing internal regret.

A.1 The construction of Gordon et al. [2008]

Gordon et al. [2008], building on earlier work by Blum and Mansour [2007] and Stoltz and Lugosi [2005], came up with a general recipe for minimizing Φ δ -regret. That construction relies on a noregret learning algorithm on the set of deviations Φ δ , which we denote by R Φ . Then, a Φ δ -regret minimizer on conv X can be constructed as follows: on each iteration t = 1 , . . . , T , the learner performs the following steps.

Receive ϕ ( t ) from R Φ . Select x ( t ) ∈ conv X as an ϵ -fixed point of ϕ ( t ) : ∥ ϕ ( t ) ( x ( t ) ) -x ( t ) ∥ X ≤ ϵ .

Upon receiving utility u ( t ) ∈ U , pass utility Φ δ ∋ ϕ δ ↦→ 〈 u ( t ) , ϕ δ ( x ( t ) ) 〉 to R Φ .

The main guarantee regarding the above algorithm is summarized below.

Theorem A.2 (Gordon et al., 2008) . Suppose that Reg T is the external regret incurred by R Φ . After T rounds of the above algorithm, we have

In Appendix C.1, we will relax the requirement of needing (approximate) fixed points, while at the same time maintaining the guarantee of Theorem A.2.

B Hardness of minimizing Φ -regret in behavioral strategies

In this section, we show that if the learner is constrained to output in reach round a strategy in conv X , then there is no efficient algorithm (under standard complexity assumptions) minimizing Φ β -regret (Theorem 3.3); here, β : conv X → ∆( X ) is the behavioral strategy mapping (introduced in the sequel as Definition C.5), the expression of which is not important for the purpose of this section. The key connection is a result by Hazan and Kale [2007], showing that any Φ β -regret minimizer is able to compute approximate fixed points of any deviations in Φ β . We then show that the set of induced deviations, even on the hypercube X = { 0 , 1 } N , is rich enough to approximate PPAD -hard fixed-point problems.

In this context, consider a transformation Φ β ∋ ϕ β : [0 , 1] N → [0 , 1] N for which we want to compute an approximate fixed point x ∈ conv X ; that is, ∥ ϕ β ( x ) -x ∥ 2 ≤ ϵ , for some precision parameter ϵ > 0 . (It is convenient in the construction below to measure the fixed-point error with respect to ∥ · ∥ 2 .) Hazan and Kale [2007] observed that a Φ β -regret minimizer can be readily turned into an algorithm for computing fixed points of any function in Φ β , as stated formally below. Before we proceed, we remind that here and throughout we operate under a strongly adaptive adversary, which is quite crucial in the construction of Hazan and Kale [2007].

Proposition B.1 (Hazan and Kale, 2007) . Consider a regret minimizer R operating over [0 , 1] N . If R runs in time poly ( N, 1 /ϵ ) and guarantees Reg T Φ β ≤ ϵ for any sequence of utilities, then there is a poly ( N, 1 /ϵ ) algorithm for computing an ( ϵ √ N ) -fixed point of any ϕ β ∈ Φ β with respect to ∥ · ∥ 2 , assuming that ϕ β can be evaluated in polynomial time.

Proposition B.1 significantly circumscribes the class of problems for which efficient Φ β -regret minimization is possible, at least when operating in behavioral strategies. Indeed, computing fixed points is in general a well-known (presumably) intractable problem. In our context, the set Φ β does not contain arbitrary (Lipschitz continuous) functions [0 , 1] N → [0 , 1] N , but instead contains multilinear functions from [0 , 1] N to [0 , 1] N . Nonetheless, we show that PPAD -hardness persists in our setting. The basic idea is as follows. We start with a generalized circuit (Definition I.3), and we show that all gates can be approximately simulated using exclusively gates involving multilinear operations (Proposition I.7). The proof of that claim appears in Appendix I.1. As a result, we arrive at the main hardness result of this section, restated below.

Theorem 3.3. If a regret minimizer R outputs strategies in [0 , 1] N , it is PPAD -hard to guarantee Reg Φ β ≤ ϵ/ √ N , even with respect to low-degree deviations and an absolute constant ϵ > 0 .

We also obtain a stronger hardness result under a stronger complexity assumption put forward by Babichenko et al. [2016] (Theorem I.9). At first glance, it may seem that the above results are at odds with the recent positive results of Dagan et al. [2024] and Peng and Rubinstein [2024], which seemingly obviate the need to compute approximate fixed points. As we have alluded to, the key restriction that drives Theorem 3.3 lies in constraining the learner to output behavioral strategies. In the coming section, we show that there is an interesting twist which justifies the discrepancy highlighted above.

C Circumventing fixed points

The previous section, and in particular Theorem 3.3, seems to preclude the ability to minimize Φ -regret efficiently when the set of (extended) deviations contains nonlinear functions. 5 In this section, we will show how to circumvent this issue via a relaxed notion of what constitutes a fixed point (Definition C.1). In the sequel, we will work with deviations ϕ with domain X instead of conv X .

C.1 Approximate expected fixed points

The key to our construction is to allow the learner to play distributions over X , not merely points in conv X , and to use a relaxed notion of a fixed point, formally introduced below.

Definition C.1. We say that a distribution π ∈ ∆( X ) is an ϵ -expected fixed point of ϕ ∈ (conv X ) X if ∥ E x ∼ π [ ϕ ( x ) -x ] ∥ X ≤ ϵ .

The key now is to replace the fixed point oracle in the framework of Gordon et al. [2008] (recalled in Appendix A) with an oracle that instead returns an ϵ -fixed point in expectation per Definition C.1. The learner otherwise proceeds as in the algorithm of Gordon et al. [2008] (our overall construction is spelled out as Algorithm 1 in Appendix I.2). It is easy to show, following the proof of Gordon et al. [2008], that a fixed point in expectation is still sufficient to minimize Φ -regret.

Theorem C.2 ( Φ -regret with ϵ -expected fixed points) . Suppose that the external regret of R Φ over Φ after T repetitions is at most Reg T . Then, the Φ -regret of Algorithm 1 can be bounded as Reg T + ϵ .

Analogously to Proposition B.1, it turns out that there is a certain equivalence between minimizing Φ in ∆( X ) and computing expected fixed points:

Proposition C.3. Consider a regret minimizer R operating over ∆( X ) . If R runs in time poly ( N, 1 /ϵ ) and guarantees Reg T Φ ≤ ϵ for any sequence of utilities, then there is a poly ( N, 1 /ϵ ) algorithm for computing ( ϵD X ) -expected fixed points of ϕ ∈ Φ , assuming that we can efficiently compute E x ( t ) ∼ π ( t ) [ ϕ ( x ( t ) ) -x ( t ) ] at any time t . Here, D X is the diameter of X with respect to ∥ · ∥ 2 .

The proof proceeds similarly to Proposition B.1, and so we include it in Appendix I.2. Next, we present a method for computing approximate expected fixed points of functions ϕ ∈ Φ without having to solve a PPAD -hard problem.

C.2 Extending deviation maps to conv X

First, since we will work both over conv X and distributions in ∆( X ) , we need efficient methods for passing between them. To that end, we introduce the following notion.

Definition C.4. A map δ : conv X → ∆( X ) is

consistent if E x ′ ∼ δ ( x ) x ′ = x , and

efficient if, given some ϕ ∈ Φ and x ∈ conv X , it is easy to compute ϕ δ ( x ) := E x ′ ∼ δ ( x ) ϕ ( x ′ ) .

We will call the map ϕ δ : conv X → conv X the extended map of ϕ .

One may ask why we use this indirect method of defining ϕ δ rather than simply directly using the representation of ϕ (for example, as a polynomial) to extend ϕ to conv X . The answer is that, even assuming that ϕ : X → X is represented as a multilinear polynomial (which is the representation assumed in the majority of this paper), naively extending that polynomial to domain conv X will not necessarily result in a function ¯ ϕ : conv X → conv X . For an example, consider the decision problem X depicted in Figure 1, and consider the function ϕ : X → X given by ϕ ( x ) = ( x 1 + x 3 , x 2 x 4 , x 2 x 5 , x 2 , 0) . One can easily check by hand that ϕ is indeed a function X → X , but also that, for the strategy x = (1 / 2 , 1 / 2 , 0 , 1 / 2 , 0) ∈ conv X , we have ϕ ( x ) = (1 / 2 , 1 / 4 , 0 , 1 / 2 , 0) / ∈ conv X . Thus, we need a more robust way of extending functions X → conv X to functions conv X → conv X , ideally one that is dependent only the function ϕ , not its representation.

We now give two methods of constructing consistent and efficient maps δ : conv X → ∆( X ) for tree-form strategy sets X . The first is the behavioral strategy map.

Definition C.5. The behavioral strategy map β : conv X → ∆( X ) is defined as follows: β ( x ) is the distribution of pure strategies generated by sampling, at each decision point j for which x [ j ] > 0 , an action a according to the probabilities x [ ja ] / x [ j ] . Formally,

It is possible for ϕ β to be not a polynomial even when ϕ is a polynomial, because β is itself not a polynomial. It is clear that β is consistent. For efficiency, we show the following claim.

Proposition C.6. Let β : conv X → ∆( X ) be the behavioral strategy map. Let ϕ : X → conv X be expressed as a polynomial of degree at most k , in particular, as a sum of at most O ( N k ) terms. Then there is an algorithm running in time N O ( k ) that, given ϕ and x ∈ conv X , computes ϕ β ( x ) .

̸

Proof. To compute E x ′ ∼ β ( x ) ϕ ( x ′ ) , since ϕ is a polynomial, it suffices to compute E x ′ ∼ β ( x ) m ( x ′ ) for multilinear monomials m of degree at most K , that is, functions of the form m S ( x ) := ∏ z ∈ S x [ z ] where S ⊆ Z has size at most k . There are two cases. First, there are monomials that are clearly identically zero: in particular, if there are two nodes ja, ja ′ ⪯ S for a = a ′ , then m S ≡ 0 because a player cannot play two different actions at j . For monomials that are not identically zero, we have

which is computable in time O ( kd ) . Thus, the overall time complexity is O ( kdN k ) ≤ N O ( k ) .

The behavioral strategy map is in some sense the canonical strategy map: when one writes a treeform strategy x ∈ conv X without further elaboration on what distribution ∆( X ) it is meant to represent, it is often implicitly or explicitly assumed to mean the behavioral strategy.

The behavioral strategy map has the unfortunate property that it usually outputs distributions of exponentially-large support; indeed, if x ∈ relint conv X then β ( x ) is fullsupport.

The second example we propose, which we call a Carath´ eodory map , always outputs low-support distributions. In particular, for any x ∈ conv X , Carath´ eodory's theorem on convex hulls guarantees that x is a convex combination of N pure strategies 6 x 1 , . . . , x N ∈ X . Gr¨ otschel et al. [1981, Theorem 3.9] moreover showed that there exists an efficient algorithm for computing the appropriate convex combination. Thus, fixing some efficient algorithm for this computational problem, we define a Carath´ eodory map γ : conv X → ∆( X ) to be any consistent map that returns a distribution of support at most N . Given such a mapping, computing ϕ γ ( x ) is easy: one simply writes x = ∑ i α i x i by computing γ ( x ) , and returns ϕ γ ( x ) = ∑ i α i ϕ ( x i ) . This only requires a poly ( N ) -time computation of γ , and N evaluations of the function ϕ . As before, when ϕ is a degreek polynomial, the time complexity of computing ϕ γ is bounded by N O ( k ) .

C.3 Efficiently computing fixed points in expectation

Now let δ : conv X → ∆( X ) be consistent and efficient. Consider the following algorithm. Given ϕ ∈ Φ , select x 1 ∈ conv X arbitrarily, and then for each ℓ > 1 set x ℓ := ϕ δ ( x ℓ -1 ) . Finally, select π := E ℓ ∼ [ L ] δ ( x ℓ ) ∈ ∆( X ) as the output distribution. By a telescopic cancellation, we have

as desired. As a result, applying Theorem C.2, we arrive at the following conclusion.

Theorem C.7. Let R Φ be an regret minimizer on Φ whose external regret after T iterations is Reg T and whose per-iteration runtime is R 1 , and assume that evaluating the extended map ϕ δ : conv X → conv X takes time R 2 . Then, for every ϵ > 0 , there is a learning algorithm on X whose Φ -regret after T iterations is at most Reg T + ϵ and whose per-iteration runtime is O ( R 1 + R 2 /ϵ ) .

The above result provides a full black-box reduction from Φ -regret minimization to external regret minimization on Φ , with no need for the possibly-expensive computation of a fixed point. We note that the iterates of the algorithm will depend on the choice of δ -for example, setting δ = β and setting δ = γ will produce different iterates.

D Low-degree regret on the hypercube

In this section, we let X be the hypercube { 0 , 1 } N . For the convenience of the reader, we first recall that the set of deviations Φ k DT is defined as follows:

The deviator observes an index j 0 ∈ [ N ] .

For i = 1 , . . . , k : The deviator selects an index j i ∈ [ N ] , and observes x [ j i ] .

The deviator selects a 0 ∈ { 0 , 1 } .

As we observed earlier, the above process describes a tree-form decision problem of size N O ( k ) . In particular, terminal nodes in this decision problem are identified by the original index j 0 ∈ [ N ] , the queries j 1 , . . . , j k ∈ [ N ] , their replies a 1 , . . . , a k ∈ { 0 , 1 } , and finally the action a 0 ∈ { 0 , 1 } that is played. Each tree-form strategy q in this decision problem defines a function ϕ q : X → conv X , which is computed by following the strategy q through the decision problem. Namely,

where x [ j i , a i ] = x [ j i ] if a i = 1 , and 1 -x [ j i ] if a i = 0 . We see that ϕ q is a degreek polynomial in x .

We define Φ k DT as the set of such functions ϕ q . The 'DT' in the name Φ k DT stands for decision tree : the set of functions ϕ : X → conv X that can be expressed in the above manner is precisely the set of functions representable as (randomized) depthk decision trees on N variables.

For intuition, we mention the following special cases:

Φ 0 DT is the set of external deviations.

Φ 1 DT is the set of all single-query deviations, which Fujii [2023] showed to be equivalent to the set of all linear deviations when X is a hypercube.

Φ N DT is the set of all swap deviations.

Since q ↦→ ϕ q ( x )[ i ] is linear, it follows that q ↦→ ⟨ u , ϕ q ( x ) ⟩ is also linear for any given u ∈ R n . Therefore, a regret minimizer on Φ k DT can be constructed starting from any regret minimizer for tree-form decision problems, such as counterfactual regret minimization [Zinkevich et al., 2007].

Proposition D.1. There is a N O ( k ) -time-per-round regret minimizer on Φ k DT whose external regret is at most ϵ after N O ( k ) /ϵ 2 rounds.

Thus, combining with Proposition C.6 and Theorem C.7, we immediately obtain a Φ k DT -regret minimizer with the following complexity.

Corollary D.2. There is a N O ( k ) /ϵ -time-per-round regret minimizer on X whose Φ k DT -regret is at most ϵ after N O ( k ) /ϵ 2 rounds.

Next, we relate depthk decision trees to low-degree polynomials. Let Φ k poly be the set of degreek polynomials ϕ : X → X . We appeal to a result from the literature on Boolean analysis, recalled below.

Theorem 4.2 (Midrijanis, 2004) . Every degreek polynomial f : { 0 , 1 } N →{ 0 , 1 } can be written as a decision tree of depth at most 2 k 3 .

In particular, Φ k poly ⊆ Φ 2 k 3 DT . Corollary D.2 thus also implies a Φ k poly -regret minimizer:

Corollary D.3. Let X = { 0 , 1 } N . There is an N O ( k 3 ) /ϵ -time-per-round regret minimizer on X whose Φ k poly -regret is at most ϵ after N O ( k 3 ) /ϵ 2 rounds.

It is reasonable to ask whether the above result generalizes to polynomials ϕ : X → conv X . Indeed, when k ≤ 1 or k = N , every degreek polynomials ϕ : X → conv X can be written as a convex combination of degreek polynomials ϕ : X → X , even for arbitrary tree-form decision problems. 7 However, this is not generally true. A brute-force search shows that the polynomial ϕ : { 0 , 1 } 4 → [0 , 1] 4 given by

is quadratic, but it is not a convex combination of quadratics whose range is { 0 , 1 } 4 . Perhaps more glaringly, if one could efficiently represent the set of quadratic functions ϕ : { 0 , 1 } N → [0 , 1] N , then one could in particular decide whether a given quadratic function ϕ : { 0 , 1 } N → R N has range [0 , 1] N . But this is a coNP -complete problem.

E Extensive-form games

The goal of this section is to extend the results in the previous section to the extensive-form setting, that is, to generalize them to all tree-form decision problems.

E.1 Interleaving decision problems

We first recall the operations of merging decision problems that will be very useful as notation in the subsequent discussion. In particular, given two decision problems X and Y with node sets S 1 and S 2 respectively, we restate the following definitions previously introduced in Section 4.

Definition 4.3. The dual ¯ X of X is the decision problem identical to X , except that the decision points and observation points have been swapped.

Definition 4.4. The interleaving X ⊗Y is the tree-form decision problem defined as follows. There is a state s = ( s 1 , s 2 ) ∈ S 1 ×S 2 . The root state is the tuple ( ∅ , ∅ ) . The decision problem is defined by the player being able to interact with both decision problems, in the following manner. At each state s = ( s 1 , s 2 ) :

If s 1 and s 2 are both terminal then so is s . Otherwise:

If either of the s i s is an observation point, then so is s . The children are the states ( s ′ i , s -i ) where s ′ i is a child of s i . (If both s i s are observation points, both children s ′ 1 , s ′ 2 are selected simultaneously. This can only happen at the root.)

Otherwise, s is a decision point. The player selects an index i ∈ { 1 , 2 } at which to act, and a child s ′ i to transition to. The next state is ( s ′ i , s -i ) .

It follows immediately from definitions that ¯ ¯ X = X , and ⊗ is associative and commutative. The name and notation for the dual is inspired by the observation that ⟨ x , y ⟩ = 1 for all x ∈ X and y ∈ ¯ X : indeed, the component-wise product x [ z ] y [ z ] is exactly the probability that one reaches terminal node z by following strategy x at X 's decision points and y at X 's observation points. We also define the notation X ⊗ k := X ⊗ · · · ⊗ X , where there are k copies of X .

It is important to observe that in X ⊗ Y , the same state ( s 1 , s 2 ) can be reachable through possibly exponentially many paths. This is because the learner may choose to interleave actions in X with actions in Y in any order, which means that a state ( s 1 , s 2 ) corresponds to exponentially many histories in X ⊗ Y . For that reason, we have to distinguish between histories and states .

In light of the above discussion, it is inefficient to represent X ⊗ Y as a tree. Indeed, Zhang et al. [2023] studied DAG -form decision problems, and showed that regret minimization on them is possible when the underlying DAG has some natural properties. We state here an immediate consequence of their analysis, which we will use as a black box. Intuitively, the below result states that, as long as utility vectors also only depend on the (terminal) state s that is reached, regret minimization on an arbitrary interleaving of decision problems X 1 ⊗··· ⊗ X k is possible, and the complexity depends only on the number of states.

Theorem E.1 (Consequence of Zhang et al., 2023, Corollary A.4) . Let X := X 1 ⊗··· ⊗ X k , where X i has terminal node set Z i . Let Z := Z 1 × · · · × Z k be the set of terminal states for X . For each such terminal state z ∈ Z , let V ( z ) be the set of histories of X whose state is z . Define the projection π : X → R Z by

Then there exists an efficient regret minimizer on π ( X ) := { π ( x ) : x ∈ X} ⊂ R Z : its per-round complexity is poly ( |Z| ) , and its regret is ϵ after poly ( |Z| ) /ϵ 2 rounds.

Whenever we speak of regret minimizing on interleavings, it will always be the case that utility vectors depend only on the state, so we will always be able to apply the above result. We will call vectors in π ( X ) reduced strategies .

Before proceeding, it is instructive to describe in more detail a result of Zhang et al. [2024], which we will also use later, in the language of this section. Let X and Y be any two decision problems with terminal node sets Z 1 and Z 2 respectively. A reduced strategy q ∈ π ( X ⊗ ¯ Y ) induces a linear map ϕ q : Y → conv X , given by

...

...

...

...

...

...

...

0

0

1

B

2

0

B

2

3

C

4

2

3

5

2

3

C

4

5

C

1

C

3

0

1

C

C

1

0

1

A

,

A

,

A

A

,

0

,

0

A

,

B

,

0

A

,

2

,

0

0

,

2

,

0

B

,

2

,

0

B

,

2

,

C

B

,

2

,

4

B

,

2

,

5

C

,

2

,

0

A

,

3

,

0

A

,

1

,

1

It is instructive to think, as Zhang et al. [2024] detailed extensively in their paper, about what strategies q ∈ π ( X ⊗ ¯ Y ) represent, and why they induce the linear maps ϕ q . Decision points j in Y become observation points in X ⊗ ¯ Y -at these observation points, the player should observe the action taken by strategy y at j . The player in X ⊗ ¯ Y is given the ability to query the strategy y by taking the role of the environment in Y , while the environment, holding a strategy y ∈ Y , takes the role of the player and answers decision point queries with the actions that it plays. The player then uses these queries to inform how it plays in the true decision problem X . This is the sense in which q induces a map ϕ q : the output ϕ q ( y ) is precisely the strategy that would be played if the environment in X ⊗ ¯ Y answers the queries by consulting the strategy y . We will call a device that answers queries using strategy y a mediator holding strategy y . Zhang et al. [2024] then showed the following fact, which we will use critically and repeatedly in the rest of this paper.

Theorem E.2 (Zhang et al., 2024, Theorem A.2) . Every linear map ϕ : Y → conv X is induced by some reduced strategy q ∈ π ( X ⊗ ¯ Y ) .

E.2 Efficient low-degree swap-regret minimization in extensive-form games

We now proceed with generalizing the results of Appendix D to extensive-form games.

Let X be any decision problem of dimension N and depth d . We will assume WLOG that every decision point in X has branching factor exactly 2 . This is without loss of generality, but it incurs a loss of O (log b ) , where b is the original branching factor, in the depth. Thus, in the below bounds, when d appears, it should be read as O ( d log b ) .

Using the previous notation, the set of k -mediator deviations Φ k med is the set of reduced strategies in the decision problem X ⊗ ¯ X ⊗ k . In particular, we recall that reduced strategies q ∈ π ( X ⊗ ¯ X ⊗ k ) induce functions ϕ q : X → conv X given by

Thus, we have that ϕ q is a degreek polynomial. Φ k med is the set of such deviations. For intuition, we once again pose a few special cases:

When the original decision problem's decision space is ∆ N 2 ( i.e. , the decision problem consists of a single root observation point with N children, each of which is a decision point with two actions), we have Φ k DT = Φ k med . Thus, the results in this section strictly generalize those in the previous section.

Φ 0 med and Φ N med are, as before, the sets of external and swap deviations respectively.

Φ 1 med is, by Theorem E.2, the set of all linear deviations.

In this context, applying Theorem E.1 gives an efficient Φ k med -regret minimizer:

Theorem E.3. There is an N O ( k ) -time-per-round regret minimizer on Φ k med whose external regret is at most ϵ after N O ( k ) /ϵ 2 rounds.

Thus, once again Proposition C.6 and Theorem C.7 have the following consequence.

Corollary E.4. There is a N O ( k ) /ϵ -time-per-round regret minimizer on X whose Φ k med -regret is at most ϵ after N O ( k ) /ϵ 2 rounds.

Next, we discuss extensions of our result to low-degree polynomials. Unfortunately, we cannot directly apply Theorem 4.2 to conclude the existence of a regret minimizer on X with Φ k poly -regret growing as N O ( k 3 ) /ϵ 2 . There are two issues in attempting to do so.

First, when X is not the hypercube, polynomials f : X → { 0 , 1 } are not total functions. That is, it is not necessarily the case that degreek polynomials f : X → { 0 , 1 } can be extended to degreek polynomials ¯ f : { 0 , 1 } N → { 0 , 1 } , which is required in order to apply Theorem 4.2. 8 For an example of this, consider X = D 4 where D N is the standard basis in R N , that is, D N = { e i : i ∈ [ N ] } where e i ∈ R N is the i th basis vector (in other words, D N is the set of vertices of the probability simplex ∆( N ) ). Let f : D 4 →{ 0 , 1 } given by f ( x ) = x 1 + x 2 . Then f is linear, but there is no linear ¯ f : { 0 , 1 } 4 →{ 0 , 1 } extending f . Indeed, there is a more general manifestation of this phenomenon:

Proposition E.5. For every N , there exists a linear map f : D N →{ 0 , 1 } such that any extension ¯ f : { 0 , 1 } N →{ 0 , 1 } of f must have degree at least Ω(log N ) .

Proof. Let ¯ f : { 0 , 1 } N →{ 0 , 1 } be any degreek function. By Theorem 3.4 of O'Donnell [2014], ¯ f is a k 2 k -junta, that is, ¯ f ( x ) depends on at most k 2 k entries of x . Now consider the map f : D N →{ 0 , 1 } given by f ( x ) = ∑ i ≤ N/ 2 x i . Let ¯ f : { 0 , 1 } N →{ 0 , 1 } be an extension of f . Then ¯ f depends on at least N/ 2 -1 inputs: if ¯ f ( 0 ) = 0 then ¯ f depends on at least x 1 , . . . , x ⌊ N/ 2 ⌋ , and if ¯ f ( 0 ) = 1 then ¯ f depends on at least x ⌊ N/ 2 ⌋ +1 , . . . , x N . Thus, we have N/ 2 -1 ≤ k 2 k , which upon rearraging gives k ≥ Ω(log N ) .

The second issue is the following. Suppose that K mediators were enough to represent a function f : X → { 0 , 1 } . How does one then represent a function ϕ : X → X ? Each coordinate of ϕ could be represented using K mediators, but that need not mean the whole function can. In game-theoretic terms, representing a coordinate of ϕ ( x ) allows the player to play a single action, not necessarily the whole game. Naively, playing the whole game would seem to require Kd mediators: K mediators for every level of the decision tree, to compute which action to take at each level.

We will show that it is possible to circumvent both of these issues: the first with a loss of O ( d ) in the degree of the polynomial that is representable, and the second with no additional loss. In particular, we state our main result below.

Theorem E.6. Φ k poly ⊆ Φ O ( kd ) 3 med . Therefore, for every k , there is a N O ( kd ) 3 /ϵ -time-per-round algorithm whose Φ k poly -regret at most ϵ after N O ( kd ) 3 /ϵ rounds.

E.3 Proof of Theorem E.6

We dedicate the rest of this section to the proof of Theorem E.6. We deal with the two aforementioned problems one by one. First, we show that every f admits an extension at a loss of a factor of d in the degree:

LemmaE.7. Every f : X → { 0 , 1 } of degree k admits a degreekd extension ¯ f : { 0 , 1 } N →{ 0 , 1 } .

Proof. We claim first that the identity function id : X → X admits a degreed extension, that is, there is a degreed function ¯ id : { 0 , 1 } N → X that is the identity on X . Indeed, consider the following function ¯ id . Then we define ¯ id( x )[ ja ] recursively as follows:

It is easy to check that ¯ id is indeed the identity on X by definition of tree-form decision spaces, and that the degree of ¯ id is at most the depth of the tree, d . But now, for any f : X → { 0 , 1 } of degree k , the map ¯ f := f ◦ ¯ id : { 0 , 1 } N →{ 0 , 1 } has degree at most kd and is an extension of f .

Now we apply Theorem 4.2. That result tells us that a degreekd function ¯ f : { 0 , 1 } N → { 0 , 1 } can be evaluated using K = O ( kd ) 3 queries. One mediator is certainly more than enough to perform a single query, and therefore such a function can also be evaluated using K mediators. It therefore remains only to address the second problem: namely, the ability to evaluate a function ¯ ϕ : { 0 , 1 } N → { 0 , 1 } , since the output is only binary, in principle only allows us to play a single action. But one needs to play d actions to reach the end of the game. Naively, this would require losing another factor of d , for a total of O ( Kd ) mediators. However, we now show that it is possible to completely circumvent this problem. The notation that we have built up will make this argument perhaps surprisingly short.

Let ϕ : X → X be any function such that each component ϕ z : X → { 0 , 1 } , given by x ↦→ ϕ ( x )[ z ] , is expressible using K mediators. By definition, ϕ z is expressible as a strategy 9 q z ∈ π ( { 0 , 1 } ⊗ ¯ X ⊗ K ) . By the argument in the previous section, q z induces a linear map ˆ ϕ z : ¯ X ⊗ k →{ 0 , 1 } .

Now let ˆ ϕ : ¯ X ⊗ K →X be the function whose z th coordinate is ˆ ϕ z . Every component of ϕ is linear, so ˆ ϕ is itself also linear. But then by Theorem E.2, there exists a strategy q ∈ π ( X ⊗ ¯ X ⊗ K ) , that is, a K -mediator deviation, that represents ϕ . This completes the proof.

F Discussion and applications

In this section, we discuss various implications and make several remarks about the framework and results that we have introduced.

F.1 Convergence to correlated equilibria

Notions of Φ -regret correspond naturally to notions of correlated equilibria. Therefore, our results also have implications for no-regret learning algorithms that converge to correlated equilibria. Here, we formalize this connection. Consider an n -player game in which player i 's strategy set is a treeform strategy set X i , and player i 's utility is given by a multilinear map u i : X 1 × · · · × X n →

[ -1 , 1] . For each player i , let Φ i ⊆ (conv X i ) X i be a set of deviations for player i . Finally let Φ = (Φ 1 , . . . , Φ n ) .

Definition F.1. A distribution π ∈ ∆( X 1 × · · · × X n ) is called a correlated profile . A correlated profile π is an ϵ -Φ -equilibrium if no player i can profit more than ϵ via any of the deviations ϕ i ∈ Φ i to its strategy. That is, E x ∼ π u i ( ϕ i ( x i ) , x -i ) ≤ E x ∼ π u i ( x i , x -i ) + ϵ for all players i and ϕ i ∈ Φ i .

For example, we can define k -mediator equilibria and degreek swap equilibria by setting Φ i to Φ k med and Φ k poly , respectively. The following celebrated result follows immediately from the definitions of equilibrium and regret.

Proposition F.2. Suppose that every player i plays according to a regret minimizer whose Φ i -regret is at most ϵ after T rounds. Let π ( t ) i ∈ ∆( X i ) be the distribution played by player i at round t . Let π ( t ) ∈ ∆( X 1 ) ×··· × ∆( X n ) be the product distribution whose marginal on X i is π ( t ) i . Then the average strategy profile , that is, the distribution 1 T ∑ t ∈ [ T ] π ( t ) , is an ϵ -Φ -equilibrium.

Thus, from Theorems E.3 and E.6 it follows, respectively, that, given a game Γ where the dimension of each player's decision problem is at most N , we have the following results.

Corollary F.3. An ϵ -k -mediator equilibrium can be computed in time N O ( k ) /ϵ 3 .

Corollary F.4. An ϵ -degreek -swap equilibrium can be computed in time N O ( kd ) 3 /ϵ 3 .

The issue of representing the induced correlated distribution is discussed in Appendix G.

F.2 Strict hierarchy of equilibrium concepts

Let c ∈ { med , poly } . For every k ≥ 0 , let E k c (Γ) be the set of Φ k c -equilibria in Γ . It is clear from definitions that E k c (Γ) ⊆ E k -1 c (Γ) . Further, even for normal-form games, it is known that coarsecorrelated equilibria are not generally equivalent to correlated equilibria, so at least one of these inclusions is strict in some games. We now show that all of these inclusions are strict, so that the deviations Φ k c form a strict hierarchy of equilibria. 10

Proposition F.5. For every k ≥ 1 , there exists a game Γ such that E k c (Γ) ⊊ E k -1 c (Γ) .

Proof. Consider the two-player game Γ defined as follows.

P1's strategy space is X = {-1 , 1 } k . Player 2's strategy space is simply Y = {-1 , 1 } . 11

P1's utility function is u 1 ( x , y ) = x 1 y . That is, P1 would like to set x 1 = y . P2 gets no utility.

Consider the correlated profile π defined as follows: π is uniform over the 2 k pure profiles ( x , y ) ∈ X×Y such that y = x 1 x 2 . . . x k . P1's expected utility is clearly 0 , and there is a swap ( i.e. , Φ k c ) deviation that yields a profit of 1 , namely x ↦→ ( x 1 x 2 . . . x k , . . . ) . (it does not matter what the swap deviation plays at coordinates other than the first one.) But, since all the x i s are independent, no function of degree less than k can have positive correlation with x 1 x 2 . . . x k , and thus, there are no profitable deviations of degree less than k . Thus, π is a Φ k -1 c -equilibrium, but not a Φ k c -equilibrium.

F.3 Characterization of recent low-swap-regret algorithms in our framework

We have, throughout this paper, introduced and used a framework of Φ -regret that involves fixed points in expectation. Proposition C.3 shows that the ability to compute fixed points in expectation is in some sense necessary for the ability to minimize Φ -regret. It is instructive to briefly discuss how the recent swap-regret-minimizing algorithm of Dagan et al. [2024] and Peng and Rubinstein [2024]

fits into this framework. Their algorithm makes no explicit reference to fixed-point computation, nor to the minimization of external regret over swap deviations ϕ -they do not explicitly invoke the framework we use in this paper, nor that of Gordon et al. [2008]. Where is the expected fixed point hidden, then? While we will not present their entire construction here, it suffices to state the following property of it. At every round t , the learner outputs a distribution π ( t ) ∈ ∆( X ) that is uniform on L strategies x ( t, 1) , . . . , x ( t,L ) . The way to map this into our framework is to consider π ( t ) an approximate fixed point in expectation of the 'function' 12 ϕ ( t ) that maps x ( t,ℓ ) ↦→ x ( t,ℓ +1) for each ℓ = 1 , . . . , L -1 . With this choice of ϕ ( t ) , their algorithm indeed fits into our framework.

F.4 Revelation principles (or lack thereof)

Most notions of correlated equilibrium obey some form of revelation principle . Informally, one can treat a player attempting to deviate profitably from a correlated equilibrium as an interaction between a mediator (who sends useful information to the player) and the player (who tries to play optimally by using the mediator). When studying the regret of online algorithms, one assumes that the interaction with the mediator is canonical : the mediator holds with it some sampled strategy profile ( x 1 , . . . , x n ) ∼ π , and in equilibrium every player indeed plays x i . We say that the revelation principle holds for a particular notion of equilibrium if allowing non-canonical equilibria would not expand the set of equilibria. In Appendix H, we give a rather general formalization of this notion, which is enough to encompass all the notions of correlated equilibrium discussed in the paper. We show that, in this formalism, the revelation principle does not hold for k -mediator equilibria or degreek swap equilibria when k > 1 , and indeed in both cases the set of outcomes that can be induced by non-canonical equilibria is the set of linear-swap outcomes (Theorems H.4 and H.5).

G Representation of strategies

In this section, we discuss how strategies π ∈ ∆( X 1 × · · · × X n ) are represented for the purposes of all of the results in this paper, and in particular for Corollaries F.3 and F.4. In both cases, at each timestep, each player's strategy π ( t ) i is a uniform mixture of L = O (1 /ϵ ) strategies δ ( x ( t, 1) i ) , . . . , δ ( x ( t,L ) i ) , and we have

that is, π is a uniform mixture of products of mixtures of strategies that are themselves outputs of δ . Thus, if the strategy map δ is established by convention (for example, as mentioned before, it is often conventional to take δ to be the behavioral strategy map), it suffices to output x ( t,ℓ ) i ∈ conv X i for each i, t, ℓ .

Suppose that we impose a slightly more stringent restriction on the output format, namely, we want π to be a uniform mixture of products of mixtures of pure strategies. In that case, we can take δ to be the Carath´ eodory map. 13 Now, writing x ( t,ℓ ) i = ∑ j ∈ [ N ] α ( t,ℓ,j ) i x ( t,ℓ,j ) i for x ( t,ℓ,j ) i ∈ X i , we set

So, our output consists of the strategies x ( t,ℓ,j ) i ∈ X i and their coefficients α ( t,ℓ,j ) i for each i, t, ℓ, j .

H On the revelation principle

In this section, we give a formalization of the revelation principle which encompass all the notions of correlated equilibrium discussed in the paper. In this formalism, the revelation principle does

not hold for k -mediator equilibria or degreek swap equilibria when k > 1 , and indeed in both cases we show that the set of outcomes that can be induced by non-canonical equilibria is the set of linear-swap outcomes (Theorems H.4 and H.5).

Let D be the class of all finite tree-form decision problems. For each pair X , Y ∈ D let Φ X , Y ⊆ (conv X ) Y be a subset of deviations. Finally let Φ = ⊔ X , Y∈ D Φ X , Y . As in the previous section, consider a game where player i has strategy set X i and utility u i : X 1 ×··· × X n → [ -1 , 1] .

Definition H.1. Let Y 1 , . . . , Y n ∈ D be arbitrary tree-form strategy spaces, let π ∈ ∆( Y 1 ×··· × Y n ) , and let ϕ i ∈ Φ X , Y for each player i . We call the tuple (( Y i , ϕ i ) n i =1 , π ) a generalized profile . A generalized profile is a generalized ϵ -Φ -equilibrium if no player i can profit by switching to a different strategy mapping ϕ ′ : Y i → conv X i . That is,

for all players i and ϕ ′ i ∈ Φ X , Y . We call a generalized profile canonical if Y i = X i and ϕ i : X i → conv X i is the identity map for every i .

In this language, the definitions of equilibrium in Appendix F.1 were definitions of canonical equilibria. Every generalized profile induces a canonical profile, namely, the distribution over strategy given by sampling y ∼ π and returning ( ϕ 1 ( x 1 ) , . . . , ϕ n ( x n )) . Call two generalized profiles equivalent if they induce the same canonical profile. We can now define the revelation principle as follows.

Definition H.2 (Revelation principle) . The class of deviations Φ satisfies the revelation principle if the induced canonical profile of every generalized ϵ -Φ -equilibrium is also an ϵ -Φ -equilibrium.

For an example, let Φ be the set of all functions, so that the notion of equilibrium is the normal-form correlated equilibrium. Then one can think of the sample y ∼ π as a profile of signals (one signal per player) from a correlation device, and ϕ i as player i 's mapping from signals to strategies. Then the revelation principle states that, without loss of generality (up to utility equivalence), one can assume that signals are recommendations of strategies ( Y i = X i ) and players in equilibrium play their recommended strategies ( ϕ i : X i → conv X i is the identity map).

All notions of equilibrium that we have mentioned can be expressed in this language, and the revelation principle applies to all of them.

Proposition H.3 (Sufficient conditon for revelation principle) . Let δ be a consistent strategy map in the sense of Appendix C.2. Suppose that, for every ϕ ∈ Φ X , Y and ψ ∈ Φ X , X , we have ψ δ ◦ ϕ ∈ Φ X , Y . Then Φ satisfies the revelation principle.

Proof. Given a generalized ϵ -Φ -equilibrium, (( Y i , ϕ i ) n i =1 , π ) , let π ′ be its induced canonical profile. Weneed to show that π ′ is also an ϵ -Φ -equilibrium. Consider any hypothetical deviation ψ i ∈ Φ X , X . We have

where the second line uses the assumed composition property.

The revelation principle has, of course, been shown for various special cases of equilibrium before us: for example, NFCE Aumann [1974], linear-swap equilibria Zhang et al. [2024], and so on. Our proposition above generalizes these proofs since each of those notions indeed satisfies the requisite compositional criterion: compositions of arbitrary functions are still arbitrary functions, and compositions of linear maps are linear.

The definition of k -mediator functions and degreek polynomials are both easy to generalize from the X → X case to the Y → X case. We can therefore define generalized k -mediator equilibria and generalized degreek swap equilibria , and ask whether the revelation principle applies to

these. Proposition H.3 does not apply, because compositions of k -mediator and degreek functions will usually require more mediators and a higher degree. Indeed, we now show that the revelation principle fails for these notions when k > 1 . We will, in fact, show something quite strong. Recall Proposition F.5, which showed that both canonical Φ k med and canonical Φ k poly are strictly tightening notions of equilibrium as k increases. Here we show that this is not the case for generalized equilibria.

Theorem H.4. For every k ≥ 1 , every linear-swap equilibrium is equivalent to a (non-canonical) k -mediator equilibrium.

Proof. Let π be any linear-swap equilibrium. We define a mediator of player i . Given a strategy x ∈ X , the mediator will interact with the player as follows. First, for each decision point j of player i , and let a j ∈ A j be the action that is played by x . (If x defines no action at j , the mediator selects one at random). The mediator samples integers a (1) j , . . . , a ( k ) j ∈ [ b j ] uniformly at random under the constraint that ∑ k ℓ =1 a ( ℓ ) j = a j (mod b j ) . The important property that is derived from the above construction is simply that, in order to learn a j , the player must know all the a ( ℓ ) j s, and without knowing all of them, the player learns no information whatsoever .

When the player arrives, the mediator has the following interaction with the player. First, the player must supply an integer ℓ ∈ [ k ] . We will call ℓ the seed . Then, whenever the player sends a decision point j , the mediator checks whether the decision point sent is consistent with the sequence of decision points previously sent by the player ( i.e. , if j is a possible next-decision-point following the previous decision point sent). If not, the mediator terminates the interaction. If so, the mediator sends a ( ℓ ) j to the player.

Now consider how a player can use k copies of such a mediator. In order to learn any useful information at all about any decision point j , the player must (1) supply a different seed to each of the k mediators, and (2) query decision point j with all k mediators. Therefore, the player can essentially only use these mediators as if they were one, giving the same sequence of queries to every mediator and computing the true action recommendation by computing ∑ ℓ a ( ℓ ) j (mod b j ) . Thus, the player can only implement deviations that it could with a single mediator, which by Theorem E.2 are the linear deviations.

What we have thus shown is the following. Let Y i be a strategy space that represents the above interaction with the mediator, and let ϕ i be the k -mediator deviation that acts according to the previous paragraph. Let π ′ ∈ ∆( Y 1 × · · · × Y n ) be the distribution in which a strategy x ∼ π is sampled, the a ( ℓ ) j s are sampled according to the first paragraph, and each mediator plays according to the strategy y i that answers queries according to those a ( ℓ ) j s. Then (( Y i , ϕ i ) n i =1 , π ′ ) is a noncanonical Φ k med -equilibrium.

Theorem H.5. Suppose that every player's strategy space in a given game Γ is the hypercube X = {-1 , 1 } N . Then every linear-swap in Γ equilibrium is equivalent to a (non-canonical) degreek -swap equilibrium.

Proof. The proof follows a similar idea to the previous one: we wish to construct a scenario such that, with any polynomial of degree less than k , the deviator cannot learn anything about x , and with a polynomial of degree k the deviator can only implement linear functions on x . Let π be any linearswap equilibrium. Let Y = {-1 , 1 } Nk , and index the coordinates of y ∈ Y by pairs ( j, ℓ ) where j ∈ [ N ] and ℓ ∈ [ k ] . Define ϕ ( y ) j = ∏ ℓ ∈ [ k ] y [ j, ℓ ] . Finally, define the distribution π ′ ∈ ∆( Y n ) as follows: sampling y ∼ π ′ is done by sampling x ∼ π , and then, for each player i and index j , sampling y i [ j, · ] ∈ {-1 , 1 } ℓ uniformly at random under the constraint that ∏ ℓ y i [ j, ℓ ] = x i [ j ] . We will write π ′ | x for the conditional distribution of y ∼ π ′ , conditioned on sampling the given x ∼ π . Now consider the generalized profile ( Y , ϕ, π ′ ) ( i.e. , every player shares the same signal set Y and equilibrium deviation function ϕ : Y → X ). We see that it is equivalent to ( X , id , π ) by construction. We claim now that it is a degreek equilibrium which would complete the proof. Consider any degreek function ϕ ′ : Y → X , and define ψ : X → X by ψ ( x ) = E y ∼ π ′ | x ϕ ′ ( y ) . It suffices to show that ψ is a linear map, since then deviating to ϕ ′ would equate to applying the linear

deviation ψ to x . Indeed, expressing

we see that E y ∼ π ′ | x m S ( y ) = 0 except when S = S j := { ( j, ℓ ) : ℓ ∈ [ k ] } for some j , in which case E y ∼ π ′ | x m S ( y ) = x [ j ] . That is, the only monomials of nonzero expectation are those which multiply together all the y [ j, · ] s, in which case the expectation is exactly x [ j ] . Thus, we have

which is indeed linear in

One may wonder at this point about why the above two results do not contradict the fact that NFCE, which supposedly are the Φ N med -equilibria, do satisfy the revelation principle and are certainly not equivalent to linear-swap equilibria. The answer is that the equivalence between swap deviations and N -mediator deviations only holds for strategy spaces of size at most N -and the strategy spaces Y i constructed as part of both proofs have size (at least) Nk . When Y i has size more than N , the set Φ N med is no longer the set of all functions Y i → conv X i . In other words, the set of canonical Φ N med -equilibria is equivalent to the set of canonical NFCE, but that does not contradict the fact that there exist non-canonical Φ N med -equilibria.

I Omitted proofs

In this section, we provide the proofs omitted from the previous sections.

I.1 Proofs from Section B

We first provide the missing proofs from Appendix B. We start with Proposition B.1, the statement of which is recalled below.

Proposition B.1 (Hazan and Kale, 2007) . Consider a regret minimizer R operating over [0 , 1] N . If R runs in time poly ( N, 1 /ϵ ) and guarantees Reg T Φ β ≤ ϵ for any sequence of utilities, then there is a poly ( N, 1 /ϵ ) algorithm for computing an ( ϵ √ N ) -fixed point of any ϕ β ∈ Φ β with respect to ∥ · ∥ 2 , assuming that ϕ β can be evaluated in polynomial time.

Proof. Suppose that R outputs a strategy x ( t ) ∈ conv X at each time t ∈ [ T ] . The basic idea is to determine whether ∥ ϕ β ( x ( t ) ) -x ( t ) ∥ 2 ≤ ϵ √ N ; if so, the process can terminate as we have identified an ( ϵ √ N ) -fixed point of ϕ β with respect to ∥ · ∥ 2 . Otherwise, we construct the utility function

We note that, by Cauchy-Schwarz, | u ( t ) ( x ) | ≤ 1 since ∥ x -x ( t ) ∥ 2 ≤ √ N ; hence, the utility function adheres to our normalization constraint. Now, if at all rounds it was the case that ∥ ϕ β ( x ( t ) ) -x ( t ) ∥ 2 > ϵ √ N , we have

since u ( t ) ( x ( t ) ) = 0 and u ( t ) ( ϕ β ( x ( t ) )) = 1 √ N ∥ ϕ β ( x ( t ) ) -x ( t ) ∥ 2 for all t ∈ [ T ] . We conclude that (5) contradicts the assumption that Reg T Φ β ≤ ϵ for any sequence of utilities, in turn implying that there exists t ∈ [ T ] such that ∥ ϕ β ( x ( t ) ) -x ( t ) ∥ 2 ≤ ϵ √ N . Given that we can evaluate ϕ β in time poly ( N ) (by assumption), we can also compute each error ∥ ϕ β ( x ( t ) ) -x ( t ) ∥ 2 in polynomial time, concluding the proof.

We next turn our attention to the proof of Theorem 3.3. We first observe that Φ β contains functions of the following form.

Lemma I.1. Φ β contains all functions of the form

where ϕ = ( ϕ 1 , . . . , ϕ N ) : { 0 , 1 } N → { 0 , 1 } N . (The convention above is that a product with no terms is to be evaluated as 1 .)

Above, we slightly abuse notation by interpreting S j ⊆ [ N ] as a point in { 0 , 1 } N . Lemma I.1 is evident from the definition of the behavioral strategy map β : ϕ β ( x ) = E x ′ ∼ β ( x ) ϕ ( x ′ ) , and expanding the expectation gives the expression of Lemma I.1-the probability of a set S is exactly ∏ j ∈ S x [ j ] ∏ j ∈ [ N ] \ S (1 -x [ j ]) . We will show that PPAD -hardness for computing fixed points persists under the set of functions contained in Φ β (per Lemma I.1).

Our starting point is the usual generalized circuit problem (abbreviated as GCIRCUIT), introduced by Chen et al. [2009]; it is a generalization of a typical arithmetic circuit but with the twist that it may include cycles . (In what follows, we borrow some notation from the paper of Filos-Ratsikas et al. [2023].)

Definition I.2 (Chen et al., 2009) . Ageneralized circuit with respect to a set of gate-types G is a list of gates g 1 , . . . , g M . Every gate g i has a type G i ∈ G . Depending on its type, g i may have zero, one or two input gates, which will be index by j, k ∈ [ M ] , with the restriction that i, j, k are pairwise distinct.

We denote by v : [ M ] → [0 , 1] a function that assigns an input gate to a value in [0 , 1] . Each gate imposes a constraint induced by its corresponding type. For example, let us consider the following types.

if G i = G + , then v ( g i ) = min(1 , v ( g j ) + v ( g k )) ;

if G i = G -, then v ( g i ) = max(0 , v ( g j ) -v ( g k )) ;

if G i = G 1 , then v ( g i ) = 1 ;

if G i = G 1 -, then v ( g i ) = 1 -v ( g j ) ; and

if G i = G × , then v ( g i ) = v ( g j ) · v ( g k ) .

In accordance with the above types, we define F : [0 , 1] M → [0 , 1] M to be the function mapping any initial evaluation of the gates to ( v ( g 1 ) , . . . , v ( g M )) . The main problem of interest can be now phrased as follows.

Definition I.3 ( ϵ -GCIRCUIT) . The problem ϵ -GCIRCUIT asks for a fixed point of F with respect to ∥ · ∥ ∞ ; that is, an assignment v : [ M ] → [0 , 1] such that all gates are ϵ -satisfied.

Asatisfying assignment always exists by Brouwer's theorem, but the associated computational problem is PPAD -hard. In fact, by virtue of a recent result of Filos-Ratsikas et al. [2023], PPAD -hardness persists even if one significantly restricts the type of gates.

Theorem I.4 (Filos-Ratsikas et al., 2023) . Even when G := { G + , G 1 -} , there is an absolute constant ϵ > 0 such that computing an ϵ -fixed point of F with respect to ∥ · ∥ ∞ is PPAD -complete. 14

Yet, the addition gate G + does not induce a multilinear in the form of Lemma I.1. We will address this by showing that the gate G + can be approximately simulated via a small number of gates with type either G 1 -or G × . To provide better intuition, we first prove this claim under the assumption that the gates are satisfied exactly, and we then proceed with the more general statement. In the sequel, we sometimes use the shorthand notation t = t ′ ± ϵ ⇐⇒ t ∈ [ t ′ -ϵ, t ′ + ϵ ] .

Lemma I.5. Any addition gate G + can be approximated with at most ϵ > 0 error using O (log(1 /ϵ ) /ϵ ) gates with type either G 1 -or G × .

Proof. Let t (1) 1 , t (1) 2 ∈ [0 , 1] . We define the mapping

We consider the sequence t ( τ +1) 1 , t ( τ +1) 2 := f ( t ( τ ) 1 , t ( τ ) 2 ) for τ = 1 , 2 , . . . . We will show that

for C ≥ log(1 /ϵ ) log ( 1 1 -ϵ ) + 1 = Θ(log(1 /ϵ ) /ϵ ) . We first observe that t ( τ +1) 1 ≥ t ( τ ) 1 and t ( τ ) 1 + t ( τ ) 2 = t (1) 1 + t (1) 2 . We consider two cases.

If t ( C ) 1 > 1 -ϵ , then t (1) 1 + t (1) 2 = t ( C ) 1 + t ( C ) 2 ≥ 1 -ϵ , in turn implying that min(1 , t (1) 1 + t (1) 2 ) ≥ 1 -ϵ . This clearly implies (6) as t ( C ) 1 ∈ [0 , 1] .

In the contrary case, if t ( C ) 1 ≤ 1 -ϵ , then t ( C ) 2 ≤ ∏ C -1 τ =1 t ( τ ) 1 ≤ (1 -ϵ ) C -1 ≤ ϵ , where we used the fact that t ( τ ) 1 ≤ t ( C ) 1 ≤ 1 -ϵ . So, min(1 , t (1) 1 + t (1) 2 ) = min(1 , t ( C ) 1 + t ( C ) 2 ) = t ( C ) 1 + t ( C ) 2 ≤ t ( C ) 1 + ϵ .

Lemma I.6. Any addition gate G + can be approximated with at most ϵ > 0 error using O (log(1 /ϵ ) /ϵ ) gates with type either G 1 -or G × , each with error ϵ ′ = O ( ϵ/C ) = O ( ϵ 2 ) .

Proof. Let t (1) 1 , t (1) 2 ∈ [0 , 1] . We consider the sequence

Here, ( t ( τ +1) 1 , t ( τ +1) 2 ) can be indeed obtained from ( t ( τ ) 1 , t ( τ ) 2 ) using 5 gates with type either G 1 -or G × , each with error at most ϵ ′ . We will show that

for C ≥ log(6 /ϵ ) log ( 1 1 -ϵ/ 12 ) + 1 = Θ(log(1 /ϵ ) /ϵ ) . Below, we will take ϵ ′ := ϵ/ (12 C ) . We first observe that t ( τ +1) 1 + t ( τ +1) 2 = t ( τ ) 1 + t ( τ ) 2 ± 6 ϵ ′ , thereby implying that t ( C ) 1 + t ( C ) 2 = t (1) 1 + t (1) 2 ± 6 Cϵ ′ . Further, t ( τ ) 1 ≤ t ( τ +1) 1 +5 ϵ ′ . Hence, t ( τ ) 1 ≤ t ( C ) 1 +5 Cϵ ′ . We consider two cases.

If t ( C ) 1 > 1 -ϵ/ 2 , then t (1) 1 + t (1) 2 ≥ t ( C ) 1 + t ( C ) 2 -6 Cϵ ′ ≥ 1 -ϵ/ 2 -6 Cϵ ′ = 1 -ϵ since ϵ ′ = ϵ/ (12 C ) . This implies that min(1 , t (1) 1 + t (1) 2 ) ≥ 1 -ϵ , and (7) follows.

In the contrary case, if t ( C ) 1 ≤ 1 -ϵ/ 2 , then t ( C ) 2 ≤ (1 -ϵ/ 2 + 5 Cϵ ′ ) t ( C -1) 2 + ϵ ′ = (1 -ϵ/ 12) t ( C -1) 2 + ϵ ′ . Thus, t ( C ) 2 ≤ Cϵ ′ +(1 -ϵ/ 12) C -1 ≤ ϵ/ 4 , where we used the fact that C ≥ log(6 /ϵ ) log ( 1 1 -ϵ/ 12 ) +1 . In turn, we have t (1) 1 + t (1) 2 ≤ t ( C ) 1 + t ( C ) 2 +6 Cϵ ′ ≤ 1 + ϵ/ 4 .

We conclude by observing that t ( C ) 1 = t (1) 1 + t (1) 2 -t ( C ) 2 ± 6 Cϵ ′ = t (1) 1 + t (1) 2 ± 3 ϵ/ 4 .

As a result, for any absolute constant ϵ > 0 , we can reduce in polynomial time an ϵ -GCIRCUIT instance consisting of M gates with a set of types G = { G 1 -, G + } to an ϵ ′ -GCIRCUIT instance consisting of Θ( M ) gates with a set of types G ′ = { G 1 -, G × } , so long as ϵ ′ = O ( ϵ 2 ) is sufficiently small. Together with Theorem I.4, we arrive at the following conclusion.

Proposition I.7. Even when G := { G × , G 1 -} , there is an absolute constant ϵ > 0 such that computing an ϵ -fixed point of F with respect to ∥ · ∥ ∞ is PPAD -complete.

Having established this reduction, a function F arising from such circuits is clearly a multilinear that can be expressed in the form of Lemma I.1. Indeed, we simply observe that, for S ⊆ [ N ] and ¯ S ⊆ [ N ] with S ∩ ¯ S = ∅ , we have

where the summation is over all partitions S ′ , S ′′ of [ N ] \ ( S ∪ ¯ S ) . So, combining with Proposition B.1, we arrive at Theorem 3.3, which is restated below for the convenience of the reader.

Theorem 3.3. If a regret minimizer R outputs strategies in [0 , 1] N , it is PPAD -hard to guarantee Reg Φ β ≤ ϵ/ √ N , even with respect to low-degree deviations and an absolute constant ϵ > 0 .

In the above reduction, we used the inequality ∥ · ∥ ∞ ≤ ∥ · ∥ 2 so as to translate the guarantee of Proposition B.1 in the language of Theorem I.4. That inequality can be loose, and so instead let us explain how one can obtain sharper hardness results by relying on a stronger complexity assumption which pertains to the so-called ( ϵ, δ ) -GCIRCUIT problem. This relaxes the ϵ -GCIRCUIT problem by allowing at most a δ -fraction of the gates to have an error larger than ϵ . In this context, Babichenko et al. [2016] put forward the following conjecture.

Conjecture I.8 (Babichenko et al., 2016) . There exist absolute constants ϵ, δ > 0 such that solving the ( ϵ, δ ) -GCIRCUIT problem with M gates requires 2 ˜ Ω( M ) time.

In light of the simplifications observed by Filos-Ratsikas et al. [2023] (Theorem I.4) in conjunction with Lemma I.6, it is not hard to show that Conjecture I.8 can be equivalently phrased by restricting the gates to involve solely multilinear operations-analogously to Proposition I.7. Namely, if the number of gates increases by a factor of C = C ( ϵ ) , it suffices to take ϵ ′ = Θ( ϵ/C ) and δ ′ = Θ( δ/C ) . Now, the point is that Conjecture I.8 is more aligned with a guarantee in terms of ∥ · ∥ 2 . Indeed, if at least a δ -fraction of the gates incur at least an ϵ error, it follows that ∥ F ( x ) -x ∥ 2 ≥ ϵ √ δ √ N (we can assume here that δN is an integer). We can thus strengthen Theorem 3.3 as follows. Theorem I.9. Suppose that Conjecture I.8 holds. If R outputs strategies in [0 , 1] N , guaranteeing Reg Φ β ≤ ϵ requires time 2 ˜ Ω( N ) , even with respect to low-degree deviations and an absolute constant ϵ > 0 .

I.2 Proofs from Section C

We first spell out the construction that establishes Theorem C.2, which is a refinement of the algorithm of Gordon et al. [2008] described earlier in Appendix A. In particular, the one but crucial difference lies in using an ϵ -expected fixed point (Line 3). In the context of Algorithm 1, we assume that the interface of a regret minimizer R consists of two components: R . NEXTSTRATEGY () , which returns the next strategy of R ; and R . OBSERVEUTILITY ( · ) , which provides to R as feedback a utility function, whereupon R may update its internal state accordingly.

We next show that there is a certain equivalence between expected fixed points and Φ -regret minimization over ∆( X ) , which mirrors the construction of Hazan and Kale [2007].

Proposition C.3. Consider a regret minimizer R operating over ∆( X ) . If R runs in time poly ( N, 1 /ϵ ) and guarantees Reg T Φ ≤ ϵ for any sequence of utilities, then there is a poly ( N, 1 /ϵ ) algorithm for computing ( ϵD X ) -expected fixed points of ϕ ∈ Φ , assuming that we can efficiently compute E x ( t ) ∼ π ( t ) [ ϕ ( x ( t ) ) -x ( t ) ] at any time t . Here, D X is the diameter of X with respect to ∥ · ∥ 2 .

Proof. Suppose that R outputs a strategy π ( t ) ∈ ∆( X ) . At each time t ∈ [ T ] , we can terminate if ∥ E x ( t ) ∼ π ( t ) [ ϕ ( x ( t ) ) -x ( t ) ] ∥ 2 ≤ ϵD X ; that is, we have identified an ( ϵD X ) -expected fixed point. Otherwise, we construct the utility function

which indeed satisfies the normalization constraint | u ( t ) ( x ) | ≤ 1 . Now, if at all iterations it was the case that ∥ E x ( t ) ∼ π ( t ) [ ϕ ( x ( t ) ) -x ( t ) ] ∥ 2 > ϵD X , we have

since u ( t ) ( π ( t ) ) = 0 and

for all t ∈ [ T ] . This contradicts the assumption that Reg T Φ ≤ ϵ for any sequence of utilities, in turn implying that there exists t ∈ [ T ] such that π ( t ) is an ϵ -expected fixed point. Given that, by assumption, we can compute E x ( t ) ∼ π ( t ) [ ϕ ( x ( t ) ) -x ( t ) ] for any time t , the claim follows.

I.3 Proof of Corollary 4.1

In this subsection, we discuss how expected fixed points (per Definition C.1) can be used to speed up equilibrium computation even in settings where actual fixed points can be computed in polynomial time. In particular, we recall the following result, which was stated earlier in the main body.

Corollary 4.1. For any n -player game in normal form, there is an algorithm that computes an ϵ -correlated equilibrium and runs in time

Indeed, using the algorithm of Blum and Mansour [2007] instantiated with MWU , 15 one can guarantee (average) swap regret bounded as O ( √ A log A/T ) , where T is the number of iterations; hence, taking T = O ( A log A/ϵ 2 ) guarantees at most ϵ swap regret for each player. Further, a function here can be represented as a stochastic matrix on A states; as such, it can be evaluated at any point in time O ( A 2 ) via a matrix-vector product. As a result, each iteration of Algorithm 1 can be implemented with a single oracle call to (2), along with n updates-one for each player-each of which has complexity bounded as O ( A 2 /ϵ ) (Theorem C.7). This implies Corollary 4.1.

Let us compare the complexity of Corollary 4.1 with prior results. First, when ϵ ≫ 0 the best running time follows from the recent algorithms of Dagan et al. [2024] and Peng and Rubinstein [2024], but those quickly became superpolynomial even when ϵ ≈ 1 / log A ; indeed, the number of iterations of their algorithm scales as (log A ) ˜ O (1 /ϵ ) . On the other end of the spectrum, one can compute an exact correlated equilibrium in polynomial time using the 'ellispoid against hope' algorithm [Papadimitriou and Roughgarden, 2008, Jiang and Leyton-Brown, 2011]. The original paper by Papadimitriou and Roughgarden [2008] did not specify the exact complexity of the algorithm, but the subsequent work of Jiang and Leyton-Brown [2011]-which analyzed a slightly different

version based on a derandomized separation oracle-came up with a bound of n 6 A 12 in terms of the number of iterations of the ellipsoid; the overall running time is higher than that. While the analysis of Jiang and Leyton-Brown [2011] can likely be significantly improved-as acknowledged by the authors, we do not expect that algorithm to be competitive unless one is searching for very high-precision solutions.

The original algorithm of Blum and Mansour [2007]-instantiated as usual with MWU -requires computing a stationary distribution of a Markov chain in every iteration. This amounts to solving a linear system, which in theory requires a running time of O ( A ω ) , where ω ≈ 2 . 37 is the exponent of matrix multiplication [Williams et al., 2024]. Without fast matrix multiplication-which is currently widely impractical-the running time would instead be O ( A 3 ) . A strict improvement over that running time can be obtained using the optimistic counterpart of MWU ( OMWU ) [Daskalakis et al., 2021], which has been shown to reduce the number of iterations to ˜ O ( A/ϵ ) without essentially affecting the per-iteration complexity. Corollary 4.1 improves over that in the regime where ϵ ≥ 1 /A ω 2 -1 , where ω ≈ 2 . 37 is the exponent of matrix multiplication [Williams et al., 2024]; without fast matrix multiplication, the lower bound is instead ϵ ≥ 1 / √ A .

Another notable attempt at improving the complexity of Blum and Mansour [2007] was made by Greenwald et al. [2008] using power iteration. For a parameter p ≥ 1 , their algorithm guarantees at most O ( √ Ap/T ) average internal regret with a per-iteration complexity of Ω( A 3 /p ) (without fast matrix multiplication). Casting this guarantee in terms of swap regret, and observing that the overall running time is invariant on p , we see that the resulting complexity is no better than that via OMWU , which we discussed earlier; the approach of Greenwald et al. [2008] is based on regret matching, which incurs an inferior regret bound compared to MWU , but is nevertheless known to perform well in practice. An improvement to the algorithm of Blum and Mansour [2007] was obtained by Yang and Mohri [2017], but requires a further assumption on the minimum probability given to every expert; it is unclear whether such an assumption can be guaranteed in general.

The above discussion concerns algorithms operating with full feedback. In the bandit feedback, Ito [2020] came up with a way to update a single column in each iteration, but it still requires computing a stationary distribution in every iteration. Ito [2020] claims that this can be achieved in time almost quadratic time through the work of Cohen et al. [2017]; however, the complexity of the algorithm of Cohen et al. [2017] depends on the condition number of the Markov chain, and it is unclear how to bound that in our setting. On the other hand, algorithms in the bandit setting do not require access to an expectation oracle. This can be beneficial in certain settings, but our discussion here focuses on the regime where the expectation oracle does not dominate the per-iteration complexity. We conjecture that Theorem C.7 can be generalized to the bandit setting, in which case our improvement will also manifest in the regime where the cost of the expectation oracle far outweighs the per-iteration complexity of Theorem C.7, but that is not within our scope in this paper.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction are proven in Appendices C to E and I.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All limitations and assumptions are stated in Appendices C to E and I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and proofs are given in Appendices C to E and I. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The contribution of the paper is theoretical, and conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The contribution of the paper is theoretical, and we do not foresee any societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores and advances the research direction of linear-swap regret in extensive-form games, extending it to low-degree swaps and bridging the gap to results on general swap regret. The technical contributions include the concept of k-mediator deviations, which relate to low-degree polynomials and depth-k decision trees on n>>k variables, and a crucial observation that if a player is allowed to output a probability distribution over strategies, computing it is sufficient to compute an approximate fixed point of the deviations in expectation.\n\nThe concept of depth-k deviations allows a deviator to select an action after observing a strategy on a single terminal for k adaptive rounds. Choosing $k=1$ or $k=N$ leads to linear-swap deviations or all swap deviations, respectively. The main results are as follows:\n\n1. For depth-k deviations and the strategy space of a hypercube, there is an online learning algorithm that achieves at most $\\\\epsilon$ average $\\\\Phi$-regret in $N^{O(k)} / \\\\epsilon^2$ rounds with $N^{O(k)} / \\\\epsilon$ running time per round.\n\n\n1. For deviations that are polynomials of degree k and the strategy space of a hypercube, there is an online learning algorithm that achieves at most $\\\\epsilon$ average $\\\\Phi$-regret in $N^{O(k)} / \\\\epsilon^2$ rounds with $N^{O(k^3)} / \\\\epsilon$ running time per round.\n2. It is PPAD-hard to guarante a regret of at most $\\\\epsilon / \\\\sqrt{N}$ if a regret minimizer outputs a single strategy in $\\[0,1\\]^N$.\n\n3. For depth-k deviations and extensive-form games, there is an online learning algorithm that achieves at most $\\\\epsilon$ average $\\\\Phi$-regret in $N^{O(k)} / \\\\epsilon^2$ rounds with $N^{O(k)} / \\\\epsilon$ running time per round.\n\n4. For deviations that are polynomials of degree k and extensive-form games of depth $d$, there is an online learning algorithm that achieves at most $\\\\epsilon \\\\Phi$ average $\\\\Phi$-regret in $N^{O(kd)^3} / \\\\epsilon^2$ rounds with $N^{O(kd)^3} / \\\\epsilon$ running time per round.\n\nSTRENGTHS:\n- The results bridge the gap between linear swap and arbitrary swap regret.\n- The technical contributions show the advantage of probabilistic strategies with respect to hardness, i.e., that a fixed point in expectation is enough for optimization.\n\nWEAKNESSES:\n- It seems the gap is bridged \"from below\", i.e., for large $k$ the results may become worse than known bounds.\n- ~~Allowing the learner to output probability distributions of strategies seems to weaken the model and/or the guarantees of results in this model, e.g., more than allowing mixed strategies over pure strategies does.~~ (edit: see rebuttal)\n\nQUESTIONS:\nCould you elaborate on the difference between mixed strategies and probability distributions over strategies? Is it true that the worst-case regret for the former is bounded, but (at least in your results) the worst-case regret is unbounded?\n\nNote: As a reader who is completely unfamiliar with the topic, the preliminaries could have covered a *deviations*.\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to bridge the gap between the $N^{O(1/\\varepsilon)}$ result for attaining the $\\varepsilon$ swap regret and the $\\operatorname{poly}(N)/\\varepsilon^2$ result for attaining the $\\varepsilon$ linear-swap regret for extensive-form games. To this end, the authors generalize the untimed communication deviations proposed by [1] to k-mediator deviations. The authors prove that obtaining the $\\varepsilon$ regret w.r.t. the set of k-mediator deviations in $N^{O(k)}/\\varepsilon^2$ rounds is achievable. As several byproducts, the authors show a sample complexity of $N^{O(kd)^3}/\\varepsilon^2$ for minimizing the regret against degree-k polynomial deviations, and an algorithm with the lowest computation complexity to compute the $\\varepsilon$-correlated equilibria in normal-form games in the medium-precision regime.\n\n[1] Zhang et al. Mediator Interpretation and Faster Learning Algorithms for Linear Correlated Equilibria in General Extensive-Form Games. ICLR, 24.\n\nSTRENGTHS:\n1.\tThe motivation to bridge the gap between the $N^{O(1/\\varepsilon)}$ result for attaining the $\\varepsilon$ swap regret and the $\\operatorname{poly}(N)/\\varepsilon^2$ result for attaining the $\\varepsilon$ linear-swap regret is of interest.\n\nWEAKNESSES:\nMy main concern regarding this work is the presentation. Several key parts of the current writing are not very clear and somewhat hard to follow, detailed as follows. \n\n1.\tThe main motivation of this work is to bridge the gap between the $N^{O(1/\\varepsilon)}$ result for attaining the $\\varepsilon$ swap regret and the $\\operatorname{poly}(N)/\\varepsilon^2$ result for attaining $\\varepsilon$ linear-swap regret in previous works. Does the current result of the $N^{O(k)}/\\varepsilon^2$ sample complexity for obtaining the $\\varepsilon$ regret w.r.t. the set of k-mediator deviations well interpolate between the above two results? It is a bit strange that there seem to be no discussions about the relationship between the results in this work and the two results in previous works. I think the result in this work replicates the $\\operatorname{poly}(N)/\\varepsilon^2$ result when $k=1$, but it is unclear to me in which sense the result in this work will replicate the previous $N^{O(1/\\varepsilon)}$ result.\n2.\tWhy do we need to relate the k-mediator deviations to low-degree polynomials? After relating the k-mediator deviations to low-degree polynomials, the convergence rate is worse by a $N^{O(k^2d^3)}$ factor. There might be some benefits in doing so, but I did not seem to find the relevant descriptions in the paper.\n3.\tThere is a rich line of most related works in the literature, while it is currently a bit heavy to fully understand the advantages of the results in this work against those in previous works merely from the statements in Section 5. It would be better if a table reflecting the convergence rate, the computation complexity, and the required regime of $\\varepsilon$ of the results in this work and those in previous works is included in the paper.\n4.\tThere is an expectation taken over the randomness of the player in the regret defined in Eq. (1), while the previous works study high probability regret (say [2,3]). As such, it seems that the algorithm in this work can only obtain approximate correlated equilibria in expectation, while previous works can obtain the approximate correlated equilibria with high probability. I would suggest the authors incorporate necessary discussions about this point.\n5.\tFor Section 4.1, it seems that the proposed operation of approximating fixed points in expectation can only work with finding approximate correlated equilibria in expectation. Can this trick be extended to the case of finding approximate correlated equilibria with high probability? This is also somewhat related to my question above.\n6.\tIn Line 135, it is not clear to me what is the purpose of constructing the depth-k decision tree deviations. It would be better to first introduce the motivation for doing so before diving into the details of its construction.\n7.\tAt the end of the statement of Theorem 3.4, it seems that it should be something like $N^{O(k)}/\\varepsilon$ (or $N^{O(k)}/\\varepsilon^2$) instead of $N^{O(k)}$?\n8.\tIn Line 228, what does it mean by “succinct representation”?\n9.\tFor Section 4.2, my understanding is that the authors consider a more convenient and suitable formulation than the tree-form formulation to better leverage the connection between low-depth decision trees and low-degree polynomials. Are there any technical difficulties in achieving this?\n10.\tIn this work, most of the time $\\Phi$ is termed as the set of the deviations while sometimes it is termed as the set of the transformation functions. I would suggest the authors unify the descriptions for clarity. \n\nOverall, the current writing prevents me from well evaluating the significance of the results and the technical novelty of this work and hence I am not able to recommend an acceptance of this work.\n\n[2] Bai et al. Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent. NeurIPS, 22.\n\n[3] Farina et al. Polynomial-Time Linear-Swap Regret Minimization in Imperfect-Information Sequential Games. NeurIPS, 23.\n\nQUESTIONS:\nPlease see the weakness part above.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This work seems to design some fast regret minimization algorithms. However, to be honest, I could not find the learning protocols nor understand the learning problem set-up. It is hard for me to find what information is revealed in each round after taking some action. 
So, I suggest adding some simple applications to help readers to understand the learning protocol, for example, the tree-form decision problems.

Major comments:

Is there any lower bound? How can I measure the tightness of the presented results?
Is it related to P vs NP?  Can it be reduced to an NP-hard problem?
For theoretical analysis, is it finite-time analysis or asymptotic analysis?
I know it is a theory work, but is it any experiments?

Minor comments:

In Line 110, a comma is missing in $\{ \ \}$.

In Line 143, it is better to say $x[j]'s$.



In Line 271, it should be \citep{} for the citation.

STRENGTHS:
see the first box

WEAKNESSES:
see the first box

QUESTIONS:
see the first box

LIMITATIONS:
I hope to see some experiments though
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15