On Tractable $\Phi$-Equilibria in Non-Concave Games
Authors: Yang Cai
Constantinos Costis Daskalakis
Haipeng Luo
Chen-Yu Wei
Weiqiang Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Non-Concave Games
$\Phi$-Equilibrium
$\Phi$-Regret Minimization
Learning in Games
Primary area: algorithmic_game_theory
TL;DR: We initiate the study of *tractable* $\Phi$-equilibria in non-concave games and examine several natural families of strategy modifications.

Abstract: While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to a coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when utilities are non-concave -- a common scenario in machine learning applications involving strategies parameterized by deep neural networks, or when agents' utilities are computed by neural networks, or both. Non-concave games introduce significant game-theoretic and optimization challenges: (i) Nash equilibria may not exist; (ii) local Nash equilibria, though they exist, are intractable; and (iii) mixed Nash, correlated, and coarse correlated equilibria generally have infinite support and are intractable. To sidestep these challenges, we revisit the classical solution concept of $\Phi$-equilibria introduced by Greenwald and Jafari [GJ03], which is guaranteed to exist for an arbitrary set of strategy modifications $\Phi$ even in non-concave games [SL07]. However, the tractability of $\Phi$-equilibria in such games remains elusive. In this paper, we initiate the study of tractable $\Phi$-equilibria in non-concave games and examine several natural families of strategy modifications. We show that when $\Phi$ is finite, there exists an efficient uncoupled learning algorithm that approximates the corresponding $\Phi$-equilibria. Additionally, we explore cases where $\Phi$ is infinite but consists of local modifications, showing that Online Gradient Descent can efficiently approximate $\Phi$-equilibria in non-trivial regimes.

On Tractable Ø-Equilibria in Non-Concave Games
Yang Cai Yale University yang. ca @yale....... edu
Constantinos Daskalakis
Haipeng Luo University Southern California haipeng @usc.edu
MIT CSAIL
postis@csail.mit.cc
Chen-Yu Yu Wei
Weiqiang Zheng
University of Virginia chenyu..ei@virgee.
Yale University weiqiang zhanggaaleeeeeeeededeeoo
Abstract
While Online Gradient Descent and other no-regret learning procedures known to efficiently converge to a coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this not the case when utilities are non-concave common scenario in machine learning applications involving strategies parameterized by deep neural networks, or when agents' utilities are computed by neural networks, both. Non-concave games introduce significant game-theoretic and optimization challenges: (i) Nash equilibria may not exist; (ii) local Nash equilibria, though they exist, are intractable; and (iii) mixed Nash, correlated, and coarse correlated equilibria generally have infinite support and are intractable. To sidestep these challenges, concept of @-cquilibria introduced by Greenwald and Jafari [GJ03], which guaranteed to exist for arbitrary set of strategy modifications & even in non- concave games [SL07]. However, the tractability of @-equilibria in such games remains elusive. In this paper, initiate the study of tractable @-equilibria in concave games and examine several natural families of strategy modifications. We show that when & is finite, there exists efficient uncoupled learning algorithm that approximates the corresponding @-equilibria. Additionally, we explore cases where & is infinite but consists of local modifications, showing that Online Gradient Descent can efficiently approximate Ø-equilibria in non-trivial regimes.
revisit the classical solution
Introduction
Von Neumann's 's celebrated minimax theorem establishes the existence of Nash equilibrium in all two-player zero-sum games where the players' utilities are continuous as well as concave in their own strategy [Neu28]. This assumption that players' utilities are concave, quasi-concave, in their own strategies has been cornersoone for the development of equilibrium theory in Economics, Game Theory, and host of other theoretical and applied fields that make use of equilibrium concepts. In particular, gquasi-)concaviig
key for showing the existence of many types of equilibrium, from generalizations of min-max equilibrium [Fan53; Sio58] to competitive equilibrium in exchange economies [AD54; McK54], mixed Nash equilibrium in finite normal-fom formal-form games [Nas50], and, more generally, Nash equilibrium in (quasi-)concave games [Deb52; Ros65].
'Throughout this paper, we model games using the standard convention in Game Theory that each player has utility function that they want maximize. This is, of course, equivalent modeling the players loss minimizers, convention more common learning. When we say that player's utility concave (respectively non-come concave) their strategy, this is the same as saying that the player's loss is convex (respectively non con-convex) in their strategy.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: A comparison between different solution concepts in multi-player non-concave games. We include definitions of Nash equilibrium, mixed Nash equilibrium, (coarse) correlated equilibrium, strict local Nash equilibrium, and second-order local Nash equilibrium in Appendix B. We also give a detailed discussion on the existence and complexity of these solution concepts in Appendix B.
Solution Concept
Inceative Guarantee
Existence
Complexity
Nash equiibrium
Mixed Nush equilibrium
Global stability
NP-hard
(Course) Correlated equilibrium
[MK87; AZ22]
Strict Nash equilibrium
Local stability
Second-order local Nash equilibrium
Second-order stubility
X
Local Nash equilihrium
First-order order stability
PPAD-hard [DS721]
Stability against
Efficient e-appoxximaton
1-cquilibrium (finite (+))
finite deviations
for any >> 0 (Theorem 2)
Efficient s-approximation
Couv(1)( (oux( eqquilbrruu (finite (b(])
for e = ((F)) (Theorem
Efficient E-approximation via GD/OG for c 12(5*) (Theorem 3,9)
4 (6) -qquiiibbrr
First-order stability when (()"))
Tm (6) -equllibbru
Elficient approxmmation via no-regret learning for = (3(6P) (Theorem 5)
Not only are equilibria guaranteed to exist in concave games, but is also well-established a long line of work the interface of game theory, learning and optimization whose origins can be traced to Dantzig's work on linear programming [Geo63], Brown and Robinson's 's work on fictitious play [Bro51; Rob51J, Blackwell's approachability theorem [BIa56] and Hannan's consistency theory [Han57]-that several solution concepts are efficiently computable both centrally and via decentralized learning dynamics. For instance, it is well-known that the learning dynamics produced when the players of a game iteratively update their strategies using no-regret learning algorithms, such as online gradient descent, guaranteed to converge to Nash equilibrium in two- player zero-sum concave games, and to coarse correlated equilibrium in multi-player general-sum concave games [CL066] The existence of such simple decentralized dynamics further justifies using these solution concepts to predict the outcome of real-life multi-agent interactions where agents deploy strategies, obtain feedback, and use that feedback to update their strategies.
While (qunsi-)concave utilities have been instrumental in the development of equilibrium theory, described above, they are also too restrictive assumption. Several modern applications and outstanding challenges in Machine Learning, from training Generative Adversarial Networks (GANs) to Multi- Ageet Reinforcement Learning (MARL) as well as generie multi-agent Deep Learning settings where the agents' strategies are parameterized by deep neural networks or their utilities are computed by deep neural networks, or both, give rise to games where the agents' utilities are non-concave in their own strategies. We call these games non-concave, following [Das22].
Unfortunately, classical equilibrium theory quickly hits wall in non-concave games. First, Nash equilibria are no longer guaranteed to exist. Second, while mixed Nash, correlated and coarse correlated equílibria do exist-under convexity and compactness of the strategy sets [Gli52], which we have been assuming all along in our discussion so far, they have infinite support, in general [Kar14]. Finally, they are computationally intractable; so, a fortiori, they are also intractable to attain via decentralized learning dynamics.
In view of the importance of non-concave games in emerging ML applications and the afore-described state-of-ad offariiirs our investigation is motivated by the following broad and largely open question:
Question from [Das22]: Is there theory of non-concave games? What solution concepts are meaningful, universal, and tractable?
1.1 Contributions
We study Daskalakis' question through the lens of the classical solution concept of deequilibria introduced by Greenwald and Jafari [GJ03]. This concept is guaranteed to exist for virtually any set of strategy modifications Φ even in non-concave games, as demonstrated by Stoltz and Lugosi [SL07].2 However, the tractability of --equilibria in such games remains elusive. In this paper, we initiate the study of tractable dh-eqiilibia in non-concave games and examine several natural families of strategy modifications.
Stoltz and Lugosi [SL07] only require the elements of & to be measurable functions.

---

Figure 1: The relationship between different solution concepts in non-coneave games. An arrow from one solution concept to another means the former is contained in the latter. The dashed arrow from Conv -quuiibqull @ Fiinite equiliiba means the former is contained in the latter when
(6) Pinitn-
CCC
Finite Eq.
Conv Conv(D(() (D))) -Eq. & -Bq.
CF 1
MNE
Local NE
NE
Global Deviation
Local Deviation
D-Equilibrium. The concept of Q-equilibrium generalizes (coarse) correlated equilibrium. d- equilibrium is joint distribution over I the Cartesian product of all players' strategy sets, and is defined in terms of set, Φ.X; of strategy modifications, for each player i. The set t. contains functions mapping X; to itself. joint distribution over strategy profiles qualifies as = IT -quiliuuii if no player can increase their expected utility by using any strategy modification function, 0; E & on the strategy sampled from the joint distribution. The larger the set 4, the stronger the incentive guarantee offered by the --equilibrium. For example, if 4 contains all constant functions, the corresponding --equilibrium coincides with the notion of coarse correlated equilibrium. Throughout the paper, we also consider e-approximate d-equilibria, where no player can gain more than e by deviating using any function from &.. We study several families of & and illustrate their relationships Figure
Finite Set of Global Deviations. The first case we consider is when each player i's set of strat- egy modifications, Φ contains finite number of arbitrary functions mapping X; itself. As shown in [GJ03], if there exists an online learning algorithm where each player i is guaranteed have sublinear -eX-rregret the empirical distribution of joint strategies played converges @ = II x equilibrium. Gordon, Greenwald, and Marks [GGM08] consider o d-regret minimiza- tion but for concave reward functions. and their results, therefore, do apply to non-concave games. Stoltz and Lugosi [SL07] provide an algorithm that achieves no rregret in non-concave games; however, their algorithm requires fixed-point computation per step, making computationally inefficient. Our first contribution is to provide an efficient randomized algorithm that achieves no oh rregret for each player with high probability.
Contribution 1:
Let X' be strategy set (not necessarily compact or convex), and Vi an arbitrary finite set strategy modification functions for X. We design a randomized online learning algorithm that achieves O (VTlog vi) Vi-regret, with high probability, for arbitrary bounded
reward functions on X' (Theorem 2). The algorithm operates in time vT T/ per iteration. every player in non-concave game adopts this algorithm, the empirical distribution of strategy profiles played forms an e-approximate & II "."=eqqilirrium, with high probability, for any E > 0, after poly (,, (max, x loggn iterations.
players have infinitely many global strategy modifications, we can extend Algorithm 1 by dis- cretizing the set of strategy modifications under mild assumptions, such the modifications being Lipschitz (Corollary 1). The empirical distribution of the strategy profiles still converges to the corresponding & @-equilibrium, but at a much slower rate of a where d is the dimension of the set of strategies. Additionally, the algorithm requires exponential time in the dimension per iteration, making it inefficient. This inefficiency is unavoidable, as the problem remains intractable even when T contains only constant functions.
'The existence of the fixed point is guaranteed by the Schauder Cahauy fixed-pone point theorem [Cau01], generalization of the Brouwer fixed-por point theorem. Hence, it's unlikely such fixed points tractable.

---

To address the limitations associated with infinitely large global strategy modifications, natural approach focus on local deviations instead. The corresponding I æ-equilibrium will guarantee local stability. The study of local equilibrium concepts in non-concave games has received significant attention in recent years- see e.g., [RBS16; HSZ.17; DP18; JNJ20; DSZ21J. However, these solution concepts either are not guaranteed to exist, are restricted to sequential two-player zero-sum games [MV21], only establish local convergence guarantees for learning dynamics see e.g.. [DP18; WZB20; FCR20J, only establish asymptotic convergence guarantees seeeess--eeeeee e.g., [Das+23], or involve non-standard solution concepts where local stability is not with respect to a distribution over strategy profiles [HSZ17].
We study the tractability of @-equilibrium with infinitely large sets that consist solely of local strategy modifications. These local solution concepts are guaranteed to exist in general plate- non-concave games. Specifically, we focus on the following three families of natural deviations.
Projection based Local Deviations: Each player i's set of strategy modifications, denoted by Proj (6), contains all deviations that attempt small step from their input in fixed direction and project if necessary, namely are of the form (x) xTx; (a v). where |||||| S 8 and xTx; stands for the fy-projection onto Xt
Convex Combination of Finitely Many Local Deviations: Each player i's set of strategy modi- fications, denoted by Conv (d)), contains all deviations that can be represented as convex combination of a finite set of d-local strategy modifications, i.c., [Ø((z) - S 5 for all Φ E x**(
Interpolation based Local Deviations: each player i's set of local strategy modifications, denoted by Ti (6), that contains all deviations that interpolate between the input strategy and another strategy X3. Formally, each element (xx,ar of X, (d) can represented as (1 - A)xr + Ax' for some x' E X, and À < 8/Dx, (Dx; is the diameter of X.).
For our three families of local strategy modifications, we explore the tractability of d-equilibrium within a regime we term the first-order stationary regime, where ((82)) )( with representing the maximum deviation allowed for a player. An e-approximate 4 ---quilibrium in this regime ensures first- order stability. This regime is particularly interesting for two reasons: (i) Daskalakis, Skoulakis, and Zampetakis [DSZ21] have demonstrated that computing an e-approximate ô-local Nash equilibrium in this regime intractable." This poses an intriguing question: can correlating the players' strategies, as in a D-equilibrium, potentially make the problem tractable? (ii) Extending our algorithm, initially designed for finite sets of strategy modifications, inefficiency; specifically, the running time becomes exponential in one of the problem's 's natural parameters. Designing efficient algorithms for this regime thus presents challenges. Despite these, we show the following:
these three sets of local deviations results in
Contribution 2: For any 5 > 0, for each of the three families of infinite 5-local strategy modifi- cations mentioned above, there exists an efficient uncoupled learning algorithm that converges to an e-approximate D-equilibrium of the non-concave game in the first-order stationary regime, i.e., e = (652).
We present our results for the projection- based local deviation in Theorem 3 and Theorem 9. Our result for the convex combination of local deviations can be found in Theorem 4. Theorem 5 contains our result for the interpolation-
local deviations. Similar to the finite case, our algorithms build on the connection between 0-regret minimization and d-equilibrium. Given that our strategy modifications are non-standard, it a priori unclear how to minimize the corresponding 0--rgrret For instance,
knowledge, no algorithm is known to minimize " (6)-regret even when the reward functions are concave, and provably PPr (6)-regret incomparable to external regret (Examples and 4). However, via a novel analysis, we show that Online Gradient Descent (GD) and Optimistic Gradient (OG) achieve a near-optimal ( (re-ret guarantee (Theorem 3 and Theorem 8). Our results provide efficient uncoupled algorithms to compute e-approximate ( -qquiiiuil in the first-order stationary regime s = 1(
Further related work is discussed in Appendix A.
'The regime e = 2()) trivial when utility Lipschitz.
A strategy profile is considered an e-approximate 5-cal Nash equilibrium if no player can gain more than by deviating within distance.

---

2
Preliminaries
centered at Rd is denoted by Ba(æ,r) := e 1>- sr.. We use || I| for lg norm throughout. We also write (ā for a ball centered at the origin with radius 8. For E RR we use [a]+ to denote max (0, a). We denote Dx as the diameter of a set X.
ball of radius
Continuous Smooth Games. An n-player continuous game has a set of n players [n] := {1,2.... n). Each player E [n]] has a nonempty convex and compact strategy set X, S R4, For joint strategy profile
(=), I[= Xx, the reward of player i is determined by utillity function u, [I X2 + J0, 1]. We denote by Σx=1 d; the dimensionality of the game and assume m (x) [Dx] = D. A smooth game is a continuous game whose utility functions further satisfy the following assumption.
Assumption (Smooth Games). The utility function u,(x;, for any player i E [n] is differen- tiable and satisfies
1. (G-Lipschitzness) V (z)ll G for all i and E [[}=1 Kji
2. (L-smoothness) there exists L3 > 0 such that VVV,, (x( - Va, (z,, S L.|la, 2iil for all z,, x E X; and x=; E H We denote L = max, Li the smoothness of the game.
Crucially, we make no assumption on the concavity of (4(x,,x-i).
@-equilibrium and &-regret. Below formally introduce the concept of Q-equilibrium and its relationship with online learning and 6-regret minimization.
Definition (#-equilibrium [GJ03; SL07]). In a continuous game, a distribution a over joint strategy profiles IT. := X, is an E-approximate @D-quiilirium for some 2 0 and profile of strategy modification sets II D if and only if for all player i E [ri], max Egzo (a(((((s)) x-)] S Ez~a [2)] + E. When E = 0, we call a @-equilibrium
We consider the standard online learning setting: at each day & E [T]. the learner chooses an action xf from a nonempty convex compact set X' C Ih and the adversary chooses possibly non-convex loss function f :x' + R, then the learner suffers a loss f' (x') and receives feedback. this paper, we focus two feedback models: (1) the player receives oracle for (-): (2) the player receives only the gradient f=²x),, The classic goal of an online learning algorithm is to minimize the external regrer defined as Reg" := max Et= (f'(z") (x)). An algorithm is called no-regret if its external regret sublinear in T. The notion of d-regret generalizes external regret by allowing more general strategy modifications.
Definition (d-regret). Let & be a set of strategy modification functions {Ø: X - X). For T 2 1, the --regret of an online learning algorithm is Reg' max ΣT (f'(z') f'(((=t))) ())( An algorithm is called no P-regret if its Q-regret sublinear in T.
funtitonnionfuucciooss
Many classic notions of regret can be interpreted F-regret. For example, the external regret is Pext-regre where Text contains all constant strategy modifications Φz- (x) = x² for all x" E X. The swap regret on simplex A is -rwgp-t where &swap contains all linear transformations 6 A + Am, A fundamental result for learning in games is that no-O-regret learning dynamics in games converge to an approximate @-equilibrium [GJ03].
Theorem ([GJ03]). If each player i's r-ereret upper bounded by Regi, then their empirical distribution of strategy profiles played is an (maxie(n) Regē. (T) apprroxxxi 8-equilibrium.
3 Tractable (-Eqqilibrium for Finite c via Sampling
In this section, we revisit the problem computing and learning an L-equilibrium in non-concave games when each player's 's set of strategy modifications X is finite.
The pioneering work of Stoltz and Lugosi [SL07] gives no-r-regret algorithm for this case where each player chooses distribution over strategies in each round. This result also implies convergence to @-equilibrium. However, the algorithm by Stoltz and Lugosi [SI.07] not computationally efficient. In each iteration, their algorithm requires computing distribution that is stationary under
5

---

transformation that can be represented as mixture of the modifications in Q. The existence of such a stationary distribution is guaranteed by the Schauder- Cauty fixed-point theorem [Cau01], but the distribution might require exponential support and be intractable to find.
Our main result in this section is an efficient Ø-regret minimization algorithm (Algorithm 1) that circumvents the step of the exact computation of a stationary distribution. Consequently, our algorithm also ensures efficient convergence a P-equilibrium when adopted by all players.
Algorithm 1: P-regret minimization for non-concave reward via sampling
Input: Froot E X' h 2 2, an external regret minimization algorithm Rg over Φ Output: 4-regret minimization algorithm for X
function NEXT TTATEEY)
p¹ f Rg. NEXTSTTRTTEEO. Note that pt a distribution over 4. return x² < SAMPLESTRATEGY (Trot: h, p²).
function OBSERVEREWARD((u()
Set (2)(0) v' (é(x")) for all @ E Φ
Rg. PR..OBSERVEREWARDD.. (-)).
Algorithm 2: SAMPLESTRATEGY
Input: Troot E EX,h 2. p² (
Output: E X
T1 < Troo--
for 2 < k S h do
3
6 + sample form @ according to p'.
Fg ((1x-1).
return & from (f1,..., 3h uniformly at random.
Theorem 2. Let X be strategy set (nol necessarily compact or convex), T be an arbitrary fimtte set of strategy modifications over X, and ut (.)....,u" be arbitrary sequence of possibly non-concave reward functions from X' to [0. 1]. If we instantiate Algorithm with Rg being the Hedge algorithm over & and h = VT, the algorithm guarantees that, with probability at least 1 - B, it produces sequence of strategies x² ,.. with v-regret at most max Σt= u² (x(x))) -t 8V T(log Φb + 3))31/3) Moreover, the algorithm runs in time O( O(vT(&1) per iteration.
If all players in a non-concave continuous game employ Algorithm 1. then with probability at least 1 B. for any O. the empirical distribution of strategy profiles played forms an e-approximate
Φ = 11 xh -quillibrr, after poly (1) log log (max; [ ,log } iterations.
High-level ideas. We adopt the framework in [SL07]. The framework contains two steps in each iteration t: (1) the learner runs a no-extermal-regeet algorithm over & which outputs p A($) in each iteration I; (2) the learner chooses a stationary distribution jr = Eeso p² (p²)), where we slightly abuse notation to use @(u') to denote the image measure of by 6. However, how to compute the stationary distribution 1² efficiently is unclear. We essentially provide a computationally efficient way to carry out step (2) without computing this stationary distribution.
We first construct an e-approximate stationary distribution by recursively applying strategy modifi- cations from क. The constructed distribution can be viewed as tree. Our construction is inspired by the recent work of Zhang, Anagnostides, Farina, and Sandholm [Zha+24] for concave games. The main difference here is that for non-concave games, the distribution needs to be approximately stationary with respect to a mixture of strategy modifications rather than a single one as in concave games. Consequently, this leads to an approximate stationary distribution with prohibitively high support size
VT (I as opposed to /T in [Zha+24] for concave games.
Despite the exponentially large support size of the distribution, we utilize its tree structure to design a simple and efficient sampling procedure that runs in time vT. Equipped with such sampling

---

procedure, we provide an efficient randomized algorithm that generates sequence of strategies so that, with high probability, the 0-regret for this sequence of strategies is at most ((V Tog (+).
We defer the full proof of Theorem 2 to Section 3.1. An extension of Theorem 2 to infinite & holds when the rewards {u' [ee(rr are G-Lipschitz and admits an a-cover with size N(a) particular, when the set of all M --Lipschitz functions over [0, 1j admits an a-cover with log (() of the order (1/a)" [SL07]. In this case, we have
Corollary There is a randomized algorithm such that, with probability at least 1 - 3, the D-regret d+i is bounded by c. T a42 1og(1/8), where only depends on G and M. The algorithm runs in time poly(T, ((T-1/((42))
3.1 Proof of Theorem 2
For a distribution μ E (x)) over strategy space X, we slightly abuse notation and define its expected
utility as u² (1) = Ezzz (x)) ] [0, 1]. We define 6(μ) the image of under transformation ń. In each iteration t. the learner chooses their strategy x² E according to the distribution μr" For
sequence of strategies {x" {x"hxe(r) he([)) the P-regret is Reg m E (u'()(x)) u' (x()))
Algorithm uses an external regret minimization algorithm Rs over & which outputs a distribution p² E ((D). We can then decompose the 4-regret into two parts.
T max Regó
Σ u' (o(x') Ey u (d'('))))
Σ Egrap' | (d'(z))) -
Φ
I: extemaa regret over +
approximation error stationary distribution
Bounding the external regret over &. The external regret over & can be bounded directly. This is equivalent online expert problem: in each iteration f, the external regret minimizer Rg chooses p' E A and the adversary then determines the utility each expert Φ E & as u' (²(())) We choose the external regret minimizer Ri to be the Hedge algorithm [FS99]. Then have max
Ei= ((((z))) Ey (d' (z())))] < 2 Ø
(Theorem 6), where we use the
fact that the utility function w' is bounded in [0,1].
II: Bounding error due to sampling from approximate stationary distribution. We first define a distribution j over X' using a complete & -ary tree with depth h. The root this tree an arbitrary strategy Troot X. Each internal node has exactly [ children, denoted as {ø(z)) }eed. The distribution μ² is supported on the nodes of this tree. Next, we define the probability for each node under the distribution y².. The root node Troa receives probability K- The probability of other nodes is defined in a recursive manner. For every node x: = Φ(xx) where Tp is its parent, x receives probability pr [xr] Pr [xrp]] (Ø). It is then clear that the total probability of the children of a node xp is exactly Pr.+ [rr is Fp s child] E. Pr[sp] p (Φ) Pr[æø] Denote the set of nodes in depth i as N₁. We have Pr [x E N&] = x for every depth
h. Thus the distribution j² points and is well-defined. By the construction above, we know x² output by Algorithm 2 is sample from p².
supports on [
Now show that the approximation error of pt is bounded by O We can evaluate the approxi- mation error of μ²:
A
Egpp' (d'("")))) - ud()) = Egpnp'
Σ
Pr(z)u((o(x)) u
Σ
Σ
Pr Priaju ne
klIzEN
No
Σ
Edanpp Pr(aju'
-Pr(aju (x))
7

---

We recall that for node - x(xp) with 2p being its parent, we have pr [a]] = Pr (xpl (o).
Thus for any S we have
E Eang' Pr(o)a((0()))
- Pppolul(a)) -
Σ
Σ
p'
NEN
NEN&
SEO
Σ
Pr(x)u((x)
Σ
Pr(a))
a
-EN
NEN
Using the above equality, we get
Egg
[u' (é(n'))) un (1¹)
h-1
h
Σ Σ Pr(x))((x) +
E E.
EE' (()) P Pr(c))'(((1))
Pr(c)]u (a)) (x)
Pr 1
Py(Fmon (Iroa)
k= 2EN24
ZEN; dES
2= N
Pr(eus -
na Pr(arool (Iroon) S h
IEN, $
where in the last inequality we use the fact that EmeN, Pr [x] 1/2 for all S & S h and the utility function u' is bounded in [o. 1]. Therefore, for x² ~ μ², the sequence random variables 21 is super-martingale. Thanks to the boundedness of the utility function, we can apply the Hoeffding-Azan -Azunn Inequality and get for any E > 0.
{Ei=1 (Bonp" (u' (o(x*))] u²(x') +))+21
1 h
Pr
Ba [x(()())) ul u'(x')
2E
exx
(1)
ST
Combining the bounds for and II with /8T log (1/3) and h VT, we have, with probability 1 - 3, that Reg S T [ + + log(1 1/3) s 8 /TTol + + log(1/3)).
Convergence to L-equilibrium. If all players in non-concave continuous game employ Algo- rithm 1, then we know for each player i, with probability 1 E. its rrerret is upper bounded by 8 T [ + lo)(n/B)). By a union bound over all n players, we get with probability 1 - B, every player i's rrerret is upper bounded by 8 / +og(n, ())] Now by The-
orem 1. we know the empirical distribution of strategy profiles played forms an E-approximate
Φ = 11 -= dh equiiibrru, as long as T 2 64(log(max; ( + log(n/3))/=² iterations.
4 Approximate Q-Equilibria under Infinite Local Strategy Modifications
This section studies L-equilibrium when [ is infinite. is. in general, computationally hard to compute fD-equilibrumm even if & contains all constant deviations. Instead, we focus on & that consists solely of local strategy modifications. We introduce several natural classes of local strategy modifications and provide efficient online learning algorithms that converge to e-approximate d- cquilibrium in the first-order stationary regime where = (88 L). These approximate d-equilibria guarantee first-order stability.
Definition (8-local strategy modification). For each agent i, we call a set of strategy modifications T. 5-local for all x X; and o; xx [1);(x)) æ1| < 8. We use notation (x)(()) (8) to denote 5-local strategy modification set for agent i. We also use ((8) 11 x (8) to denote a profile of ô-local strategy modification sels.
Below we present a useful reduction from computing an e-approximate ( -quiiiibii in non- concave smooth games to x (or-rerrttt minimization against convex losses for any > 8-L The key observation here is that the L-smoothness of the utility function permits, within a d-neighborhood.
L a 2 approximation using a linear function. We defer the proof to Appendix D.
Lemma (No & (e)-Rett for Convex Losses to Approximate & -qqiliuuii
Con-Conc Games). For any T 2 and 8 > 0, let A be algorithum that guarantees achieve more than (8) (8))())reggre for convex loss functions for each agent [n]. Then
8

---

1. The &x reerret of for non-convex and L-smooth losses is at most Reg (6)
LTTT Vi.
2
2. When every agent employs A in non-concave I--mooohh game, their empirical distribution of the joint strategies played converges to a (maxig)) (Reg" (s //T 5 "apppppooi 4 ((q-liiiii
4.1 Projection- Local Strategy Modifications
In this section, we study a set of local strategy modifications based on projection. Specifically, the set Proj (6) encompasses all deviations that essentially add fixed displacement vector e to the input strategy and project back to the feasible set: "i
{ (a) xx u] v
Ba())). It is clear that (() - xlI & ||||| < 6. The induced $ (r)-ret is RegProj.6 = max ( Ei= (f'(z*) - ('Ixx - =])). We also define (Prro(6)] = II (
By Lemma to achieve convergence an approximate (f)eeeiiiiuuu suffices to minimize (o: regrrtt against convex losses. However, to our knowledge, there does not exist an algorithm that minimize "/"r (6) -regret even in the convex case. In fact, external regret and 4i% (6) -regret are provably incomparable: sequence of actions may suffer high Reg" but low Reg (Example 3) or vise versa (Example 4). At a high level, the external regret competes against fixed action, whereas $1 (r)-reret more akin to the notion of dynamic regret, competing with a sequence varying actions. Despite this, surprisingly, we show that classical algorithms like Online Gradient Descent (GD) and Optimistic Gradient (OG), known for minimizing external regret, also attain near-optimal o (r)rrett.. We defer the examples and missing proof's to Appendix E.
Proj (6)-Reret Minimization in the Adversarial Setting. We show that (GD) enjoys an O(GVTDxT)) (6) despite the difference between the external regret and & (6)- regret. First, let us recall the update rule of GD: given initial point x¹ E X and step size > 0, GD updates in each iteration
x 1+1 IIx [x - 1 (
(GD)
The key step in our analysis for GD is simple but novel and general (See Appendix E.2). We extend the analysis to the Optimistic Gradient (OG) algorithm in Appendix E.4.
Theorem 3. Let 5 > O and T E N. For any convex and G- G-Lipschitz loss functions {f X + Rhter, the 4Proi rregret of (GD) with step size 1j > 0 is RegProj S 2q 3 G + 6Dx We
vo( and attain RegT..j.8 S 2G /065 + Dx)T. For any O and any 8 > 0, when all player employ GD in a smooth game, their empirical distribution of played (P)eu(()-euullibriiu in O(L/=²) iterations.
can choose y, optimally as Gv/T
strategy profiles converges to ( 5 )=ap)roxiimaatt
Remark 1. The Prog (8)o can also be viewed as the dynamic regret [Zin03] with changing comparators (p' == xx v]}. However, our analysis does not follow from standard O (1+Pr) q/T) dynamic regret bound of GD [Zin03] since Pr, defined as Σt llp' -"- [|, can be D(1TT).
n
Lower bounds for D re)-ret We complement our upper bound with two lower bounds for x
Proj (6) (regrttt minimization. The first one is 2(5GVT) lower bound for any online learning algorithms against linear loss functions (Theorem 7). The second one is an 225 LT)) lower bound for any algorithm that satisfies the linear span assumption (Proposition 1), which holds for GD and OG against L-smooth non-convex losses. Combining with Lemma 1, this lower bound suggests that GD attains nearly optimal Φ (5) rregrett even in the non-convex setting, among natural family of muddient-based algorithms. We defer the theorem statements and detailed discussion to Appendix E.3.
Improved P (0) RRegRtt in the Game Setting. We complement the R(VT) lower bound in the adversarial setting by considering the game setting where players interact with each other using the same algorithm setting, which has been extensively studied for concave games [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a]. We prove an improved O(T)) individual ("",, rrerret bound for OG (Theorem 9). We defer the details to Appendix E.4

---

4.2 Convex Combination of Finite Local Strategy Modifications
This section considers Conv( where is finite set of local strategy modifications. The set of infinite strategy modifications Conv(@) is defined as Conv( () = (0p(z) = Eseo p(O)ő(z) &p e ( Our main result is an efficient algorithm (Algorithm 3) that guarantees convergence to an E-approximate Conv -equilibriu in a smooth game satisfying Assumption I for any e > L.L. Due to space constraints, we defer Algorithm 3 and the proof to Appendix F.
Theorem 4. Let X be a convex and compact set, & be an arbitrary finite set of ò-local strategy modification functions for X, and u (.).... u²"()) (-) be sequence of G-Lipschitz and L-smooth but possibly non-concave reward functions from X to [O,
1]. If we instantiate Algorithm 3 with
Ra being the Hedge algorithm over A and K = VT, the algorithm guarantees that, with probability at least 1 33, it produces sequence of strategies x² u²....,3² with Conv (onv(o)-rre -regret at most 8 (VTTG5 log [ + vlog(1/3) + 82 LT. The algorithm runs in time VT per iteration.
If all players in a non-concave smooth game employ Algorithm 3, then with probability 1 - 3. any > 0, the empirical distribution of strategy profiles played forms an (E + 8 apprrxxiite
c 11 "i..bii--eeuiiinn after poly (t G,log (max; [b>x .log 3) iterations.
4.3 Interpolation-Baseee Local Strategy Modifications
We introduce natural set of local strategy modifications and the corresponding local equilibrium notion. Given any set of (possibly non-local) strategy modifications Vi = {16 : x X), we define a set of local strategy modifications as follows: for 8 Dx and À E [o, 1]. each strategy modification Φx.v interpolates the input strategy x with the modified strategy (b:)::: formally,
X2,,
fine, v (ō) = (x) == (1- + (x(x) : 1 E V, \
Note that for any si E V and X S 5. we have 1 - z|| A|Iz +|>=~(<z)| S 8, re- specting the locality constraint. The induced Dit (0)-retret can be written as Regiat, 2,6 pa D' E.S (f'(x*) - ('((1 A)x² + An)(="))). To guarantee convergence to the corre-
T
sponding i-qquilibrium, it suffices minimize D v (6) against convex losses, which show further reduces to V-regret minimization against convex losses (Theorem 10 in Appendix G).
CCE-like Instantiation. In the special case where Vi contains only constant strategy modifications (i.e., (xx)= = x= for all r), we get coarse correlated equilibrium (CCE)-like instantiation of local equilibrium, which limits the gain by interpolating with any fixed strategy. We denote the resulting set of local strategy modification simply as $ (6). We can apply any no-external regret algorithm for efficient x (8)-regret minimization and computation of e-approximate & P (m) equiilibrium in the first-order stationary regime as summarized in Theorem 5. We also discuss faster convergence rates the game setting in Appendix G.
GVT its Xxx rrerret is at most 28G Furthermore, for any 8 > O and any = >> LLL, when all players employ the GD) algorithm in smooth game, their empirical distribution of played strategy profiles converges to an (++ LLL
Theorem 5. For the Online Gradient Descent algorithm (GD) [Zin03) with step size vj
2
appproximatee P (8) -eveeiiiii in O(1/²²) iterations.
2
5 Discussion and Future Directions
Lower Bound in the Global Regime When å equals the diameter of our strategy set, it is NP-hard to compute an e-approximate -qquiliiii (for 46)) (d)) P (6)), even when = = (1) and G, L = (p)ly(dd). Moreover, given black-box access to value and gradient queries, finding such equilibria requires exponentially many queries in at least one of the parameters d, G, L, 1/E. These results are presented as Theorem 12 and Theorem 13 in Appendix I.
More Efficient æ-Equilibria We include the discussion of another natural class of local strategy modifications that is based on beam search, where GD suffers linear regret to Appendix H. This result shows that even for simple local strategy modification sets (6)), the landscape of efficient local (e)-regret minimization is already quite rich and many questions remain open. fruitful future direction is to identify more classes of & that admit efficient regret minimization.
10

---

Acknowledgements
We thank the anonymous reviewers for their constructive comments that improves the paper. Y.C. acknowledge the support from the NSF Awards CCF-1942588 (CAREER) and CCF-2342642. W.Z. was supported by the NSF Awards CCF-1942583 (CAREER), CCF-2342642, and a Research Fellowship from the Center for Algorithms, Data, and Market Design at Yale (CADMY), H.L. was supported by NSF award IIS-1943607,
References
[AD54]
Kenneth Arrow and Gerard Debreu. "Existence of an equilibrium for competitive economy". In: Econometrica (1954), pp. 265-290.
[AFS23]
Ioannis Anagnostides, Gabriele Farina, and Tuomas Sandholm. "Near- Opeaaal Phi- Regret Learning in Extensive-Form Games". In: International Conference on Machine Learning (ICML). 2023.
[AGHI9]
Naman Agarwal, Alon Gonen, and Elad Hazan. "Learning non-conve games with optimization oracle". In: Conference on Learning Theory. PMLR. 2019, pp. 18-29. [AHK12] Sanjeev Arora, Elad Hazan, and Satyen Kale. "The multiplicative weights update method: a meta-algorithm and applications". In: Theory of computing 8.1 (2012), pp. 121-164.
[ALW21] Jacob Abernethy, Kevin A Lai, and Andre Wibisono. "Lasttiterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization". In: Algorithmic Learning Theory. PMLR. 2021, pp. 3-47.
[Ana+22a] Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. "Near-optimal no-regret learning for correlated equilibria in multi Malli- general-sum games". In: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing (STOC). 2022.
[Ana+22b] Ioannis Anagnostides, Gabriele Farina, Christian Kroer, Chung- Wein Lee, Haipeng Luo, and Tuomas Sandholm. "Uncoupled Learning Dynamics with O(log T)) Swap Regret in Multiplayer Games". In: Advances in Neural Information Processing Systems (NeurIPS). 2022.
[Aue+02] Peter Auer, Nicolo Gesa-Bianchi, Yoav Freund, and Robert E Schapire. 'The non- stochastic multiarmed bandit problem". In: SIAM journal computing 32.1 (2002), pp. 48-77.
[AZ22]
Amir Ali Ahmadi and Jeffrey Zhang. "On the complexity of finding a local minimizer of a quadratic function over a polytope". In: Mathematical Programming 195.1 (2022),
pp. 783-792.
[AZF19]
Sergul Aydore, Tianhao Zhu, and Dean P Foster. "Dynamic local regret for non-convex online forecasting". In: Advances in neural information processing systems 32 (2019).
[Bai+22]
Yu Bai, Chi Jin, Song Mei, Ziang Song, and Tiancheng Yu. "Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent". In: Advances in Neural Information Processing Systems 35 (2022), pp. 22313-22322
[Ber+23]
Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Francesco Trovò, and Nicola Gatti. "Constrained Phi-Equilibria", In: International Conference on Machine Learning. 2023.
[Bla56]
David Blackwell. "An analog of the minimax theorem for vector payoffs." In: Pacific Journal of Mathematics 6.1 (Jan. 1956). Publisher: Pacific Journal of Mathematics, A Non-profit Corporation, pp. 1-8.
[Bro51]
George W Brown. "Iterative solution of games by fictitious play". In: Act. Anal. Prod Allocation 13.1 (1951), p. 374.
[Bub+15]
Sébastien Bubeck et al. "Convex optimization: Algorithms and complexity". In: Foun- dations and Trendy@ in Machine Learning 8.3-4 (2015), pp. 231-357. Robert Cauty. "Solution du problème de point fixe de Schauder". en. In: Fundamenta Mathematicae 170 (2001). Publisher: Instytut Matematyczny Polskiej Akademii Nauk, pp. 231-244
[Cau01]
(CDT099
Xi Chen, Xiaotie Deng, and Shang-Hua Teng. "Settling the complexity computing two-player Nash equilibria", In: Journal of the ACM (JACM) 56.3 (2009), pp. 1-57.
11

---

[CL06]
Nicolo Desa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[CP20]
Xi Chen and Binghui Peng. "Hedging in games: Faster convergence of external and swap regrets". In: Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), pp. 18990-18999.
[CZ23]
Yang Cai and Weiqiang Zheng. "Accelerated Single-Call Methods for Constrained Min-Max Optimization" In: International Conference on Learning Representations (ICLR) (2023).
[Dag+24]
Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. "From External to Swap Regret 2.0: An Efficient Reduction for Large Action Spaces". In: Proceedings of the 56th Annual ACM Symposium Theory of Computing. 2024,
pp. 1222.
[Das+23]
Constantinos Daskalakis, Noah Golowich, Stratis Skoulakis, and Emmanouil Zam- petakis. "STay-ON-the-Ridee Guaranteed Convergence to Local Minimax Equilibrium in Jonconvex-Nonconeee Games". In: The Thirty Sixth Anmual Conference on Learn- ing Theory. PMLR. 2023, pp. 5146-5198.
[Das22]
Constantinos Daskalakis. "Non-Concave Games: A Challenge for Game Theory's Next 100 Years". In: Cowles Preprints (2022).
[DDJ21]
Jelena Diakonikolas, Constantinos Daskalakis, and Michael Jordan. "Efficient meth- ods for structured nonconvex-nonconcane min-max optimization". In: International Conference on Artificial Intelligence and Statistics (2021).
[Deb52]
Gerard Debreu. "A social equilibrium existence theorem". In: Proceedings of the National Acudemy of Sciences 38. 10 (1952), pp. 886-893.
[DFG21]
Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. "Near-optimal no- regret learning in general games". In: Advances in Neural Information Processing Systems (NeurIPS) (2021).
[DGPO9]
Constantinos Daskalakis, Paul W Goldberg, and Christos Papadimitriou. "The com- plexity of computing a Nash equilibrium", In: Communications of the ACM 52.2 (2009), pp. 89-97.
[DP18]
Constantinos Daskalakis and Ioannis Panageas. "The limit points (optimistic) gra- dient descent in min-man max optimization" " In: the 32nd Annual Conference Neural Information Processing Systems (NeurIPS). 2018.
[DSZ21]
Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. "The complex- ity of constrained min-max optimization". In: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC). 2021.
[Fan53]
Ky Fan. "Minimax theorems". In: Proceedings of the National Academy of Sciences 39.1 (1953), pp. 42-47.
[Far+22a]
Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung- Wei Lee, Christian Kroer, and Tuomas Sandholm. "Near-optimal no-regret learning dynamics for general convex games". In: Advances in Neural Information Processing Systems 35 (2022), pp. 39076- 39089.
[Far+22b]
Gabriele Farina, Andrea Celli, Alberto Marchesi, and Nicola Gatti. "Simple Uncoupled No-regret Learning Dynamics for Extensive- Correlated Equilibrium", In: Journal of the ACM 69.6 (2022).
[FCR20]
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. "Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study". In: International Conference on Machine Learning. PMLR. 2020, pp. 3133- 3144.
[FR21]
Tanner Fiez and Lillian Ratliff. "Local convergence analysis of gradient descent ascent with finite timescale separation". " In: Proceedings of the International Conference on Learning Representation. 2021.
[FS99]
Yoav Freund and Robert E. Schapire. "Adaptive Game Playing Using Multiplicative Weights". In: Games and Economic Behavior 29 (1999), pp. 79-103.
[Geo63]
George B. Dantzig. Linear Programming and Extensions. Princeton University Press, 1963.
12

---

[GGM08] Geoffrey J Gordon, Amy Greenwald, and Casey Marks. "No-regret learning in convex games". In: Proceedings of the 25th international conference on Machine learning. 2008, pp. 360-367.
[GJ03]
Amy Greenwald and Amir Jafari. "A general class of no-regret learning algorithms and game- theoretic equilibria". In: Learning Theory and Kernel Machines: 16th An- nual Conference Learning Theory and 7th Kernel Workshop. COLT/Kernel 2003, Washingion, DC, USA, August 24-27, 2003. Proceedings. Springer. 2003, pp. 2-12. Irving L Glicksberg. "A further generalization of the Kakutani fixed theorem, with application to Nash equilibrium points". In: Proceedings of the American Mathematical Society 3.1 (1952), pp. 170-174.
[Gli52]
[GLZI8] Xiand Gao, Xiaobo Li, and Shuzhong Zhang. "Online learning with non-convex losses and non-stationary regret". . In: International Conference on Artificial Intelligence and Statistics. PMLR. 2018, pp. 235-243.
[GZL23]
Ziwei Guan, Yi Zhou, and Yingbin Liang. "Online Nonconvex Optimization with Limited Instantaneous Oracle Feedback". In: The Thirty Sixth Annual Conference on Learning Theory. PMLR. 2023, pp. 3328-3355.
[Han57]
"Aapaan.AAppiiiimmaaa "Approximation to Bayes risk in repeated play". In: Contributions to James Hanman. the Theory of Games (1957), pp. 97-139.
(Hél+20)
Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. "On- line non-convex optimization with imperfect feedback". In: Advances in Neural Infor- mation Processing Systems 33 (2020), pp. 17224-17235.
[HMC21a]
Nadav Hallak, Panayotis Mertikopoulos, and Volkan Cevher. "Regret minimization in stochastic non-convex learning via proximal-gradien approach". In: International Conference on Machine Learning. PMIR. 2021, pp. 4008-4017.
[HMC21b] Ya-Ping Hsieh, Panayotis Mertikopoulos, and Volkan Cevher. "The limits of min-max optimization algorithms: Convergence spurious non-critical sets". In: International Conference on Machine Learning. PMLR. 2021, pp. 4337-4344
[HSZ17] Elad Hazan, Karan Singh, and Cyril Zhang. "Efficient regret minimization in non- convex games". " In: International Conference on Machine Learning. PMLR. 2017, pp. 1433-1441.
[JNJ20]
Chi Jin, Praneeth Netrapalli, and Michael Jordan. "What is local optimality in nonconvex-monconent nonconcave minimax optimization?" In: International conference on ma- chine learning (ICML). PMLR. 2020, pp. 4880-4889.
[Kar14]
Samuel Karlin. Mathematical Methods and Theory Games, Programming, and Economics: Volume 2: The Theory of Infinite Games. Elsevier, 2014.
[Kar59]
Samuel Karlin. Mathematical methods and theory in games, programming, and eco- nomics: Volume II: the theory of infinite games. Vol. 2. Addision- -eellyy, 1959.
[Kri+15]
Walid Krichene, Maximilian Balandat, Claire Tomlin, and Alexandre Bayen. "The hedge algorithm
continuum". In: International Conference on Machine Learning. PMLR. 2015, pp. 824-832.
[McK54]
Lionel McKenzie, "On equilibrium in Graham's 's model of world trade and other com- petitive systems". In: Econometrica (1954), pp. 147-161.
[MK87]
Katta G. Murty and Santosh N. Kabadi. "Some NP-complete problems in quadratic and nonlinear programming". en. In: Mathematical Programming 39.2 (June 1987), pp. 117-129.
[MM10]
Odalric-Ambbym Maillard and Rémi Munos. "Online learning in adversarial lipschitz environments". In: Joint european conference on machine learning and knowledge discovery in databases. Springer. 2010, pp. 305-320.
[Mor+2la]
Dustin Morrill, Ryan D'Orazi,, Reca Sarfati, Mare Lanctot, James R Wright, Amy R Greenwald, and Michael Bowling. "Hindsight and sequential rationality of correlated play". In: Proceedings of the AAAI Conference Artificial Intelligence. Vol. 35. 6. 2021, pp. 5594.
[Mo++2lb] Dustin Morrill, Ryan D'Orazio, Marc Lanctot, James R Wright, Michael Bowling, and Amy R Greenwald. "Efficient deviation types and learning for hindsight rationality in extensive-fore foors games". In: International Conference on Machine Learning. PMLR. 2021, pp. 7818-7828.
13

---

[MRS20]
Eric Mazumdar, Lillian J Ratliff, and S Shankar Sastry. "On gradient- based learning in continuous games". In: SIAM Journal on Mathematics of Data Science 2.1 (2020), pp. 103-131.
[MV21]
Oren Mangoubi and Nisheeth K Vishnoi. "Greedy adversarial equilibrium: efficient alternative to nonconvex nonconcave min-max optimization". In: Proceedings of the 53rd Annual ACM SICACT Symposium on Theory of Computing. 2021. pp.
nonconvex-nonconcave
[MZ19]
Panayotis Mertikopoulos and Zhengyuan Zhou. "Learning in games with continuous action sets and unknown payoff functions". In: Mathematical Programming 173 (2019), pp. 465-507.
[Nas50]
John Nash Jr. "Equilibrium points in n- n-person games". . In: Proceedings of the national academy of sciences 36.1 (1950), pp. 48 49.
[Neu28]
J Neumann. "Zur theorie der gesellschaftsspieee" In: Mathematische annalen 100.1 (1928), pp. 295-320.
[Pet+22]
Thomas Pethick, Puya Latafat, Panagiotis Patrinos, Olivier Fercoq, and Volkan Cevherå. "Escaping limit cycles: Global convergence for constrained snnconvex-nonccocave minimax problems". In: International Conference on Learning Representations (ICLR).
2022.
[Pil+22]
Georgios Piliouras, Mark Rowland, Shayegan Omidshafiei, Romuald Elie, Daniel Hennes, Jerome Connor, and Karl Tuyls. "Exolutionary dynamics and phi-regret mini- mization in games". In; Journal of Artificial Intelligence Research 74 (2022), pp. 1125- 1158.
[PR24]
Binghui Peng and Aviad Rubinstein. "Fast swap regret minimization and applications to
approximate correlated equilibria". . In: Proceedings of the 56th Annual ACM Symposium on Theory of Computing. 2024, pp. 1223-1234.
[RBS16]
Lillian J Ratliff, Samuel A Burden, and Shankar Sastry. "On the characterization of local Nash equilibria in continuous games". In: IEEE transactions automatic control 61.8 (2016), pp. 2301-2307.
[Rob51]
Julia Robinson. "An iterative method of solving game". In: Annals of mathematics (1951), pp. 296-301.
[Ros65]
Ben Rosen. "Existence and uniqueness of equilibrium points for concave n-person games". In: Econometrica (1965), pp. 520-534.
[RS13]
Sasha Rakhlin and Karthik Sridharan. "Optimization, learning, and games with pre- dictable sequences". In: Advances in Neural Information Processing Systems (2013). Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. "Online learning: Beyond regret". In: Proceedings of the 24th Annual Conference on Learning Theory. JMLR Workshop and Conference Proceedings. 201 1, pp. 359-594.
[RST11]
[Sha24]
Dravyansh Sharma. "No Internal Regret with Non-convex Loss Functions". In: Proceed- ings of the AAAI Conference on Artificial Intelligence. Vol. 38. 13. 2024, pp. 14919 14927.
[Sio58]
Maurice Sion. "On general minimax theorems." " In: Pacific J. Math. 8.4 (1958), pp. 171- 176.
[SL07]
Gilles Stoltz and Gábor Lugosi. "Learning correlated equilibria in games with compact sets of strategies". In: Games and Economic Behavior 59.1 (2007), pp. 187-208. Ziang Song, Song Mei, and Yu Bai. "Sample-efficient learning of correlated equilibria in extensive-form games". In: Advances in Neural Information Processing Systems 35 (2022), pp. 4099-4110.
[SMB22]
[SN20]
Arun Sai Suggala and Praneeth Netrapalli. "Online non-convex learning: Following the perturbed leader is optimal". In: Algorithmic Learning Theory. PMLR. 2020, pp. 845- 861.
[5yr+15]
Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. "Fast conver- gence of regularized learning in games". In: Advances in Neural Information Processing Systems (NeurIPS) (2015).
[VF08]
Bernhard Von Stengel and Françoise Forges. "Extensive-form correlated equilibrium: Definition and computational complexity". In: Mathematics of Operations Research 33.4 (2008), pp. 1002-1022.
14

---

[WZB20]
Yuanhao Wang. Guodong Zhang, and Jimmy Ba. "On Solving Minimax Optimization Locally: A Follow-the-Riie Ridge Approach". In: International Conference on Learning Representations (ICIR). 2020.
[Zha+24]
Brian Hu Zhang, Ioannis Anagnostides, Gabriele Farina, and Tuomas Sandholm. "Effi- cient 0-Regret Minimization with Low-Degree Swap Deviations in Extensive-Form Games". In: Advances in Neural Information Processing Systems. 2024.
[Zin03]
Martin Zinkevich. "Online convex programming and generalized infinitesimal gradient ascent". In: Proceedings of the 20th international conference on machine learning (ICML). 2003.
15

---

Contents
Introduction
1.1 Contributions
2
Preliminaries
3
Tractable @-Equilibrium for Finite & via Sampling
3.1 Proof of Theorem
4
Approximate L-Equilibria under Infinite Local Strategy Modifications
4.1 Projection-Base Local Strategy Modifications
4.2 Convex Combination of Finite Local Strategy Modifications
10
4.3
Interpolation-Based Local Strategy Modifications
10
5
Discussion and Future Directions
10
Related Work
16
B Additional Preliminaries: Solution Concepts in Non-Concave Games
17
Rerget Bound for Hedge
19
Proof of Lemma
19
E Missing Details in Section 4.1
19
E.1 Differences between External Regret and Pegoet
19
E.2 Proof of Theorem
20
E.3 Lower bounds for PP Regret
21
E.3.1 Proof of Theorem
21
E.3.2 Proof of Proposition
22
E,4
Improved PPPoo ( Regret in the Game Setting and Proof of Theorem
22
E.4.1 Proof of Theorem 8
23
E.4.2 Proof of Theorem
F Missing Details in Section 4.2
24
F.1 Proof of Theorem
24
G Missing details in Section 4.3
26
H Beam-Search Local Strategy Modifications and Local Equilibria
27
H.l Proof of Theorem 11
28
Hardness in the Global Regime
28
L.1 Proof of Theorem 13
29

---

1.2
Proof of Corollary 2
30
Removing the D dependence for x' Proj -regret
30
J.1 One-Dimensional Case
30
J.2 d-Dimensional Box Case
A Related Work
Non-Concave Games. An important special case of multi-player games are two-player zero-sum games, which are defined in terms of some function f: X' y R. that one of the two players say the one choosing x E X' wants to minimize, while the other player, the one choosing y E y, wants to maximize. Finding Nash equilibrium in such games is tractable in the convex-concave setting, Le. when f(z y) is convex with respect to the minimizing player's s strategy, I, and concave with respect to the maximizing player's strategy, y, but is computationally intractable in the general setting. Namely, a Nash equilibrium may not exist, and it is NP-hard to determine if one exists and, if so, find it. Moreover, in this case, stable limit points of gradient- based dynamics are not necessarily Nash equilibria, not even local Nash equilibria [DP18; MRS20). Moreover, there are examples including the "Polar Game" [Pet+22] and the "Forsaken Matching Pennies" [HMC21b] showing that for GD / OG and many other no-regret learning algorithms in manconvex-nonconcete
nonconvex-nomcoomau
min-max optimization, the last-iterate does not converge and even the average- iterate fails to be a stationary point. We emphasize that the convergence guarantees we provide for GD OG in Section 4.1 and Section 4.3 holds for the empirical distribution of play, not the average-iterate or the last-iterate.
line of work focuses computing Nash equilibrium under additional structure in the game. This encompasses settings where the game satisfies the (weak) Minty variational inequality [MZ19; DDJ21; 2et+ C723), sufficiently close to being bilinear [ALW21]. However, the study of setting is sparse. Daskalakis, Skoulakis, and Zampetakis [DSZ21] proved the existence and computational hardness of local Nash equilibrium. In more recent work, [Das+23] proposes second-order algorithms with asymptotic convergence to local Nash equilibrium. Several works study sequential two-player zero-sum games with additional assumptions about the player who goes second. They propose equilibrium concepts such as local minimax points [JNJ20], differentiable Stackelberg equilibrium [FCR20], and greedy adversarial equilibrium [MV21]. Notably, local minimax points are stable limit points of Gradient-Descent Ascent (GDA) dynamics [JÑJ20; WZB20; FR21] while greedy adversarial equilibrium can be computed efficiently using second-order algorithms in the unconstrained setting [MV21]. In contrast. to these studies, we focus on the more general case of multi-player non-concave games.
universal solution concepts in the
Local Equilibrium. To address the limitations associated with classical, global equilibrium con- cepts, a natural approach is to focus on developing equilibrium concepts that guarantee local stability instead. One definition of interest is the strict local Nash equilibrium, wherein each player's strat- egy corresponds to a local maximizer of their utility function, given the other players' strategies. Unfortunately, strict local Nash equilibrium may not always exist, as demonstrated in Example 1. Furthermore, a weaker notion-the second-orde local Nash equilibrium, where each player has no incentive to deviate based on the second-order Taylor expansion estimate of their utility, is also not guaranteed to exist as illustrated in Example 1. What's 's more, it is NP-hard to check whether a given strategy profile is a strict local Nash equilibrium or a second-order local Nash equilibrium, implied by the result of Murty and Kabadi [MK87] and Ahmadi and Zhang [AZ22]. Finally, one can consider local Nush equilibrium, first-order stationary solution, which is guaranteed to exist [DS7.2]]. Unlike non-convex optimization, where targeting first-order local optima sidesteps the intractability of global optima, this first-order local Nash equilibrium has been recently shown to be intractable, even in
'Murty and Kabadi [MK87] shows that checking whether a point is a local maximizer of a multi-variate quadratic function is NP hard, and Ahmadi and Zhang [AZ22] shows that computing local maximize of multi-variate quadratic function polytope NP-hard.

---

two-player zero sum non con-coeeave games with joint feasibility constraints [DSZ21]." See Table I a summary of solution concepts in non-concave games.
for
Online Learning with Non-Convex Losses. A line of work has studied online learning against non-convex losses. To cireumvent the computational intractability of this problem, various approaches have been pursued: some works assume restricted set of non-convex loss functions [GLZ18], while others assume access to a sampling oracle [MM10; Kri+15] or access to an offline optimization oracle [AGH19; SN20; Hél+20J
weaker notion of regret [HSZ17; AZF19; HMC2 la; GZL23). The work most closely related to ours is [HSZ.1]]. The authors propose a notion of w-smoothed local regrel against non-convex losses, and they also define local equilibrium concept for non-concave games. They use the idea of smoothing to average the loss functions in the previous w iterations and design algorithms with optimal w-smoothed local regret. The concept of regret they introduce suggests a local equilibrium concept. However, their local equilibrium concept is non-standard in that its local stability is not with respect to distribution over strategy profiles sampled by this equilibrium concept. Moreover, the path to attaining this local equilibrium through decentralized learning dynamics remains unclear. The algorithms provided in [HSZ17; GZL23] require that every agent experiences (over several rounds) the average utility function of the previous w iterates, denoted as F" == - ===0 un ",, Implementing this imposes significant coordination burden on the agents. In contrast, we focus on a natural concept of ((d)-equilibrium which is incomparable that of [HSZ17], and we also show that efficient convergence to this concept is achieved via decentralized gradient- based learning dynamics.
d-regret and 0-equilibrium. The concept of Ø-regret and the associated d-equilibrium is in- troduced by Greenwald and Jafari [GJ03] and has been broadly investigated in the context of concave games [GJ03; SL07; GGM08; RSTII: Pil+22; Ber+23; Dag+24; PR24] and extensive- form games [VF08; Mor+2la: Mor+2lb; Far+22b; Bai+22: SMB22; AFS23; Zha+24]. The work of [Sha24] studies internal regret minimization against non-convex losses. To our knowledge, no efficient algorithm exists for the classes of D-equilibria we consider for non-concave games. Specifi- cally, all of these algorithms, when applied compute (s, ((6)) ((6))--quliibrim for a general 6-local strategy modification set (6)) (using Lemma 1), require running time exponential in either 1/e or the dimension d. In contrast, we show that for several natural choices of (()), e-approximate (8)) -qquilibbr can be computed efficiently, i.e. polynomial in 1/c and d, using simple algorithms.
B Additional Preliminaries, Solution Concepts in Non-Concave Games
We present definitions of several solution concepts in the literature as well as the existence and computational complexity of each solution concept.
Definition 4 (Nash Equilibrium). In continuous game, a strategy profile x E IIi= Xj is a Nash equilibrium (NE) if and only for every player E [n])
u(I', x) S u,(2), Vz' E &
Definition 5 (Mixed Nash Equilibrium). In continuous game, mixed strategy profile p I[; ((X)) (here we denote ((X)) as the set of probability measures over X.) is a mixed Nash equilibrium (MNE) if and only for every player E [n],
(..1..p.)) S u. (p), Vpl E A(2))
Definition 6 ((Coarse) Correlated Equilibrium). In continuous game, distribution a over joint strategy profiles 1
X; is a correlated equilibrium (CE) if and only ffr all player i E [n].
max &. -X
Exso (d.(s:), S (uxo(u((x))
Similarly, a distribution a over joint strategy profiles M is coarse correlated equilibrium (CCE) and only for all player i [n],
max Epso E.E
(2),,_1)))) (elllsE,,,a [u())
II general sum games, it is not hard to see that the intractability results [DGP09; CDT09] for computing global Nash equilibria in bimatrix games imply intractability for computing local Nash equilibria.
17

---

Definition 7 (Strict Local Nash Equilibrium). In a continuous game, strategy profile E II; X3 is strict local Nash equilibrium if and only if for every player E [n], there exists ô > 0 such that (x,,,,,)) S u,(x), Val E (zi, (0 n
Definition 8 (Second-order Local Nash Equilibrium). Consider a continuous game where each utility with respect to I, for any fixed x-i- A strategy profile
function u, ==4) is wwic-differreniinn x E II X3 is second-order local Nash equilibrium if and only if for every player i E [n]. z; maximizes the second-order Taylor expansion of its utility functions at xs, or formally,
(Va,ui(z), zi) + (x' -x;) T Va (x(x)((( - xz) S O, Val E X,
Existence Mixed Nash equilibria exist in continuous games, thus smooth games [Deb52; Gli52; Fan53J. By definition, an MNE is also CE and CCE. This also proves the existence of CE and CCE. In contrast, strict local Nash equilibria, second-order Nash equilibria, or (pure) Nash equilibria may not exist in smooth non-concave game, as we show in the following example.
Example 1. Consider a two-player zero-sum non-concave game: the action sets are X1 = N22
[-1,]] and the utility functions are u1 (211, = u2 ;2) (x1 xz)²² 2. Let x (2) E Xi x X2 be any strategy profile: æ1 = 22, then player 1 is not at a local maximizer; if & x2, then player 2 is not local maximizer. Thus is not strict local Nash equilibrium. Since the utility
function is quadratic, we conclude that the game also has no second-order local Nash equilibrium.
Computational Complexity Consider a single-player smooth non-concave game with a quadratic utility function f - RR. The problem of finding a local maximizer of ʃ can be reduced to the problem of computing NE, MNE, CE, a CCE, a strict local Nash equilibrium, or second-order local Nash equilibrium. Since computing a local maximizer or checking if a given point is local maximizer is NP-hard [MK87], we know that the computational complexities of NE, MNE, CE, CCE, strict local Nash equilibria, and second-order local Nash equilibria are all Na-hard.
Representation Complexity Karlin [Kar59] present a two-player zero-sum non-concave game whose unique MNE has infinite support. Since in a two-player zero-sum game, the marginal distribution of a CE or a CCE is an MNE, it also implies that the representation complexity of any CE or CCE is infinite. We present the example in Karlin [Kar59] here for completeness and also prove that the game is Lipschitz and smooth.
Example 2 ([Kar59, Chapter 7.1, Example 3)). We consider two-player zero-sum game with action
sets X1 X22 = [0, 1]. Let p and q be Iwo distributions over [0, 1). The only requirement for p and q is that their cumulative distribution functions are not finite-step functions. For example, we can take p = q to be the uniform distribution.
Let ur and vn denote the n-th moments of p and g, respectively. Define the utility function
u(x, v) = -ul(,, y) = - U2 (2( == Σ
== 2a (x" - mn) - vn), 05x,yS S 1.
Claim 1. The game in Example 2 is 2-Lipschitz and 6-smooth, and (p, 9) is its unique (mixed) Nash equilibrium.
Proof. Fix any y E (0, 1). since I (y" Un n 2 the series of V.u(x, y) is uniformly convergent. We have VV=u( y)1 < Dm t S 2, y e [0, 1]. Similarly, we have Vu,(z, v) S
n² < 6 for all y E [0, 1]. By symmetry, we also have [, y)1 2 and (3 (z, v) < 6 for all I. [0, 1]. Thus, the game is 2-Lispchitz and 6-smooth.
E 2"
Since + (xx (n) pn)) < x the series of u(x, y)) is absolutely and uniformly convergent.
We have
L (u(,,,)dF, Σ
(y" vn) S (x" (n-d-)ddEpe = O,
2n
n=0
h u(z, vv) =
2 nx - Mn) (8-- vn (n) = O.
-
0
18

---

In particular, (p. q) is mixed Nash equilibrium, and the value of the game is O. Suppose (p', ('))))) is also mixed Nash equilibrium. Then (p. 9') is mixed Nash equilibrium. Note that p supports on every point in (0, 1]. As consequence, we have
0 / y(z.
00 Σ 2" (x" -Mn((// vn))
for all E [0, 1], where V/2 is the n-th moment of q' Since the series vanished identically, the coefficients of each power of a must vanish. Thus u/ = Vn and q' = q. Similarly, we have p' and the mixed Nash equilibrium is unique.
C Rerget Bound for Hedge
Theorem 6 ([AHK12]). In an N- N-expert problem, assume all the rewards are bounded, i.e., u'
[-M, M then the Hedge algorithm with step size = min L Nlog has regret
N7 MMT
T
T
Σ
(u'
Σ t=1
(u' < 2M Vlg
mAX PEA(N)
D Proof of Lemma 1
Let (f") hetr be sequence non-convex T-smooth loss functions satisfying Assumption Let {x* hee(T) be the iterates produced by against {f he(T) Then {x* he') is also the iterates produced by against sequence of linear loss functions ((VS'(x"),)) For the latter, we know
((V o(x²)) s Regex(())
max Φ (8)
Then using L- L-smoothness of {f*} and the fact that 11(()) =1| s 8 for all @ E o(6), we get
(V)' o(x²)) + _ _ x(x')) (x())l²)
E - -( < @E (8)
Σ
max
max
@ (6)
LT
< Reg (6)
This completes the proof of the first part.
Let each player [n] employ algorithm A a smooth game independently and produces iterates
{z*). The averaged joint strategy profile oT that chooses z' uniformly at random from é e [?]
satisfics for any player i e [n].
max Exwo (u,(o(z)), Eera [u [x)] @eo (8)
T (u,(o(a)), - u4 (x(x))
max (E)
t=1
Reg Rega..:
82
+
T
Thus T is a (maxee(n) [Reg (8)) T 8 aaprooxmaatee (6))) -qquliirru.. This completes the proof of the second part.
E
Missing Details in Section 4.1
E.1
Differences between External Regret and Proj -regret
In the following two examples, we show that & (0) regret is incomparable with exteral regret for convex loss functions. sequence actions may suffer high Reg" but low RegProj, (Example 3), and vise versa (Example 4).
19

---

Then the ( rrgrrtt of the sequence {x² ---2=-1b for any ò (0, (0, is O. However, the external regret of the same sequence is 1. By repeating the construction for T.
Example 3. Let fl (x) = P(x) = 2
2 times, we conclude that there exists a sequence of actions with RegProj.8 = 0 and Reg" for all T 22
Example 4. Let (x) 2x and (x) = x foro x E xx= [-1,1 Then the & ( of the sequence {x' 1 x² = 0) for any d E (0, is ö. However, the external regret of the same 2:
sequence is 0. By repeating the construction for &/2 times, conclude that there exists a sequence of 8T and Reg" = for all T 2.
actions with RegProj.s = 2
At a high level, the external regret competes against fixed action, whereas PPPro (6) regrrtt is more akin the notion of dynamic regret, competing with a sequence of varying actions. When the environment is stationary, i.e., f* = f (Example 3), a sequence of actions that are far away from the global minimum must suffer high regret, but may produce low 0y (o)-ret since the change to the cumulative loss caused by fixed-direction deviation could be neutralized across different actions in the sequence. In contrast, in a non-stationary (dynamic) environment (Example 4), every fixed action performs poorly, and sequence of actions could suffer low regret against a fixed action but the oproo (6)-regre that competes with a exed-direction deviation could be large. Nevertheless, despite these differences between the two notions of regret shown above, they are compatible for convex loss functions: our main results in this section provide algorithms that minimize external regret and (6) regret simultaneously.
T Proj
E.2 Proof of Theorem 3
Proof. Let denote e Ba(ō) fixed deviation and define p' = [xx - v). By standard analysis of GD [Zin03] (see also the proof of [Bub+15, Theorem 3.2] ), we have
T
T
Σ
1
(1x²
p²1
|2/41 - p²1 + +1² Vf
217
t=1
T
(1
- p² 11 || p¹ 1²)
+ 82
+ " G2T,
2n
2n 2
where the last step uses ||
p²1 S 8 and
[
S G. Here the terms || - +2
| - p²1 do not telescope, and we further relax them in the following key step.
Key Step: We relax the first term as:
||ot11
x² 11
|
p² 1² = (p' - p2+1 20 - pp² - p² 11
<p' - -p'+1, 2x -2p++1 -112" -1pp'-p5 -p²ta12
22(p' -
, .u) + 2(pp' - 111 1 1 p(1)) - -Ilp² 12
< 2(p' - u)) || 111
where in the last inequality we use the fact that p² is the projection x++1 - v onto X and p
is in X. Now we get telescoping term 2'p'' p²+1, u) and a negative term -lip* 11 The negative term useful for improving the regret analysis the game setting, but we ignore it for now Combining the two inequalities above, we have
8² n
GT
+
+ ,0)
2n
"1
82
" CT + -
F.
82
" GT +
8Dx "
+
1)
2n
27
2
Since the above holds for any with ||||| < 6, it also upper bounds Regrroj.
20

---

E.3 Lower bounds for &proj Reegret
Theorem 7 (Lower bound for (0) -regret against convex losses). For any 'T' 2 2
Dx >
O < 6 < Dx, and G 2 0, there exists distribution DD G-Lipschitz linear loss functions f f'...., over X = [-Dx,, Dx] such that for any online algorithm, its Vrroj -regret on the loss sequence satisfies ED ED|Reg Ep(Regrroj..) D(BGvT).
Remark A keen reader may notice that the ((G lower bound in Theorem does not match the O(GVODxT) upper bound in Theorem 3, especially when Dx >> 8. A natural question is: which of them is tight? We conjecture that the lower bound is tight. In fact, for the special case where the feasible set X is a box, we obtain a Dx-independent bound (d G8VT) using a modified version of GD, which tight when d 1. See Appendix for a detailed discussion.
This lower bound suggests that GD achieves near-optimal & (6) for convex losses. For L-smooth non-convex loss functions, we provide another (2 lower bound for algorithms that satisfy the linear span assumption. The linear span assumption states that the algorithm produces rf-1 E {IIx|ExE ax-x* +b, V fx())] ag, b, E R., Vi E [t]] as essentially the linear combination of the previous iterates and their gradients. Many online algorithms such as online gradient descent and optimistic gradient satisfy the linear span assumption. Combining with Lemma 1, this lower bound suggests that GD attains nearly optimal (rro (6) even in the non-convex setting, among natural family of gradient-based algorithms.
Proposition 1 (Lower bound for Diros (6) -regret against non-convex losses). For any T 8 E (0, 1), and L f'....,f fr on X' = 1] such that for any algorithm that satisfies the linear span assumption, its Proj (0) rregret on the loss sequence Regr 2 LTTT
there exists a sequence of L-Lipschitz and L-smooth non-convex loss functions
E.3.1 Proof of Theorem
Our proof technique comes from the standard one used in multi bandits [Aue+02, Theorem 5.1]. Suppose that f*(x) = g" x. We construct two possible environments. In the first environment, g¹ G with probability with probability ع= and g¹ G with probability 2ts, We use E, and P. to denote the expectation and probability measure under environment i, respectively, for = 1.2. Suppose that the true environment is uniformly chosen from one of these two environments. Below, we show that the expected regret of the learner is at least (T)GVTT)
and g G with probability 1==; in the second environment, g G
Define N4 Di=TH++ 2 0} be the number of times x² is non-negative, and define f (f)..... ). Then we have
E [ - E&
Σ
E [N+ [ - PE [N+ f
f1.
(enumerate all possible sequences of fI:T)
< TT) Σ
IP (f2.T) )
fi.r
TI P22ITV
< T / (2 In 2)LL
(Pinsker's 's inequality)
(2In2) KL. Bernouli (1+e)
Bernuulii (1 2 ))
=T
(2In2)TeIn 1
T
< (4
(2)
--
21

---

In the first environment, we consider the regret with respect to v = 8. Then we have
T
E1 RegProj.s
2 E1
Σ f((')- ff - 8))
E1
Σ
g (x'
- -Ix[x* - -5]]
E
Σ cG(x' -Ix[=' - 81) 2 c0GTE,
Σ
T(x²
2 0} =EÔGIE, [N+],
where in the last inequality
the fact that if xt 2 0 then x'
Txx 8) = x.' - (x' - 5)
because D 2 5. In the second environment, we consider the regret with respect to v = -6. Then
similarly, we have
E2 [eeg R
2 Eg
Σ (at)) (1 [2' + +8])
E2
Σ g'
-xx +5])
= E2
Σ =G(( - =Ix + +5))
2 E0GIB2
Σ Ifoe < 0] = (T R2 (N+)).
Summing up the two inequalities, get
2 [ [RegProid] + E2 [RegProi.))
(e0GT [6 E2(4)))
2
ESGT - e e (4
(by (2))
Choosing E
11 2yT' we can lower bound the last expression by ((GGVT). The theorem is
proven by noticing that 2 (B, [RegFroj.8] + E2 [RegFroj.s)) is the expected regret of the learner.
E.3.2 Proof of Proposition 1
Proof. Consider f [_,, - R such that f(x) 2 x² and let f f for all t [T]. Then any first-order methods that satisfy the linear span assumption with initial point x 0 will produce x² for all E [T]. The "rrr reg-ret thus >x (f(0) f()) -
E.4 Improved Dyro -egrett in the Game Setting and Proof of Theorem 9
Any online algorithm suffers an ((VT) " (re-ret even against linear loss functions by The- orem 7. This lower bound, however, holds only in the adversarial setting. In this section, show improved O individual ( (6)-regret bound under slightly stronger smoothness assumption (Assumption 2) in the game setting, where players interact with each other using the same algorithm, previous results show improved external regret [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a). This assumption is naturally satisfied by finite normal-form games and is also made for results about concave games [Far+22a].
Assumption 2. For any player i E [n], the utility u((x) satisfies || ux(2) - Vz, ux(')l S
L| - for all x, X.
We study the Optimistic Gradient (OG) algorithm [RS13], an optimistic variant of GD that has been shown to have improved individual external regret guarantee in the game setting [Syr+15]. The OG algorithm initializes w E X arbitrarily and g" 0. In each step 1, the algorithm plays xt. receives feedback g¹ = (x²), and updates w², as follows:
/' = x w0 -ngg'1 '], w' 1Ix
(OG)
We show that OG has ((VT) 4 (r)-ret in the adversarial setting and fast (4T)/4) &PPro(() regret and convergence to approximate & equiliibriu in games.
(8)-
Theorem 8 (Adversarial Regret Bound for OG). Let > and T E N. For convex and G-Lipschitz loss functions {f" : X + RheeT, the b (r)-rer of (OG) with step size nj > 0 is
Proj
8Dx
T
RegEroj. 5
17
||g
gd 111
1=1
22

---

Choosing step size 1)
J6Dx we have RegPros.& S 4G 2GVT'
Theorem 9 (Improved Individual $ dr (6)-Regret of OG in the Game Setting). In G-Lipschitz L-smooth (in the sense of Assumption 2) game, when all players employ OG with step size 1] > 0, then for each player i, 5 > 0, and T 2 1, their individual | (6) denoted as Regr G/ + 3 Choosing 1j = min (ni((6D((1II (8D) /G), we have Regrroj. I 4(8D) T33
T&
Regreoj.8 S Tí
+ 2/8DG. Furthermore, for any 8 > and any & > their empirical distribution of played strategy profiles converges to an (E + )approxximmmte - equuiuibir in O(1/E3) iterations.
E.4.1 Proof of Theorem
Proof.
Fix any deviation e that is bounded by 6. Let us define p" w and p' IIx [x* v).
Following standard analysis of OG [RS13], we have
T
Σ
Ma+)) - f'(p') S (ff'z'),, - -p")
1=1
T
1
-p'l1
||10" 2/11²) nl19²* -g--111-- 1
(1=²* w0'l2 + 1||0" - w '11²)
27
2n
T
|
p²11²
1 ||--11 11- + 7/1g² g¹ 111
1
x² w¹ 113) (3)
2n
2n
27
Now we apply similar analysis from Theorem 3 to upper bound the term | - 11:
- -p'11
- -p'1²- ||000 11
= -ppt11 p',2w'
-220- pp' 11 p¹)
-ppt11 p',2u' -nt - 2p') Ip' -1-
= -2(p p', v0 + +2(p" p¹, w 1-1 p²) - -11p² pp' 1|²
= 2(p'-1 - u) + 22pp' p² x² - p²) + +2(p² 11 - pp,, w w-- x²) -x) - -11p² - -p' 11
22(p0 - u' + || w¹ w²111
where in the last- insuuuiuuylltyy we use (p' - p¹ uu p²) < since p² = II [x' - o] and X' is
compact convex set; we also use 2 2a, b) b² S a² In the analysis above, unlike the analysis of GD
where we drop the negative term - -lp' p² 11 we use -lpp' - 121 to get a term || - --------p w²
which can be canceled by the last term in (3).
Now we combine the above two inequalities. 2(p--1 - p², v) telescopes, we get
Since the term || 1²1 cancels out and
T Σ ~ (3) < 40' -p" v) t=1
n6lg' - --111/2 8Dx + 1)
||g 9" 11.
"1
11
1=1
1=1
E,4.2 Proof of Theorem
In the analysis of Theorem 8 for the adversarial setting, the term
2 can be as large 4G In the game setting where every player i employs OG, of ji.e,, V u, (x). depends on other players' action z',- Note that the change of the players' actions ||a* -²=1|1 is only O(7?). Such stability of the updates leads to improved upper bound on ||g 9.- " and hence also improved O Φix xyh,,,()) ree-rtt for the player.
Ig g 1'
23

---

Proof. Let us fix any player i E [n] in the smooth game. In every step t, player i's loss function X is ( V-V,, .) determined by their utility function u, and all players' actions z'. Therefore, their gradient feedback is y = (x.")(x)). For all / 2 2, we have
11g g" 1 = [uz(x')
1,2 1/21|at x2'11" 11
L
b - xx. 212
< 3T,2
||
will + || wn
|| 1-1-1 x 11))
where we use L-smoothness of the utility function in the first inequality; we use the update rule of OG and the fact that gradients are bounded by G in the last inequality.
Applying the above inequality to the regret bound obtained Theorem 8, the individual PProo (6)- regret of player is upper bounded by
5D + nJ + 3nLLC
RegProj.o I
77
Choosing 7 = min((DD/ (6D) /G}, we have RegProj.s ( 2/8DG. Using Lemma we have the empirical distribution of played strategy profiles converge to an (s + apprroxmmmtt (())((@iiuuiuli in O(1/e)) iterations.
F Missing Details in Section 4.2
Algorithm 3: Conv (aanvl))-eegee -reggret minimization for Lipschitz smooth non-concave rewards
Input: D1 X, K 2 2, a no-extermal-eerre algorithm Rg against linear reward over A(D) Output: A Conv( -regret minimization algorithm over X'
function NEXTSTRATEGYO
D' REg.EEXTSTRATTGYY Note that pe is distribution over Φ.
3 Ik c &pe (xx-1), , for all S S K
return x² + uniformly at random from {ZI....,
5 function OBSERVEREWARD V u. u=²())))
(-) f linear reward over A with un (Ø) Vzu'(z'),(rt)) - x²) for all o t.
Rg. OBSERVEREWARD(OO (-)).
F1 Proof of Theorem
Proof Sketch. We adopt the framework in [SL07: GGM08] (as deseribed in Section 3) with two main modifications. First, we utilize the L-smoothness the utilities to transform the problem of external regret over A($) against non-concave rewards into linear optimization problem. Second, we use the technique of "fixed point expectation" [Zha+24] to circumvent the intractable problem of finding fixed point.
Proof. For a sequence strategies f2' (2')e its Conv( -regret is
(u'()(²))) ul (x*)) }
Regconv(
max (ECouv(
Σ ut (op((a)) - u adp (2¹1)
u (Op (x')) ul u'(x))
max pEA(S)
I: exteraa regret (
1:: approximation error of fixod point

---

Bounding External Regret over ()) We can define new reward function (p) := u² (op(2*)) over p ( Since u' non-concave, the reward f is also non-concan and it is computational intractable to minimize external regret. We use locality to avoid computational barrier. Here we use the fact that +- (()) contains only d-local strategy modifications. Then by T-smoothness of u', we know for any p E ((b)
T 1 |10)((0)) -,a"===)) -0'1) -10,,(='-='111 ='12=
8L 2
Ju' (0p(2=) u (x')
Thus we can approximate the non-concave optimization problem by linear optimization problem over (()) with only second-order error L Here we use the notation b + to mean b S a S b c.
2
u' u0pp(x') - u' (x²) (uu (x))) -0) + 8²L
2
Vu'(o'),, Σ p()))(z')) -2²²
82L
=
2
8
Σ (io' x²) 1
2
We can then instantiate the external regret Rg as the Hedge algorithm over reward (p) Eest (V''(x) p(x") 2²) and get
T
n (0p(z²)) u' (6)(())))
Σ
max pEA(.)
Σ (p(0) D' (v))) -29)) { + LT
max pEA(
1=1 PEt
s 22G6///Ia log [ + LT,
where we use the fact that (Vu'(x'), 6(x²) - x²) - [x'))') [|(((=))- - x'1l s Go.
Bounding error due to sampling from fixed point in expectation We choose mı arbitrary point in X. Then we recursively apply op' to get
36 Opr (xx-1) = Σ 12
dE@
We denote 1² Uniform : 1 1 k < KJ. Then the strategy x' ~ p³ is sampled from n², We have that nt is an approximate fixed-point expectation stationary distribution in the sense that
K
Exe [u' (m))(()))) - u (x²)] =
1
Σ #=1
u (ppp - n² (xxx)
K
1 (or (2x)) at (x11)
K
1
K
Thanks to the boundedness of u², we can use Hoeffding-Azuma's inequality to conclude that
1
g²
Pr
u' (6) (x()
2
exp
(-ST
4
K
8T
for any > 0. Combining the above with E ST (log(11/))) and K VT, we get with probability at least 1 - B,
RegConv & 2G8 Tlog /
8 + VT + (8Tlog(1/8)
I 8VT SVVT(GEVlog)
x + + 82
25

---

Convergence to L-equilibrium If all players in a non-conean coneave continuous game employ Algo- rithm 1, then we know for each player i, with probability f its T; reegrtt is upper bounded by
8VT GB G log | + v vlog(n/B) + LT.
By a union bound over all players, we get with probability 1 B, every player i's - -regret is upper bounded by 8vT(G8 log Φ + (o)(m/3)) + 8 L... Now by Theorem 1, we know the empirical distribution of strategy profiles played forms an (s + 82 appppoximmat & -1 4x 128(G2 log 14 "1 Hog(n/A)) iterations.
cquilibrium, as long as T 2
2²
G Missing details in Section 4.3
We introduce natural set of local strategy modifications and the corresponding local equilibrium notion. Given any set of (possibly non-local) strategy modifications V = { : X' + 23, we define a set of local strategy modifications as follows: for ò Dx and À E [0, 1], each strategy modification Φx.e interpolates the input strategy z with the modified strategy (x)): formally,
D Int, v (6) = {ox (x) = (1 - A)ar + (b(x)) + E v.
Note that for any v E Vp and X S Dx we have | (2) z|| ^|az (z))l S 8, re- specting the locality constraint. The induced D v ro)-ret can be written as Regiat, v,, == max max DEV & Et (f*(x*) f((( A)r² + (b(())))) We now define the corresponding DX Pnnt, (8) (8)ucuiii.rrium.
int. ( In continuous game, distribution over strategy profiles is an (e-approximate L -e)equibbrii if and only for all player i E [r]].
Definition 9. Define 1n (0) n "
EuNo [u: - \)2; + App(x;); x>1) < Ezzoluu [u: (x)(z))]= +E.
max NEW AS8/Dxx
Intuitively speaking, when a correlation device recommends strategies to players according to an e-approximate Pint, (d)-liuuli,rium, no player can increase their utility by more than through local deviation by interpolating with (possibly global) strategy modification 1b E The richness of V determines the incentive guarantee provided by an e-approximate P v -'quuuubbiri well as its computational complexity. When we choose V to be the set all possible strategy modifications, the corresponding notion of local squilibrium-limitnn the gain of player by interpolating with any strategy-resemble resembles that correlated equilibrium.
Computation of e-approximate int, v (8) Eqquilibrium By Lemma 1, we know computing an e-approximate L v (6)--eqilibrium reduces to minimizing Φ Pitt (5)-regret against convex loss functions. We show that minimizing & v (6) rregrtt against convex loss functions further reduces to V-regret minimization against linear loss functions.
Theorem 10. Let A be algorithm with Vi rrrrreet Reg (G, for linear and G-Lipschitz loss functions over X. Then, for any 8 > the $ (8)-regret of for convex and G G-Liscchitz loss
functions over X is at most Dx
8_ [RegT(G,D)
Proof. By definition and convexity of ft, we get
T
T
-f(a)-((((())
Σ (2x0---((1 A) + xxb(x!))
max 6 "nt.g (6) 1=1
max
SEV,AS &
t=1
T
(V) -
v Dx
max WEV
26

---

Figure 2: Illustration of (Proj (z) and Beaam, (z)
+ Reaam
(x)
Note that when f is linear, the reduction is without loss. Thus, any worst-case )((r(T))-lwwe) bound for V-regret implies a 52( Dx r(()) lower bound for Pint, v (5)-rerret. Moreover, for any set that admits efficient V V-regret minimization algorithms such as swap transformations over the simplex and more generally any set such that (i) all modifications in the transformations some finite-dimensional space and (ii) fixed point computation can be carried out efficiently for any linear transformations [GGM08], we also get efficient algorithm for computing an e-approximate Pn v (i)-equilibrium the first-order stationary regime.
be represented linear
CCE-like Instantiation In the special case where V contains only constant strategy modifications (i.e. (br) x² for all x), we get a coarse correlated equilibrium (CCE)-like instantiation of local equilibrium, which limits the gain by interpolating with any fixed strategy. We denote the resulting set of local strategy modification simply as T. We can apply any no-external regret algorithm for efficient T -regret minimization and computation of e-approximate Pine (5) eqquilibrri in the first-order stationary regime as summarized in Theorem 5.
The above T (5)-regret bound of O(VT) derived for the adversarial setting. In the game setting, where each player employs the same algorithm, players may have substantially lower external regret [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a] but we need a slightly stronger smoothness assumption than Assumption 1. This assumption naturally satisfied by finite normal- form games and is also made for results about concave games [Far+22a]. Using Assumption and Lemma the no-regret learning dynamics of [Far+22a] that guarantees Olog T) individual external regret in concave games can be applied to smooth non-concave games so that the individual a.Y regret of each player is at most Olog T + 8 LT This gives algorithm with faster ((1/s)
convergence to an (e + L appproximate 2 (0)-equilibrium than GD.
H Beam-Search Local Strategy Modifications and Local Equilibria
In Section 4.1 and Section 4.3, we have shown that GD achieves near-optimal performance for both Φx (6)-regret and & (6)-rreet In this section, we introduce another natural set of local strategy modifications, &.Y ())mm())) which similar to Φ ( Specifically, the set Φ.Y Be (6) contains deviations that try to move far as possible in a fixed direction (see Figure 2 for an illustration of difference between OBeam, (Ir and (Proj..()))
Beam (0) = (1BBeam, (xx) = x X u E Ba( X = max (A x Au E X. 1]}-
It is clear that | (z) x S ||||| < 8. We can similarly derive the notion of Beam regrett and (e, & Beam (0) -qquilirbb. Surprisingly, show that GD suffers linear &x D ( -rggret (proof deferred to Appendix H.I).
Theorem 11. For any 5,n < and T 2 1. there exists sequence of linear loss functions {f' X C [0, 112 IR Reer such that GD with step size n suffers ((8T) x (6)-retret
27

---

H.1 Proof of Theorem 11
Let X' R2 be a triangle region with vertices A (0,0), B (1,1), C (6,0) 0). Consider
e = (-6,0) The initial point is = (0.
The adversary will choose l, adaptively that 2, remains the boundary of X' and cycles clockwise (i.e., A + ... > B + C ... ..).. To achieve this, the adversary will repeat the following three phases:
1. Keep choosing U BA (" denotes the unit vector in the direction of BA) until #++1 reaches B.
2. Keep choosing le "c until 2441 reaches C.
3. Keep choosing & "AC until xt-1 reaches A.
In Phase 1, 3r E AB. By the choice of i = (-ò, 0), we have #t - -x(())) = (-ô(1 2t,1), 0), and 6(1 8(1-a;,1) 0.
the instantaneous regret is
V2
In Phase 2, 22 E BC. By the choice of = (-6,0), we have tt - -c(xz) (0, 0). and the instantaneous regret is O.
In Phase 3, E C'A. By the choice of v (-5, 0), we have 2, - - = (-ō + Tr, 0), and the instantaneous regret is - + Jt,1 S
In each cycle, the number of rounds in Phase is of order O /2) the number of rounds in Phase 2 is between ( and O (22 the number of rounds in Phase 3 is of order O (
Therefore, the cumulative regret in each cycle is roughly
V2 0.55
0+ 5 (-0.58)
0.55 0.55
7
V2
"
On the other hand, the number of cycles no less than
T = (17T). Overall, the cumulative J
+
regret is at least 0.54 0
(n7)) O(T)) as long as 5 < 0.5.
7
I Hardness in the Global Regime
In the first-order stationary regime 8 I /2=/L, (e, 6)-caal Nash equilibrium is intractable, and have shown polynomial-time algorithms for computing the weaker notions of e-approximate P (6))- equilibrium and e-approximate (r)oj( -equiiiiiii A natural question is whether correlation enables efficient computation of e-approximate -(u))-equilibrium when 8 is in the global regime, i.e., 8 = (2v(( d). In this section, we prove both computational hardness and a query complexity lower bound for both notions in the global regime
To prove the lower bound results, we only require a single-player game. The problem of computing an e-approximate ( -qquilibrim becomes: given scalars e, 6, G, L 0 and polynomial-time Turing machine C₁ evaluating a G-Lipschitz and L-smooth function f [0, 1jd - [0, 1] and its gradient v [0, 1] -> R&, we are asked to output distribution that is an e-approximate -eq)euiiii or if such equilibrium does not exist.
Hardness of finding e-approximate Φ (6)-equilibria in the global regime When 8 Vd. which equals to the diameter / of [0, 1] then the problem of finding e-approximate T. ( (0)- equilibrium is equivalent to finding a (s, 6l-ccal minimum of f: assume a is an s-approximate o (6)- f, then there exists r [0, 1] the support of a such that min "E(o.i]'nBaat".0 f(x") Se.
f(x)
Then hardness of finding an s-approximate T (8) (duueiuii follows from hardness of finding (e, -lcal minimum of [DS721]. The following Theorem corollary of Theorem 10.3 and 10.4 in [DSZ21].
28

---

Theorem 12 (Hardness of finding e-approximate $ ( worst case, the following two holds.
eqquilibra in the global regime), In the
Computing an E-approximate ii (f)-euillirrium for a game on X' [0, 1] with G = Vd.
L d 24' = vd is NP-hard.
32(2%/d) value/gradient queries are needed to determine an E-approximate $1 (6)- equilibrium for game on X' [0, 10 with G = (d L= - 1, vd.
Hardness of finding e-approximate ( -0)- in the global regime
Theorem 13 (Hardness of of finding E-approximate Prooj (6) the global regime). In the worst case, the following two holds.
Compuling an E-approximate b -ouuquiii for game on X [0, 1] with G = L = vd
NP-hard.
2d) value/gradient queries needed to determine s-approximate T Proj (6)- equilibrium game on X = [0, with G = Odd L= ((( < 1, 8= vd.
The hardness of computing e-approximate (0) -qquilibrui also implies a lower bound on Proj (é)-rertt in the global regime.
Corollary 2 (Lower bound of ( (re-rrr against non-convex functions), In the worst case, the & Proj (re-ret of any online algorithm is at least (2(2 /d, T) even for loss functions f [0, 1)d [0, 1] with G, L poly(d) and u = vd.
The proofs of Theorem 13 and Corollary can be found in the next two sections.
I.1 Proof of Theorem 13
We will reduce the problem of finding an E-approximate 4 ( equilibrium in smooth games to finding satisfying assignment of boolean function, which is No-complete.
Fact 1. Given only black-box access a boolean formula Φ (0.
{0,1), least (2(2)) queries are needed in order to determine whether 6 admits a satisfying assignment x" such that given,
<(x*)= 1. The term black-box access refers the fact that the clauses of the formula and the only way to determine whether specific boolean assignment is satisfying is by querying the specific binary string. Moreover, the problem of finding satisfying assignment of a general boolean function is NP-hard.
We revisit the construction the hard instance in the proof of [DSZ21, Theorem 10.4] and use its specific structures. Given black-box access to boolean formula o as described in Fact 1, following [DSZ21], l], we construct the function fa) [0, 1]d + [0, 1] as follows:
1. for each corner E V = {0, 1 of the [0, 1] hypercube, we set fa(x) = o(x).
2. for the rest of the points x e [o, 1V we set fz(z) = =. .,(x)) fo( where (x(x) are non-negative coefficients defined in [DSZ21, Definition 8.9].
The function f satisfies the following properties:
1. if à is not satisfiable, then fa(x) 1 for all x E [0, 1] since fa(v) 1 for all Vf has a satisfying assignment v", then fa())) 0.
2. fo is -Lipshiiis and ((d25)-smooth
3. for any point.z & E J0. 1]d the set V(x) == (u e Pa(a:) o 0} has cardinality at most
d while Ever P. 1; any value gradient query of fa can be simulated by + queries on
29

---

In the case there exists a satisfying argument u², then fa(e²) = 0. Define the deviation e so that
eli| if v" [i] and efi] -1 if [1]]] 1. It is clear that ||e|| vd -d=dd 6. By properties of projection on [0, 11{, for any x E [o, we have 11 [z tj = v². Then any e-approximate a must include some x² & with (x in the support, since e < 1. In case there exists an algorithm A that computes an e-approximate @f (e)eeiiii,,r, A must < 1, there exists é E (2²") such that fū) = O. Since [V(x*)| I 1, it takes addition d ++1 1 queries to find 6 with fō(0) By Fact 1 and the fact that we can simulate every value/gradient query of fo by + 1 queries on Φ, A makes least 02 /d) value/gradient queries.
have queried some x² with fo (z*) < 1. Since fa(x*) = Ever(.))
Suppose there exists an algorithm B that outputs an e-approximate ( -qquuiibbrri a in time T(B) for E and å vd. We construct another algorithm C for SAT that terminates in time T(B) poly(d). C: (1) given a boolean formula o, construct fg as described above; (2) run B and get output (3) check the support of a to find E {0, 1 such that fa(v) O; (3) if finds v E {0, 1]dd such that (v) 0, then Φ is satisfiable, otherwise @i is not satisfiable. Since we can evaluate So and f& in poly (d) time and the support of a is smaller than T(B), the algorithm C terminates in time ( poly(d)). The above gives polynomial time reduction from SAT to finding an e-approximate Proj (qquuiiii and proves the NP- hardddddess of the latter problem.
1.2 Proof of Corollary 2
Let Φ [0,1]a + 1) be a boolean formula and define fg 10, 1]4 + 1] the same as that and ((poly(d))-smooot Now we let the adversary pick fo each time. For any T S (2 /d), in case there exists an online learning algorithm with RegProj,s < 1.4 then o : + Σ L is an (g) -)eqquilbrrumm Applying Theorem 13 and the fact that in this case, RegProj.s is non-decreasing with respect to T concludes the proof.
in Theorem 13. We know Sa is ((poly(d)--Lipshhl))
J Removing the D dependence for Φ P -regret
For the regime < Dx which we are more interested in, the lower bound in Theorem is ((G while the upper bound in Theorem 3 is O(GVDDzT), They are not tight especially when Dx >> A natural question is: which of them is the tight bound? We conjecture that the lower bound is tight. In fact, for the special case where the feasible set X' is a box, we have a way to obtain a Dx-independent bound Gdt which is tight when d = 1. Below, we first deseribe the improved strategy in 1-dimension. Then we show how extend it to the d-dimensional box setting.
J.1 One-Dimensional Case
In one-dimension, we assume that X' = |a, b) for some b a 28 (if a S 28, then our original bound in Theorem is already of order GTv We first investigate the case where f'(x) is linear function, i.c., f*x) = g x for some g¹ E G.. The key idea is that we will only select x² from the two intervals [a, + 5] and [6 - 6, b]. and never play z' E (a + 6,6 6). To achieve concatenate these two intervals, and run an algorithm in this region whose diameter is only 25. The key property would like to show is that the regret is preserved in this modified problem.
we
More precisely, given the original feasible set X = [a, b). we create a new feasible set y = (-8, 8] and apply our algorithm GD in this new feasible set. The loss function is kept as f'(x) = g'r. Whenever the algorithm for y outputs y¹ E [-5, 0), we play s' = y' + a + 5 in X: whenever outputs y' (0, 8), we play x' o 5. Below we show that the regret is the same in these two
30

---

problems. Notice that when y I 0, we have for any E [-8,ō],
x¹ 5'-11x 11x [x' 2/ x² max (min (x' - v, b).b)) a)
x² max (x' v.
(x' U y2 -
28 always holds)
yf + 6 max (y² a + v.a)
y' max (y' U, -6)
y' max (min (g² v, 6) -6)
(y' v S always holds)
= y' Ty v)
Similarly, when y" > O, we can follow the same calculation and prove x² Iyll"" e]. Thus, the regret in the two problems:
IIx [x - v] = 3²
g6 (x' 1xx - vl) and Ilyly - vl)
are exactly the same for any v. Finally, observe that the diameter of y is only of order O()). Thus,
the upper bound in Theorem 3 would give us an upper bound of O 0 87) OGGVTT). For convex S we run the algorithm above with g² = Vz'(z²). Then by convexity we have
f'(x') ('( v)) s g' - -1xx - v)) = g (v' Hyyly -l),
so the regret in the modified problem (which O(G5vT)) still serves regret upper bound for the original problem.
J.2 d- d-Dimensional Box Case
d-dimensional box is of the form [01, [a2, b2] x .... [ad. bu]. The box case easy
to deal with because we can decompose the regret into individual components in each dimension. Namely, we have
( v)) (a') (x* -Tx - w))
d Σ of (z) -
where we define X; [a,, b.]. [a,,,,,, g"
(z''"") and use subscript i to indicate the i-th component of vector. The last equality above is guaranteed by the box structure. This decomposition allows as to view the problem as d independent 1 1-dimensional problems.
Now we follow the strategy described in Section J. to deal with individual dimensions bi < 26 then we do not modify X;; otherwise, we shrink X; to be of length 26). Applying the analysis of Theorem each dimension, we get
d
9
(I)
II, j - vil)
|ey| x 26
+ Σ
(ot)² 2
(the diameter in each dimension is now bounded by 26)
21
17
1=1
1=1
8 ا
|ual
00
+ GTTTT
"
dd + GTT
(by Cauchy-Schwarz, E, |ua S Vd Σ, 1013 < <o0))
"
Choosing the optimal 7 = d2
we get the regret upper bound of order 0 (dtGovT).
GVT
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
We state the problem setting and our main contributions in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
32

---

Our results provide efficient uncoupled algorithms to compute e-approximate @ --equilibria in the first-order stationary regime. We leave the computational complexity of finding e-approximate #-equilibria beyond the first-order stationary regime as an open question.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide assumptions and proofs for our theoretical results in the main body and the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [NA]
33

---

Justification:
This paper does not contain experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
This paper does not contain experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /tte://nipppp. cc/ public/ guides Cooeeuuameiii ceionnooiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
httppsscc/pbbbcccc / /nipp. cc/ ptbppp/blcccccc public/quides guti//////////eeee//eo Cooee SrodeSubmis Submi as wsssonooollloooooooo oollcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
34

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: This paper does not contain experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This paper does not contain experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
This paper does not contain experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /neurips. wttpppbbicc/hhhhccuuii.eeoeo
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and the current paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is purely theoretical paper and we do not see any immediate societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
36

---

Answer:
[NA]
Justification:
This is purely theoretical paper
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This is purely theoretical paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper sowerwwii wit thoode. com/ da t tasets t has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This is purely theoretical paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
37

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not contain experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not contain experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "jVdcx3hFUy", "reviewer": "Reviewer_r7Yv", "summary": "The paper studies nonconcave games and $\\Phi$-equilibrium. When $\\Phi$ is finite, the paper showed that there exists an uncoupled learning algorithm that can efficiently find the equilibria. Under certain classes of strategy modification, online gradient descent can also approximate $\\Phi$-equilibria when $\\Phi$ is infinite.", "questions": "I am not an expert in non-concave games and $\\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses", "limitations": "yes", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am not an expert in non-concave games and $\\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses", "weaknesses": "I am not an expert in non-concave games and $\\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C9neiXq362", "reviewer": "Reviewer_GeMu", "summary": "This work studies the $\\Phi$-equlibrium in non-concave games and discuss when the $\\Phi$-equlibrium of a non-concave game can be learned in polynomial time. The theoretical results presented in this paper indicate that if the set $\\Phi$ is finite, then there exists an efficient uncoupled algorithm that converges to the corresponding  $\\Phi$-equilibria. Moreover, this work also considers the case when $\\Phi$ is not finite but consist of local modifications.", "questions": "1. I find some papers showing that for Markov games, the CE or CCE can be learned in polynomial time. For example: *Jin, Chi, et al. \"V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL.\" arXiv preprint arXiv:2110.14555 (2021).* From my understanding, when setting their horizon $H=1$, the Markov game seems to be a classical game. Is there any difference between this setting and yours? \n\n2. Is there any applications of $\\Phi$-equilibrium? For example, when do we need to efficiently solve some  $\\Phi$-equilibrium? \n\n3. The definition of $\\Phi$-equilibrium looks really similar to the Correlated Equilibrium (CE) in your Definition 6. Is CE as the same as $\\Phi$-equilibrium if $\\Phi$ is choosen as the whole space?", "limitations": "This is a purely theoretical work so there is no any negative impact.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance: This paper gives a non-trivial results on the $\\Phi$-equilibrium of non-concave games. It provides a simple condition to tell if $\\Phi$-equilbria can be learned in polynomial iterations. \n\nOriginality: There are no existing literature that have resolved the $\\Phi$-equilibrium problem. So, the result of this paper is new. This paper extends existing framework to consider infinite local strategy modifications, which also leads to new technical novelty. \n\nClarity: This work includes sufficient backgroun introduction and put adequate materials in the appendix for references. I feel easy to understand the main contribution of this work.", "weaknesses": "I believe this is a high quality paper and I give a clear accept. But I do have some confused points. I put them in the next sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive review. Below, we address your questions.\n\nQ: *I found some papers showing that for Markov games, the CE or CCE can be learned in polynomial time. For example Jin, Chi, et al. \"V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL.\" arXiv preprint arXiv:2110.14555 (2021). From my understanding, when setting their horizon H = 1, the Markov game seems to be a classical game. Is there any difference between this setting and yours?*\n\nA: If the horizon $H = 1$, then a general-sum Markov game becomes a general-sum normal-form game where each player’s utility function is **linear** in their own strategy. This is a special case of the non-concave games we consider in this paper since, in a non-concave game, each player’s utility could be **non-concave** in their own strategy.  Additionally, Markov games with $H \\ge 1$ are **special cases** of non-concave games that include additional structure, such as the Markovian property. Since our focus is on the more general class of non-concave games, these specific results do not apply.\n\nQ: *Is there any applications of $\\Phi$-equilibrium? For example, when do we need to efficiently solve some $\\Phi$-equilibrium?*\n\nA: The notion of $\\Phi$-equilibrium is very general. If $\\Phi$ is all constant strategy modifications, the corresponding $\\Phi$-equilibrium coincides with the notion of coarse correlated equilibrium (CCE); If $\\Phi$ is all strategy modifications, then the corresponding $\\Phi$-equilibrium coincides with the notion of correlated equilibrium (CE). CE and CCE are both central and classical notions extensively studied in the literature and used in practice. Generally, a $\\Phi$-equilibrium guarantees that no single agent has an incentive to unilaterally deviate by using strategy modifications from their set $\\Phi_i$. This provides a stability guarantee that we believe will be relevant for many economics and machine learning applications, including game AI and multi-agent reinforcement learning. \n\nQ: *The definition of $\\Phi$-equilibrium looks really similar to the Correlated Equilibrium (CE) in your Definition 6. Is CE as the same as $\\Phi$-equilibrium if $\\Phi$ is chosen as the whole space?*\n\nA: Yes. If we choose $\\Phi$ as all possible strategy modifications, the corresponding $\\Phi$-equilibrium is exactly CE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tzI8OlhPc0", "reviewer": "Reviewer_oGwK", "summary": "This paper describes a concept of $\\Phi$-equilibrium in non-concave games, which is a rarely discussed notion in the recent literature. This notion of equilibrium is defined over a set of strategy modifications, with specific definitions of $\\Phi$, $\\Phi$-equilibrium can recover commonly known notions such as CCEs, though this paper mostly focuses on more limiting choices of $\\Phi$. The paper shows that there exists efficient algorithms to find an $\\epsilon$-approximate $\\Phi$-equilibrium in tractable time for some specific families of set $\\Phi$.", "questions": "No", "limitations": "The authors adequately address the limitations of this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although some assumptions on families of set $\\Phi$ seem restrictive, it is known that without any assumptions, the task of finding e.g. CCE, even approximately and locally, is intractable. This paper provides a solid direction towards better understanding of algorithms in non-convex games. \n\nThis paper is quite self-contained, and section 1 explained the problem formulation such that the reviewer find easy to follow.\n\n\nFor the infinitely large $\\Phi$, the three families offered good insight and can potentially cover many existing strategy modification schemes in application.", "weaknesses": "The structure of this paper is questionable. Section 1 alone takes 4 pages, and the remainder of this paper seems rushed and/or crammed. The authors did address the space constraint of this paper, but I believe the paper can be greatly improved in terms of better readability.\nFor instance, I believe it would be beneficial to address algorithm 1 in the main paper. And more space could be assigned to the finite $\\Phi$ setting, as a warm-up. The reviewer suggests that some paragraphs or even theorems in section 4 could be moved to the appendix.\n\nSome statements throughout the paper appear to be slightly repetitive and reiterates of previous sections, many statements can be shortened and more concise. For example, section 1.1 interweaves between introduction and contributions.\n\nAlthough this paper is quite high-level and technical, it would be reasonable to consider some empirical results to validate and strengthen the theoretical results such as the realistic complexity for the sampling procedure. This can perhaps be added in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and constructive comments on the structure and presentation of the paper!\n\nWe will incorporate your suggestions and improve the presentation in the revised version of the paper. Since one more page is allowed in the camera-ready version, we will assign more space to the finite $\\Phi$ setting and include Algorithms 1 and 2 and relevant discussions in the main paper. We will also adjust the structure of section 4 and make all statements in the main paper more concise. \n\n**Complexity of our algorithms**: The algorithms we study (except for Algorithms 1 and 3) are gradient-based algorithms like gradient descent and optimistic gradient descent, which have efficient practical implementations. Regarding the complexity of the sampling procedure (Algorithm 2), the per-iteration complexity is exactly $\\sqrt{T} \\cdot |\\Phi|$, where we need to sample from a distribution that supports on $|\\Phi|$ points, evaluate the strategy modification function $\\phi$, and repeat the above procedure for $\\sqrt{T}$ steps. \nThe above complexity analysis for Algorithm 2 is tight. There is an equivalent implementation that improves the expected running time by a factor of 2: we could first sample a number N uniformly between $[1, \\sqrt{T}]$ and then only run the sampling procedure for $N$ steps rather than $\\sqrt{T}$ steps, and finally return the last point. This improves the expected per-iteration complexity to  $(\\sqrt{T} \\cdot |\\Phi|) / 2$.\nWe will add the discussion in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xN89kQrR5Z", "reviewer": "Reviewer_14T6", "summary": "The paper presents new results about no-$\\Phi$-regret learning when the utilities are not concave. When $\\Phi$ is either finite or in a precise sense \"local\", it is shown that no-$\\Phi$-regret learning is possible, and hence that the corresponding notions of $\\Phi$-equilibria can be efficiently approximated at rate $\\text{poly}(1/\\epsilon)$.", "questions": "1. Could you elaborate on Corollary 1? It is not obvious to me how that bound follows from Theorem 2. \n1. I find it interesting that the algorithms in this paper are presented as *randomized* algorithm outputting a *single* $x^t$ rather than a *determinisitic* algorithm outputting a *distribution* $\\mu^t \\in \\Delta(\\mathcal X)$ (as done by Peng and Rubinstein [2024], Dagan et al [2024], and Zhang et al [2024]). At least with your techniques, this difference seems to be an unavoidable consequence of the fact that the utilities $u^t$ are nonlinear, and therefore one cannot say $\\mathbb E_{\\phi \\sim p} u^t(\\phi(\\cdot)) = u^t(\\mathbb E_{\\phi \\sim p} \\phi(\\cdot))$. Do I understand this correctly?\n1. If I understood the previous point correctly: in Theorem 5, the nonlinearity of $u^t$ is a nonissue by assumption. So, is it possible to de-randomize that result by instead deterministically outputting the uniform distribution $\\mu^t := \\text{unif} \\\\{ x_1, \\dots, x_K \\\\}$?\n1. Is there an efficient algorithm for $\\Phi^\\mathcal{X}_\\text{Beam}(\\delta)$-regret minimization?\n1. Do the results of this paper have any novel implications for interesting special cases, such as nonconvex-concave zero-sum games, or simply the classical setting of concave no-regret learning? For example the results of Section 4 seem to have interesting (new?) implications for local (mixed) Nash equilibria in nonconvex-nonconcave zero-sum games. \n\nPapers cited here but not in the submission:\n* Dagan, Y., Daskalakis, C., Fishelson, M., & Golowich, N. (*STOC* 2024). From External to Swap Regret 2.0: An Efficient Reduction for Large Action Spaces.\n* Peng, B., & Rubinstein, A. (*STOC* 2024). Fast swap regret minimization and applications to approximate correlated equilibria.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This presents some interesting and novel results in a regime (non-concave games) where positive results seem to be relatively difficult to come by. The paper was also very well written and easy to read. I vote to accept and only have some minor comments.", "weaknesses": "It was useful to me to keep in mind that the regime $\\delta = O(\\epsilon)$ (where $O$ hides game/instance-dependent constants) is trivial, because no $\\delta$-local change can affect a Lipschitz utility function by more than $O(\\delta)$. It would be nice to explicitly state this somewhere.\n\nIn Table 1, it would be nice to distinguish the contributions of this paper from past contributions. Perhaps for each cell, include either the past citation or the theorem number in the current paper.\n\nMinor things and typos (not affecting score):\n* Table 1: effifient -> efficient\n* Missing close-parens: Algorithm 1, line 5; two in the botton half of page 18\n* Algorithm 2, line 3: form -> from", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review and constructive comments! We will add a note on the trivial regime where $\\delta = O(\\epsilon)$. We will update Table 1 to distinguish our contributions better. Below, we address your questions.\n\nQ: *Could you elaborate on Corollary 1? It is not obvious to me how that bound follows from Theorem 2.*\n\nA: We will add the proof of Corollary 1 in the revised version of the paper. In Corollary 1, we consider the case where $\\Phi$ is the class of $M$-Lipschitz functions from $[0,1]^d$ to $[0,1]^d$ and the utility is $G$-Lipschitz. The $\\alpha$-covering number of $\\Phi$ satisfies $\\log N(\\alpha) = \\Theta((1/\\alpha)^d)$ (this fact is also used in [1, Example 23]). Thus, we can run Algorithm 1 over the finite $\\alpha$-cover of $\\Phi$. This leads to a regret bound of $\\sqrt{T(M/\\alpha)^d)} + \\alpha G T$, where the second term is due to the covering error. By choosing $\\alpha = M T^{-1/(d+2)}$, we get regret bound $O(c \\cdot T^{(d+1)/(d+2)})$ where c is some constant that only depends on $G$ and $M$.  \n \n\t[1] Stoltz, Gilles, and Gábor Lugosi. \"Learning correlated equilibria in games with compact sets of strategies.\" Games and Economic Behavior, 2007\n\nQ: *I find it interesting that the algorithms in this paper are presented as randomized algorithm outputting a single $x^t$ rather than a deterministic algorithm outputting a distribution $\\mu_t \\in \\Delta(\\mathcal{X})$ (as done by Peng and Rubinstein [2024], Dagan et al [2024], and Zhang et al [2024]). At least with your techniques, this difference seems to be an unavoidable consequence of the fact that the utilities ut are nonlinear, and therefore one cannot say $\\mathbb{E}_{\\phi \\sim p}[u^t(\\phi(\\cdot))] = u^t( \\mathbb{E}\\_{\\phi \\sim p} \\phi(\\cdot))$. Do I understand this correctly?*\n\nA: Two issues prevent us from outputting a distribution. The first one, as you pointed out, is the nonlinearity of the utility function. The second is that the distribution we constructed has an exponentially large support of $|\\Phi|^\\sqrt{T}$. To achieve an $\\varepsilon$-approximate $\\Phi$-equilibrium, we need $T$ to be $\\textnormal{poly}(1/\\varepsilon)$, so outputting the entire distribution requires time exponential in $1/\\varepsilon$. Instead, using a sampling procedure allows us to significantly improve the dependence on $1/\\varepsilon$ and obtain an algorithm whose running time is only polynomial in $1/\\varepsilon$.\n\nQ: *If I understood the previous point correctly: in Theorem 5, the nonlinearity of ut  is a nonissue by assumption. So, is it possible to de-randomize that result by instead deterministically outputting the uniform distribution $\\mu^t = \\\\{x_1, \\ldots, x_K\\\\}$?*\n\nA: We can derandomize this result by directly outputting the distribution $\\mu^t$. In this case, we allow the algorithm to output a mixed strategy and consider regret over the expected utility in each iteration. Thank you for this insightful observation. \n\nQ: *Is there an efficient algorithm for $\\Phi^{\\mathcal{X}}\\_{\\textnormal{Beam}}(\\delta)$-regret minimization?*\n\nA: We currently don’t have an efficient algorithm and believe this is an interesting open question. We introduce $\\Phi^{\\mathcal{X}}\\_{\\textnormal{Beam}}(\\delta)$-regret to show that even for simple local strategy modification sets $\\Phi(\\delta)$, the landscape of efficient local $\\Phi(\\delta)$-regret minimization is already quite rich, and many basic and interesting questions remain open.\n\nQ: *Do the results of this paper have any novel implications for interesting special cases, such as nonconvex-concave zero-sum games, or simply the classical setting of concave no-regret learning? For example the results of Section 4 seem to have interesting (new?) implications for local (mixed) Nash equilibria in nonconvex-nonconcave zero-sum games*.\n\nA:  To the best of our knowledge, the notion of $\\Phi_{\\textnormal{proj}}$-regret is new, and its efficient minimization is unknown even in the classical setting of no-regret learning with concave utilities. We think our upper and lower bound results for the $\\Phi_{\\textnormal{proj}}$-regret are, therefore, interesting even in this setting. We do not see any immediate implications for local mixed Nash equilibria in nonconvex-nonconcave zero-sum games, but this is a very interesting open question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper focuses on no-$\Phi$-regret learning when the utilities of the agents are not concave. It is shown that when $\Phi$ is either finite or in a precise sense "local" (local deviations are allowed), no-$\Phi$-regret learning is possible and therefore the corresponding notions of $\Phi$-equilibria can be efficiently approximated in time poly( $1/\epsilon$ ). On the negative side, the authors did not discuss the suboptimality aspects of the equilibrium notion considered (most notably, that even a global utility minimizer can be a $\Phi$-equilibrium in the range of $\epsilon$-$\delta$ paramters considered by the authors) but, nonetheless, the reviewers were positive about this paper and the AC recommends acceptance.
Decision: Accept (poster)