Title: On Tractable $\Phi$-Equilibria in Non-Concave Games
Venue: NeurIPS 2024
Accepted: yes
Keywords: Non-Concave Games

$\Phi$-Equilibrium

$\Phi$-Regret Minimization

Learning in Games
Abstract: While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to a coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when utilities are non-concave -- a common scenario in machine learning applications involving strategies parameterized by deep neural networks, or when agents' utilities are computed by neural networks, or both. Non-concave games introduce significant game-theoretic and optimization challenges: (i) Nash equilibria may not exist; (ii) local Nash equilibria, though they exist, are intractable; and (iii) mixed Nash, correlated, and coarse correlated equilibria generally have infinite support and are intractable. To sidestep these challenges, we revisit the classical solution concept of $\Phi$-equilibria introduced by Greenwald and Jafari [GJ03], which is guaranteed to exist for an arbitrary set of strategy modifications $\Phi$ even in non-concave games [SL07]. However, the tractability of $\Phi$-equilibria in such games remains elusive. In this paper, we initiate the study of tractable $\Phi$-equilibria in non-concave games and examine several natural families of strategy modifications. We show that when $\Phi$ is finite, there exists an efficient uncoupled learning algorithm that approximates the corresponding $\Phi$-equilibria. Additionally, we explore cases where $\Phi$ is infinite but consists of local modifications, showing that Online Gradient Descent can efficiently approximate $\Phi$-equilibria in non-trivial regimes.

On Tractable Φ -Equilibria in Non-Concave Games

Abstract

While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to a coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when utilities are non-concave - a common scenario in machine learning applications involving strategies parameterized by deep neural networks, or when agents' utilities are computed by neural networks, or both. Non-concave games introduce significant game-theoretic and optimization challenges: (i) Nash equilibria may not exist; (ii) local Nash equilibria, though they exist, are intractable; and (iii) mixed Nash, correlated, and coarse correlated equilibria generally have infinite support and are intractable. To sidestep these challenges, we revisit the classical solution concept of Φ -equilibria introduced by Greenwald and Jafari [GJ03], which is guaranteed to exist for an arbitrary set of strategy modifications Φ even in nonconcave games [SL07]. However, the tractability of Φ -equilibria in such games remains elusive. In this paper, we initiate the study of tractable Φ -equilibria in nonconcave games and examine several natural families of strategy modifications. We show that when Φ is finite, there exists an efficient uncoupled learning algorithm that approximates the corresponding Φ -equilibria. Additionally, we explore cases where Φ is infinite but consists of local modifications, showing that Online Gradient Descent can efficiently approximate Φ -equilibria in non-trivial regimes.

1 Introduction

Von Neumann's celebrated minimax theorem establishes the existence of Nash equilibrium in all two-player zero-sum games where the players' utilities are continuous as well as concave in their own strategy [Neu28]. 1 This assumption that players' utilities are concave, or quasi-concave, in their own strategies has been a cornerstone for the development of equilibrium theory in Economics, Game Theory, and a host of other theoretical and applied fields that make use of equilibrium concepts. In particular, (quasi-)concavity is key for showing the existence of many types of equilibrium, from generalizations of min-max equilibrium [Fan53; Sio58] to competitive equilibrium in exchange economies [AD54; McK54], mixed Nash equilibrium in finite normal-form games [Nas50], and, more generally, Nash equilibrium in (quasi-)concave games [Deb52; Ros65].

Existence

Complexity

Nash equilibrium

✗

✓

(Coarse) Correlated equilibrium

Global stability

✓

Strict local Nash equilibrium

Local stability

✗

Second-order local Nash equilibrium

Second-order stability

✗

NP-hard

[MK87; AZ22]

First-order stability

✓

PPAD-hard [DSZ21]

Φ

-equilibrium (finite

|

Φ

|

)

Stability against

finite deviations

✓

Efficient

ε

-approximation

for any

ε >

0

(Theorem 2)

Conv(Φ(

δ

))

-equilibrium (finite

|

Φ(

δ

)

|

)

✓

Efficient

ε

-approximation

for

ε

= Ω(

δ

2

)

(Theorem 4)

Φ

proj

(

δ

)

-equilibrium

✓

Efficient

ε

-approximation

via GD/OG for

ε

= Ω(

δ

2

)

(Theorem 3,9)

Φ

Int

(

δ

)

-equilibrium

First-order stability

when

ε

= Ω(

δ

2

)

✓

Efficient

ε

-approximation

via no-regret learning for

ε

= Ω(

δ

2

)

(Theorem 5)

Not only are equilibria guaranteed to exist in concave games, but it is also well-established-thanks to a long line of work at the interface of game theory, learning and optimization whose origins can be traced to Dantzig's work on linear programming [Geo63], Brown and Robinson's work on fictitious play [Bro51; Rob51], Blackwell's approachability theorem [Bla56] and Hannan's consistency theory [Han57]-that several solution concepts are efficiently computable both centrally and via decentralized learning dynamics. For instance, it is well-known that the learning dynamics produced when the players of a game iteratively update their strategies using no-regret learning algorithms, such as online gradient descent, is guaranteed to converge to Nash equilibrium in twoplayer zero-sum concave games, and to coarse correlated equilibrium in multi-player general-sum concave games [CL06]. The existence of such simple decentralized dynamics further justifies using these solution concepts to predict the outcome of real-life multi-agent interactions where agents deploy strategies, obtain feedback, and use that feedback to update their strategies.

While (quasi-)concave utilities have been instrumental in the development of equilibrium theory, as described above, they are also too restrictive an assumption. Several modern applications and outstanding challenges in Machine Learning, from training Generative Adversarial Networks (GANs) to Multi-Agent Reinforcement Learning (MARL) as well as generic multi-agent Deep Learning settings where the agents' strategies are parameterized by deep neural networks or their utilities are computed by deep neural networks, or both, give rise to games where the agents' utilities are non-concave in their own strategies. We call these games non-concave , following [Das22].

Unfortunately, classical equilibrium theory quickly hits a wall in non-concave games. First, Nash equilibria are no longer guaranteed to exist. Second, while mixed Nash, correlated and coarse correlated equilibria do exist-under convexity and compactness of the strategy sets [Gli52], which we have been assuming all along in our discussion so far, they have infinite support, in general [Kar14]. Finally, they are computationally intractable; so, a fortiori, they are also intractable to attain via decentralized learning dynamics.

In view of the importance of non-concave games in emerging ML applications and the afore-described state-of-affairs, our investigation is motivated by the following broad and largely open question:

Question from [Das22]: Is there a theory of non-concave games? What solution concepts are meaningful, universal, and tractable?

1.1 Contributions

We study Daskalakis' question through the lens of the classical solution concept of Φ -equilibria introduced by Greenwald and Jafari [GJ03]. This concept is guaranteed to exist for virtually any set of strategy modifications Φ , even in non-concave games, as demonstrated by Stoltz and Lugosi [SL07]. 2 However, the tractability of Φ -equilibria in such games remains elusive. In this paper, we initiate the study of tractable Φ -equilibria in non-concave games and examine several natural families of strategy modifications.

Φ

!"#$

(𝛿)

-Eq.

Φ

%&'

(𝛿)

-Eq.

CCE

Local NE

NE

MNE

CE

Φ

()&)'*

-Eq.

Conv Φ 𝛿

-Eq.

Global

Deviation

Local

Deviation

Φ -Equilibrium. The concept of Φ -equilibrium generalizes (coarse) correlated equilibrium. A Φ -equilibrium is a joint distribution over Π n i =1 X i , the Cartesian product of all players' strategy sets, and is defined in terms of a set, Φ X i , of strategy modifications , for each player i . The set Φ X i contains functions mapping X i to itself. A joint distribution over strategy profiles qualifies as a Φ = Π n i =1 Φ X i -equilibrium if no player i can increase their expected utility by using any strategy modification function, ϕ i ∈ Φ X i , on the strategy sampled from the joint distribution. The larger the set Φ , the stronger the incentive guarantee offered by the Φ -equilibrium. For example, if Φ X i contains all constant functions, the corresponding Φ -equilibrium coincides with the notion of coarse correlated equilibrium. Throughout the paper, we also consider ε -approximate Φ -equilibria, where no player can gain more than ε by deviating using any function from Φ X i . We study several families of Φ and illustrate their relationships in Figure 1.

Finite Set of Global Deviations. The first case we consider is when each player i 's set of strategy modifications, Φ X i , contains a finite number of arbitrary functions mapping X i to itself. As shown in [GJ03], if there exists an online learning algorithm where each player i is guaranteed to have sublinear Φ X i -regret, the empirical distribution of joint strategies played converges to a Φ = Π n i =1 Φ X i -equilibrium. Gordon, Greenwald, and Marks [GGM08] consider Φ -regret minimization but for concave reward functions, and their results, therefore, do not apply to non-concave games. Stoltz and Lugosi [SL07] provide an algorithm that achieves no Φ X i -regret in non-concave games; however, their algorithm requires a fixed-point computation per step, making it computationally inefficient. 3 Our first contribution is to provide an efficient randomized algorithm that achieves no Φ X i -regret for each player i with high probability.

Contribution 1: Let X be a strategy set (not necessarily compact or convex), and Ψ an arbitrary finite set of strategy modification functions for X . We design a randomized online learning algorithm that achieves O ( √ T log | Ψ | ) Ψ -regret, with high probability, for arbitrary bounded reward functions on X (Theorem 2). The algorithm operates in time √ T | Ψ | per iteration. If every player in a non-concave game adopts this algorithm, the empirical distribution of strategy profiles played forms an ε -approximate Φ = Π n i =1 Φ X i -equilibrium, with high probability, for any ε > 0 , after poly ( 1 ε , log ( max i | Φ X i | ) , log n ) iterations.

If players have infinitely many global strategy modifications, we can extend Algorithm 1 by discretizing the set of strategy modifications under mild assumptions, such as the modifications being Lipschitz (Corollary 1). The empirical distribution of the strategy profiles still converges to the corresponding Φ -equilibrium, but at a much slower rate of O ( T -1 d +2 ) , where d is the dimension of the set of strategies. Additionally, the algorithm requires exponential time in the dimension per iteration, making it inefficient. This inefficiency is unavoidable, as the problem remains intractable even when Φ contains only constant functions.

To address the limitations associated with infinitely large global strategy modifications, a natural approach is to focus on local deviations instead. The corresponding Φ -equilibrium will guarantee local stability. The study of local equilibrium concepts in non-concave games has received significant attention in recent years-see e.g., [RBS16; HSZ17; DP18; JNJ20; DSZ21]. However, these solution concepts either are not guaranteed to exist, are restricted to sequential two-player zero-sum games [MV21], only establish local convergence guarantees for learning dynamics-see e.g., [DP18; WZB20; FCR20], only establish asymptotic convergence guarantees-see e.g., [Das+23], or involve non-standard solution concepts where local stability is not with respect to a distribution over strategy profiles [HSZ17].

We study the tractability of Φ -equilibrium with infinitely large Φ sets that consist solely of local strategy modifications. These local solution concepts are guaranteed to exist in general multi-player non-concave games. Specifically, we focus on the following three families of natural deviations.

-Projection based Local Deviations: Each player i 's set of strategy modifications, denoted by Φ X i Proj ( δ ) , contains all deviations that attempt a small step from their input in a fixed direction and project if necessary, namely are of the form ϕ v ( x ) = Π X i [ x -v ] , where ∥ v ∥ ≤ δ and Π X i stands for the ℓ 2 -projection onto X i .

-Convex Combination of Finitely Many Local Deviations: Each player i 's set of strategy modifications, denoted by Conv(Φ X i ( δ )) , contains all deviations that can be represented as a convex combination of a finite set of δ -local strategy modifications, i.e., ∥ ϕ ( x ) -x ∥ ≤ δ for all ϕ ∈ Φ X i ( δ ) .

-Interpolation based Local Deviations: each player i 's set of local strategy modifications, denoted by Φ X i Int ( δ ) , that contains all deviations that interpolate between the input strategy and another strategy in X i . Formally, each element ϕ λ,x ′ ( x ) of Φ X i Int ( δ ) can be represented as (1 -λ ) x + λx ′ for some x ′ ∈ X i and λ ≤ δ/D X i ( D X i is the diameter of X i ).

For our three families of local strategy modifications, we explore the tractability of Φ -equilibrium within a regime we term the first-order stationary regime , where ε = Ω( δ 2 ) 4 , with δ representing the maximum deviation allowed for a player. An ε -approximate Φ -equilibrium in this regime ensures firstorder stability. This regime is particularly interesting for two reasons: (i) Daskalakis, Skoulakis, and Zampetakis [DSZ21] have demonstrated that computing an ε -approximate δ -local Nash equilibrium in this regime is intractable. 5 This poses an intriguing question: can correlating the players' strategies, as in a Φ -equilibrium, potentially make the problem tractable? (ii) Extending our algorithm, initially designed for finite sets of strategy modifications, to these three sets of local deviations results in inefficiency; specifically, the running time becomes exponential in one of the problem's natural parameters. Designing efficient algorithms for this regime thus presents challenges. Despite these, we show the following:

Contribution 2: For any δ > 0 , for each of the three families of infinite δ -local strategy modifications mentioned above, there exists an efficient uncoupled learning algorithm that converges to an ε -approximate Φ -equilibrium of the non-concave game in the first-order stationary regime, i.e., ε = Ω( δ 2 ) .

We present our results for the projection-based local deviation in Theorem 3 and Theorem 9. Our result for the convex combination of local deviations can be found in Theorem 4. Theorem 5 contains our result for the interpolation-based local deviations. Similar to the finite case, our algorithms build on the connection between Φ -regret minimization and Φ -equilibrium. Given that our strategy modifications are non-standard, it is a priori unclear how to minimize the corresponding Φ -regret. For instance, to our knowledge, no algorithm is known to minimize Φ X Proj ( δ ) -regret even when the reward functions are concave, and provably Φ X Proj ( δ ) -regret is incomparable to external regret (Examples 3 and 4). However, via a novel analysis, we show that Online Gradient Descent (GD) and Optimistic Gradient (OG) achieve a near-optimal Φ X Proj ( δ ) -regret guarantee (Theorem 3 and Theorem 8). Our results provide efficient uncoupled algorithms to compute ε -approximate Φ( δ ) -equilibria in the first-order stationary regime ε = Ω( δ 2 ) .

Further related work is discussed in Appendix A.

2 Preliminaries

A ball of radius r > 0 centered at x ∈ R d is denoted by B d ( x, r ) := { x ′ ∈ ❘ d : ∥ x -x ′ ∥ ≤ r } . We use ∥ · ∥ for ℓ 2 norm throughout. We also write B d ( δ ) for a ball centered at the origin with radius δ . For a ∈ ❘ , we use [ a ] + to denote max { 0 , a } . We denote D X as the diameter of a set X .

Continuous / Smooth Games. An n -player continuous game has a set of n players [ n ] := { 1 , 2 , . . . , n } . Each player i ∈ [ n ] has a nonempty convex and compact strategy set X i ⊆ ❘ d i . For a joint strategy profile x = ( x i , x -i ) ∈ ∏ n j =1 X j , the reward of player i is determined by a utility function u i : ∏ n j =1 X j → [0 , 1] . We denote by d = ∑ n i =1 d i the dimensionality of the game and assume max i ∈ [ n ] { D X i } ≤ D . A smooth game is a continuous game whose utility functions further satisfy the following assumption.

Assumption 1 (Smooth Games) . The utility function u i ( x i , x -i ) for any player i ∈ [ n ] is differentiable and satisfies

( G -Lipschitzness) ∥∇ x i u i ( x ) ∥ ≤ G for all i and x ∈ ∏ n j =1 X j ;

̸

( L -smoothness) there exists L i > 0 such that ∥∇ x i u i ( x i , x -i ) -∇ x i u i ( x ′ i , x -i ) ∥ ≤ L i ∥ x i -x ′ i ∥ for all x i , x ′ i ∈ X i and x -i ∈ Π j = i X j . We denote L = max i L i as the smoothness of the game.

Crucially, we make no assumption on the concavity of u i ( x i , x -i ) .

Φ -equilibrium and Φ -regret. Below we formally introduce the concept of Φ -equilibrium and its relationship with online learning and Φ -regret minimization.

Definition 1 ( Φ -equilibrium [GJ03; SL07]) . In a continuous game, a distribution σ over joint strategy profiles Π n i =1 X i is an ε -approximate Φ -equilibrium for some ε ≥ 0 and a profile of strategy modification sets Φ = Π n i =1 Φ i if and only if for all player i ∈ [ n ] , max ϕ ∈ Φ i E x ∼ σ [ u i ( ϕ ( x i ) , x -i )] ≤ E x ∼ σ [ u i ( x )] + ε. When ε = 0 , we call σ a Φ -equilibrium.

We consider the standard online learning setting: at each day t ∈ [ T ] , the learner chooses an action x t from a nonempty convex compact set X ⊆ ❘ m and the adversary chooses a possibly non-convex loss function f t : X → ❘ , then the learner suffers a loss f t ( x t ) and receives feedback. In this paper, we focus on two feedback models: (1) the player receives an oracle for f t ( · ) ; (2) the player receives only the gradient ∇ f t ( x t ) . The classic goal of an online learning algorithm is to minimize the external regret defined as Reg T := max x ∈X ∑ T t =1 ( f t ( x t ) -f t ( x )) . An algorithm is called no-regret if its external regret is sublinear in T . The notion of Φ -regret generalizes external regret by allowing more general strategy modifications.

Definition 2 ( Φ -regret) . Let Φ be a set of strategy modification functions { ϕ : X → X} . For T ≥ 1 , the Φ -regret of an online learning algorithm is Reg T Φ := max ϕ ∈ Φ ∑ T t =1 ( f t ( x t ) -f t ( ϕ ( x t ))) . An algorithm is called no Φ -regret if its Φ -regret is sublinear in T .

Many classic notions of regret can be interpreted as Φ -regret. For example, the external regret is Φ ext -regret where Φ ext contains all constant strategy modifications ϕ x ∗ ( x ) = x ∗ for all x ∗ ∈ X . The swap regret on simplex ∆ m is Φ swap -regret where Φ swap contains all linear transformations ϕ : ∆ m → ∆ m . A fundamental result for learning in games is that noΦ -regret learning dynamics in games converge to an approximate Φ -equilibrium [GJ03].

Theorem 1 ([GJ03]) . If each player i 's Φ i -regret is upper bounded by Reg T Φ i , then their empirical distribution of strategy profiles played is an (max i ∈ [ n ] Reg T Φ i /T ) -approximate Φ -equilibrium.

3 Tractable Φ -Equilibrium for Finite Φ via Sampling

In this section, we revisit the problem of computing and learning an Φ -equilibrium in non-concave games when each player's set of strategy modifications Φ X i is finite.

The pioneering work of Stoltz and Lugosi [SL07] gives a noΦ -regret algorithm for this case where each player chooses a distribution over strategies in each round. This result also implies convergence to Φ -equilibrium. However, the algorithm by Stoltz and Lugosi [SL07] is not computationally efficient. In each iteration, their algorithm requires computing a distribution that is stationary under a

transformation that can be represented as a mixture of the modifications in Φ . The existence of such a stationary distribution is guaranteed by the Schauder-Cauty fixed-point theorem [Cau01], but the distribution might require exponential support and be intractable to find.

Our main result in this section is an efficient Φ -regret minimization algorithm (Algorithm 1) that circumvents the step of the exact computation of a stationary distribution. Consequently, our algorithm also ensures efficient convergence to a Φ -equilibrium when adopted by all players.

Algorithm 1: Φ -regret minimization for non-concave reward via sampling

Theorem 2. Let X be a strategy set (not necessarily compact or convex), Φ be an arbitrary finite set of strategy modifications over X , and u 1 ( · ) , . . . , u T ( · ) be an arbitrary sequence of possibly non-concave reward functions from X to [0 , 1] . If we instantiate Algorithm 1 with R Φ being the Hedge algorithm over Φ and h = √ T , the algorithm guarantees that, with probability at least 1 -β , it produces a sequence of strategies x 1 , . . . , x T with Φ -regret at most max ϕ ∈ Φ ∑ T t =1 u t ( ϕ ( x t )) -∑ T t =1 u t ( x t ) ≤ 8 √ T (log | Φ | +log(1 /β )) . Moreover, the algorithm runs in time O ( √ T | Φ | ) per iteration.

If all players in a non-concave continuous game employ Algorithm 1, then with probability at least 1 -β , for any ε > 0 , the empirical distribution of strategy profiles played forms an ε -approximate Φ = Π n i =1 Φ X i -equilibrium, after poly ( 1 ε , log ( max i | Φ X i | ) , log n β ) iterations.

High-level ideas. We adopt the framework in [SL07]. The framework contains two steps in each iteration t : (1) the learner runs a no-external-regret algorithm over Φ which outputs p t ∈ ∆(Φ) in each iteration t ; (2) the learner chooses a stationary distribution µ t = ∑ ϕ ∈ Φ p t ϕ ( µ t ) , where we slightly abuse notation to use ϕ ( µ t ) to denote the image measure of µ by ϕ . However, how to compute the stationary distribution µ t efficiently is unclear. We essentially provide a computationally efficient way to carry out step (2) without computing this stationary distribution.

We first construct an ε -approximate stationary distribution by recursively applying strategy modifications from Φ . The constructed distribution can be viewed as a tree. Our construction is inspired by the recent work of Zhang, Anagnostides, Farina, and Sandholm [Zha+24] for concave games. The main difference here is that for non-concave games, the distribution needs to be approximately stationary with respect to a mixture of strategy modifications rather than a single one as in concave games. Consequently, this leads to an approximate stationary distribution with prohibitively high support size ( | Φ | ) √ T , as opposed to √ T in [Zha+24] for concave games.

Despite the exponentially large support size of the distribution, we utilize its tree structure to design a simple and efficient sampling procedure that runs in time √ T . Equipped with such a sampling

Φ

procedure, we provide an efficient randomized algorithm that generates a sequence of strategies so that, with high probability, the Φ -regret for this sequence of strategies is at most O ( √ T log | Φ | ) .

We defer the full proof of Theorem 2 to Section 3.1. An extension of Theorem 2 to infinite Φ holds when the rewards { u t } t ∈ [ T ] are G -Lipschitz and Φ admits an α -cover with size N ( α ) . In particular, when Φ is the set of all M -Lipschitz functions over [0 , 1] d , Φ admits an α -cover with log N ( α ) of the order (1 /α ) d [SL07]. In this case, we have

Corollary 1. There is a randomized algorithm such that, with probability at least 1 -β , the Φ -regret is bounded by c · T d +1 d +2 · log(1 /β ) , where c only depends on G and M . The algorithm runs in time poly( T, N ( T -1 / ( d +2) )) .

3.1 Proof of Theorem 2

For a distribution µ ∈ ∆( X ) over strategy space X , we slightly abuse notation and define its expected utility as u t ( µ ) := E x ∼ µ [ u t ( x )] ∈ [0 , 1] . We define ϕ ( µ ) the image of µ under transformation ϕ . In each iteration t , the learner chooses their strategy x t ∈ X according to the distribution µ t . For a sequence of strategies { x t } t ∈ [ T ] , the Φ -regret is Reg T Φ := max ϕ ∈ Φ { ∑ T t =1 ( u t ( ϕ ( x t )) -u t ( x t )) } . Algorithm 1 uses an external regret minimization algorithm R Φ over Φ which outputs a distribution p t ∈ ∆(Φ) . We can then decompose the Φ -regret into two parts.

I: Bounding the external regret over Φ . The external regret over Φ can be bounded directly. This is equivalent to an online expert problem: in each iteration t , the external regret minimizer R Φ chooses p t ∈ ∆(Φ) and the adversary then determines the utility of each expert ϕ ∈ Φ as u t ( ϕ ( x t )) . We choose the external regret minimizer R Φ to be the Hedge algorithm [FS99]. Then we have max ϕ ∈ Φ { ∑ T t =1 u t ( ϕ ( x t )) -E ϕ ′ ∼ p t [ u t ( ϕ ′ ( x t ))] } ≤ 2 √ T log | Φ | (Theorem 6), where we use the fact that the utility function u t is bounded in [0 , 1] .

II: Bounding error due to sampling from an approximate stationary distribution. We first define a distribution µ t over X using a complete | Φ | -ary tree with depth h . The root of this tree is an arbitrary strategy x root ∈ X . Each internal node x has exactly | Φ | children, denoted as { ϕ ( x ) } ϕ ∈ Φ . The distribution µ t is supported on the nodes of this tree. Next, we define the probability for each node under the distribution µ t . The root node x root receives probability 1 h . The probability of other nodes is defined in a recursive manner. For every node x = ϕ ( x p ) where x p is its parent, x receives probability Pr µ t [ x ] = Pr µ t [ x p ] · p t ( ϕ ) . It is then clear that the total probability of the children of a node x p is exactly Pr µ t [ x is x p 's child ] = ∑ ϕ Pr[ x p ] · p t ( ϕ ) = Pr[ x p ] . Denote the set of nodes in depth k as N k . We have Pr µ t [ x ∈ N k ] = 1 h for every depth 1 ≤ k ≤ h . Thus the distribution µ t supports on | Φ | h -1 | Φ |-1 points and is well-defined. By the construction above, we know x t output by Algorithm 2 is a sample from µ t .

Now we show that the approximation error of µ t is bounded by O ( 1 h ) . We can evaluate the approximation error of µ t :

We recall that for a node x = ϕ ( x p ) with x p being its parent, we have Pr µ t [ x ] = Pr µ t [ x p ] · p t ( ϕ ) . Thus for any 1 ≤ k ≤ h -1 , we have

Using the above equality, we get

where in the last inequality we use the fact that ∑ x ∈ N k Pr µ t [ x ] = 1 h for all 1 ≤ k ≤ h and the utility function u t is bounded in [0 , 1] . Therefore, for x t ∼ µ t , the sequence of random variables { ∑ τ t =1 ( E ϕ ∼ p t [ u t ( ϕ ( x t ))] -u t ( x t ) -1 h ) } τ ≥ 1 is a super-martingale. Thanks to the boundedness of the utility function, we can apply the Hoeffding-Azuma Inequality and get for any ε > 0 .

Combining the bounds for I and II with ε = √ 8 T log(1 /β ) and h = √ T , we have, with probability 1 -β , that Reg T Φ ≤ 2 √ T log | Φ | + T h + √ 8 T log(1 /β ) ≤ 8 √ T (log | Φ | +log(1 /β )) .

Convergence to Φ -equilibrium. If all players in a non-concave continuous game employ Algorithm 1, then we know for each player i , with probability 1 -β n , its Φ X i -regret is upper bounded by 8 √ T (log | Φ X i | +log( n/β )) . By a union bound over all n players, we get with probability 1 -β , every player i 's Φ X i -regret is upper bounded by 8 √ T (log | Φ X i | +log( n/β )) . Now by Theorem 1, we know the empirical distribution of strategy profiles played forms an ε -approximate Φ = Π n i =1 Φ X i -equilibrium, as long as T ≥ 64(log(max i | Φ X i | ) + log( n/β )) /ε 2 iterations.

4 Approximate Φ -Equilibria under Infinite Local Strategy Modifications

This section studies Φ -equilibrium when | Φ | is infinite. It is, in general, computationally hard to compute a Φ -equilibrium even if Φ contains all constant deviations. Instead, we focus on Φ that consists solely of local strategy modifications. We introduce several natural classes of local strategy modifications and provide efficient online learning algorithms that converge to ε -approximate Φ -equilibrium in the first-order stationary regime where ε = Ω( δ 2 L ) . These approximate Φ -equilibria guarantee first-order stability.

Definition 3 ( δ -local strategy modification) . For each agent i , we call a set of strategy modifications Φ X i δ -local if for all x ∈ X i and ϕ i ∈ Φ X i , ∥ ϕ i ( x ) -x ∥ ≤ δ . We use notation Φ X i ( δ ) to denote a δ -local strategy modification set for agent i . We also use Φ( δ ) = Π n i =1 Φ X i ( δ ) to denote a profile of δ -local strategy modification sets.

Below we present a useful reduction from computing an ε -approximate Φ( δ ) -equilibrium in nonconcave smooth games to Φ X i ( δ ) -regret minimization against convex losses for any ε ≥ δ 2 L 2 . The key observation here is that the L -smoothness of the utility function permits, within a δ -neighborhood, a δ 2 L 2 -approximation using a linear function. We defer the proof to Appendix D.

Lemma 1 (No Φ( δ ) -Regret for Convex Losses to Approximate Φ( δ ) -Equilibrium in Non-Concave Games) . For any T ≥ 1 and δ > 0 , let A be an algorithm that guarantees to achieve no more than Reg T Φ X i ( δ ) Φ X i ( δ ) -regret for convex loss functions for each agent i ∈ [ n ] . Then

The Φ X i ( δ ) -regret of A for non-convex and L -smooth losses is at most Reg T Φ X i ( δ ) + δ 2 LT 2 , ∀ i .

When every agent employs A in a non-concave L -smooth game, their empirical distribution of the joint strategies played converges to a ( max i ∈ [ n ] { Reg T Φ X i ( δ ) } / T + δ 2 L 2 ) -approximate Φ( δ ) -equilibrium.

4.1 Projection-Based Local Strategy Modifications

In this section, we study a set of local strategy modifications based on projection. Specifically, the set Φ X Proj ( δ ) encompasses all deviations that essentially add a fixed displacement vector v to the input strategy and project back to the feasible set: Φ X Proj ( δ ) := { ϕ Proj ,v ( x ) = Π X [ x -v ] : v ∈ B d ( δ ) } . It is clear that ∥ ϕ v ( x ) -x ∥ ≤ ∥ v ∥ ≤ δ . The induced Φ X Proj ( δ ) -regret is Reg T Proj ,δ := max v ∈ B d ( δ ) ∑ T t =1 ( f t ( x t ) -f t (Π X [ x t -v ])) . We also define Φ Proj ( δ ) = Π n i =1 Φ X i Proj ( δ )

By Lemma 1, to achieve convergence to an approximate Φ Proj ( δ ) -equilibrium, it suffices to minimize Φ X Proj ( δ ) -regret against convex losses. However, to our knowledge, there does not exist an algorithm that minimize Φ X Proj ( δ ) -regret even in the convex case. In fact, external regret and Φ X Proj ( δ ) -regret are provably incomparable: a sequence of actions may suffer high Reg T but low Reg T Proj ,δ (Example 3) or vise versa (Example 4). At a high level, the external regret competes against a fixed action, whereas Φ X Proj ( δ ) -regret is more akin to the notion of dynamic regret , competing with a sequence of varying actions. Despite this, surprisingly, we show that classical algorithms like Online Gradient Descent (GD) and Optimistic Gradient (OG), known for minimizing external regret, also attain near-optimal Φ X Proj ( δ ) -regret. We defer the examples and missing proofs to Appendix E.

Φ X Proj ( δ ) -Regret Minimization in the Adversarial Setting. We show that (GD) enjoys an O ( G √ δD X T ) Φ X Proj ( δ ) -regret despite the difference between the external regret and Φ X Proj ( δ ) -regret. First, let us recall the update rule of GD: given initial point x 1 ∈ X and step size η > 0 , GD updates in each iteration t :

The key step in our analysis for GD is simple but novel and general (See Appendix E.2). We extend the analysis to the Optimistic Gradient (OG) algorithm in Appendix E.4.

Theorem 3. Let δ > 0 and T ∈ N . For any convex and G -Lipschitz loss functions { f t : X → ❘ } t ∈ [ T ] , the Φ X Proj ( δ ) -regret of (GD) with step size η > 0 is Reg T Proj ,δ ≤ δ 2 2 η + η 2 G 2 T + δD X η . We can choose η optimally as √ δ ( δ + D X ) G √ T and attain Reg T Proj ,δ ≤ 2 G √ δ ( δ + D X ) T . For any δ > 0 and any ε > 0 , when all player employ GD in a smooth game, their empirical distribution of played strategy profiles converges to an ( ε + δ 2 L 2 ) -approximate Φ Proj ( δ ) -equilibrium in O (1 /ε 2 ) iterations.

Remark 1. The Φ X Proj ( δ ) -regret can also be viewed as the dynamic regret [Zin03] with changing comparators { p t := Π X [ x -v ] } . However, our analysis does not follow from standard O ( (1+ P T ) η + ηT ) dynamic regret bound of GD [Zin03] since P T , defined as ∑ T t =2 ∥ p t -p t -1 ∥ , can be Ω( ηT ) .

Lower bounds for Φ X Proj ( δ ) -regret. We complement our upper bound with two lower bounds for Φ X Proj ( δ ) -regret minimization. The first one is an Ω( δG √ T ) lower bound for any online learning algorithms against linear loss functions (Theorem 7). The second one is an Ω( δ 2 LT ) lower bound for any algorithm that satisfies the linear span assumption (Proposition 1), which holds for GD and OG against L -smooth non-convex losses. Combining with Lemma 1, this lower bound suggests that GD attains nearly optimal Φ X Proj ( δ ) -regret, even in the non-convex setting, among a natural family of gradient-based algorithms. We defer the theorem statements and detailed discussion to Appendix E.3.

Improved Φ X Proj ( δ ) -Regret in the Game Setting. We complement the Ω( √ T ) lower bound in the adversarial setting by considering the game setting where players interact with each other using the same algorithm setting, which has been extensively studied for concave games [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a]. We prove an improved O ( T 1 4 ) individual Φ X Proj ( δ ) -regret bound for OG (Theorem 9). We defer the details to Appendix E.4

4.2 Convex Combination of Finite Local Strategy Modifications

This section considers Conv(Φ) where Φ is a finite set of local strategy modifications. The set of infinite strategy modifications Conv(Φ) is defined as Conv(Φ) = { ϕ p ( x ) = ∑ ϕ ∈ Φ p ( ϕ ) ϕ ( x ) : p ∈ ∆(Φ) } . Our main result is an efficient algorithm (Algorithm 3) that guarantees convergence to an ε -approximate Conv(Φ) -equilibrium in a smooth game satisfying Assumption 1 for any ε > δ 2 L . Due to space constraints, we defer Algorithm 3 and the proof to Appendix F.

Theorem 4. Let X be a convex and compact set, Φ be an arbitrary finite set of δ -local strategy modification functions for X , and u 1 ( · ) , . . . , u T ( · ) be a sequence of G -Lipschitz and L -smooth but possibly non-concave reward functions from X to [0 , 1] . If we instantiate Algorithm 3 with R Φ being the Hedge algorithm over ∆(Φ) and K = √ T , the algorithm guarantees that, with probability at least 1 -β , it produces a sequence of strategies x 1 , . . . , x T with Conv(Φ) -regret at most 8 √ T ( Gδ √ log | Φ | + √ log(1 /β )) + δ 2 LT . The algorithm runs in time √ T | Φ | per iteration.

If all players in a non-concave smooth game employ Algorithm 3, then with probability 1 -β , for any ε > 0 , the empirical distribution of strategy profiles played forms an ( ε + δ 2 L ) -approximate Φ = Π n i =1 Φ X i -equilibrium' after poly ( 1 ε , G, log ( max i | Φ X i | ) , log n β ) iterations.

4.3 Interpolation-Based Local Strategy Modifications

We introduce a natural set of local strategy modifications and the corresponding local equilibrium notion. Given any set of (possibly non-local) strategy modifications Ψ = { ψ : X → X} , we define a set of local strategy modifications as follows: for δ ≤ D X and λ ∈ [0 , 1] , each strategy modification ϕ λ,ψ interpolates the input strategy x with the modified strategy ψ ( x ) : formally,

Note that for any ψ ∈ Ψ and λ ≤ δ D X , we have ∥ ϕ λ,ψ ( x ) -x ∥ = λ ∥ x -ψ ( x ) ∥ ≤ δ , respecting the locality constraint. The induced Φ X Int , Ψ ( δ ) -regret can be written as Reg T Int , Ψ ,δ := max ψ ∈ Ψ ,λ ≤ δ D X ∑ T t =1 ( f t ( x t ) -f t ((1 -λ ) x t + λψ ( x t ))) . To guarantee convergence to the corresponding Φ -equilibrium, it suffices to minimize Φ X Int , Ψ ( δ ) -regret against convex losses, which we show further reduces to Ψ -regret minimization against convex losses (Theorem 10 in Appendix G).

CCE-like Instantiation. In the special case where Ψ contains only constant strategy modifications (i.e., ψ ( x ) = x ∗ for all x ), we get a coarse correlated equilibrium (CCE)-like instantiation of local equilibrium, which limits the gain by interpolating with any fixed strategy. We denote the resulting set of local strategy modification simply as Φ X Int ( δ ) . We can apply any no-external regret algorithm for efficient Φ X Int ( δ ) -regret minimization and computation of ε -approximate Φ Int ( δ ) -equilibrium in the first-order stationary regime as summarized in Theorem 5. We also discuss faster convergence rates in the game setting in Appendix G.

Theorem 5. For the Online Gradient Descent algorithm (GD) [Zin03] with step size η = D X G √ T , its Φ X Int ( δ ) -regret is at most 2 δG √ T . Furthermore, for any δ > 0 and any ε > δ 2 L 2 , when all players employ the GD algorithm in a smooth game, their empirical distribution of played strategy profiles converges to an ( ε + δ 2 L 2 ) -approximate Φ Int ( δ ) -equilibrium in O (1 /ε 2 ) iterations.

5 Discussion and Future Directions

Lower Bound in the Global Regime When δ equals the diameter of our strategy set, it is NP-hard to compute an ε -approximate Φ( δ ) -equilibrium (for Φ( δ ) = Φ Proj ( δ ) , Φ Int ( δ ) ), even when ε = Θ(1) and G,L = O (poly( d )) . Moreover, given black-box access to value and gradient queries, finding such equilibria requires exponentially many queries in at least one of the parameters d, G, L, 1 /ε . These results are presented as Theorem 12 and Theorem 13 in Appendix I.

More Efficient Φ -Equilibria We include the discussion of another natural class of local strategy modifications that is based on beam search, where GD suffers linear regret to Appendix H. This result shows that even for simple local strategy modification sets Φ( δ ) , the landscape of efficient local Φ( δ ) -regret minimization is already quite rich and many questions remain open. A fruitful future direction is to identify more classes of Φ that admit efficient regret minimization.

References

[AD54] Kenneth J Arrow and Gerard Debreu. 'Existence of an equilibrium for a competitive economy'. In: Econometrica (1954), pp. 265-290.

[AFS23] Ioannis Anagnostides, Gabriele Farina, and Tuomas Sandholm. 'Near-Optimal PhiRegret Learning in Extensive-Form Games'. In: International Conference on Machine Learning (ICML) . 2023.

[AGH19] Naman Agarwal, Alon Gonen, and Elad Hazan. 'Learning in non-convex games with an optimization oracle'. In: Conference on Learning Theory . PMLR. 2019, pp. 18-29.

[AHK12] Sanjeev Arora, Elad Hazan, and Satyen Kale. 'The multiplicative weights update method: a meta-algorithm and applications'. In: Theory of computing 8.1 (2012), pp. 121-164.

[ALW21] Jacob Abernethy, Kevin A Lai, and Andre Wibisono. 'Last-iterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization'. In: Algorithmic Learning Theory . PMLR. 2021, pp. 3-47.

[Ana+22a] Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. 'Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games'. In: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing (STOC) . 2022.

[Ana+22b] Ioannis Anagnostides, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, and Tuomas Sandholm. 'Uncoupled Learning Dynamics with O (log T ) Swap Regret in Multiplayer Games'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2022.

[Aue+02] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 'The nonstochastic multiarmed bandit problem'. In: SIAM journal on computing 32.1 (2002), pp. 48-77.

[AZ22] Amir Ali Ahmadi and Jeffrey Zhang. 'On the complexity of finding a local minimizer of a quadratic function over a polytope'. In: Mathematical Programming 195.1 (2022), pp. 783-792.

[AZF19] Sergul Aydore, Tianhao Zhu, and Dean P Foster. 'Dynamic local regret for non-convex online forecasting'. In: Advances in neural information processing systems 32 (2019).

[Bai+22] Yu Bai, Chi Jin, Song Mei, Ziang Song, and Tiancheng Yu. 'Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 22313-22325.

[Ber+23] Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Francesco Trovò, and Nicola Gatti. 'Constrained Phi-Equilibria'. In: International Conference on Machine Learning . 2023.

[Bla56] David Blackwell. 'An analog of the minimax theorem for vector payoffs.' In: Pacific Journal of Mathematics 6.1 (Jan. 1956). Publisher: Pacific Journal of Mathematics, A Non-profit Corporation, pp. 1-8.

[Bro51] George W Brown. 'Iterative solution of games by fictitious play'. In: Act. Anal. Prod Allocation 13.1 (1951), p. 374.

[Bub+15] Sébastien Bubeck et al. 'Convex optimization: Algorithms and complexity'. In: Foundations and Trends® in Machine Learning 8.3-4 (2015), pp. 231-357.

[Cau01] Robert Cauty. 'Solution du problème de point fixe de Schauder'. en. In: Fundamenta Mathematicae 170 (2001). Publisher: Instytut Matematyczny Polskiej Akademii Nauk, pp. 231-246.

[CDT09] Xi Chen, Xiaotie Deng, and Shang-Hua Teng. 'Settling the complexity of computing two-player Nash equilibria'. In: Journal of the ACM (JACM) 56.3 (2009), pp. 1-57.

[CL06] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

[CP20] Xi Chen and Binghui Peng. 'Hedging in games: Faster convergence of external and swap regrets'. In: Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), pp. 18990-18999.

[CZ23] Yang Cai and Weiqiang Zheng. 'Accelerated Single-Call Methods for Constrained Min-Max Optimization'. In: International Conference on Learning Representations (ICLR) (2023).

[Dag+24] Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. 'From External to Swap Regret 2.0: An Efficient Reduction for Large Action Spaces'. In: Proceedings of the 56th Annual ACM Symposium on Theory of Computing . 2024, pp. 1216-1222.

[Das+23] Constantinos Daskalakis, Noah Golowich, Stratis Skoulakis, and Emmanouil Zampetakis. 'STay-ON-the-Ridge: Guaranteed Convergence to Local Minimax Equilibrium in Nonconvex-Nonconcave Games'. In: The Thirty Sixth Annual Conference on Learning Theory . PMLR. 2023, pp. 5146-5198.

[Das22] Constantinos Daskalakis. 'Non-Concave Games: A Challenge for Game Theory's Next 100 Years'. In: Cowles Preprints (2022).

[DDJ21] Jelena Diakonikolas, Constantinos Daskalakis, and Michael Jordan. 'Efficient methods for structured nonconvex-nonconcave min-max optimization'. In: International Conference on Artificial Intelligence and Statistics (2021).

[Deb52] Gerard Debreu. 'A social equilibrium existence theorem'. In: Proceedings of the National Academy of Sciences 38.10 (1952), pp. 886-893.

[DFG21] Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. 'Near-optimal noregret learning in general games'. In: Advances in Neural Information Processing Systems (NeurIPS) (2021).

[DGP09] Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. 'The complexity of computing a Nash equilibrium'. In: Communications of the ACM 52.2 (2009), pp. 89-97.

[DP18] Constantinos Daskalakis and Ioannis Panageas. 'The limit points of (optimistic) gradient descent in min-max optimization'. In: the 32nd Annual Conference on Neural Information Processing Systems (NeurIPS) . 2018.

[DSZ21] Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. 'The complexity of constrained min-max optimization'. In: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC) . 2021.

[Fan53] Ky Fan. 'Minimax theorems'. In: Proceedings of the National Academy of Sciences 39.1 (1953), pp. 42-47.

[Far+22a] Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. 'Near-optimal no-regret learning dynamics for general convex games'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 3907639089.

[Far+22b] Gabriele Farina, Andrea Celli, Alberto Marchesi, and Nicola Gatti. 'Simple Uncoupled No-regret Learning Dynamics for Extensive-form Correlated Equilibrium'. In: Journal of the ACM 69.6 (2022).

[FCR20] Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. 'Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study'. In: International Conference on Machine Learning . PMLR. 2020, pp. 31333144.

[FR21] Tanner Fiez and Lillian J Ratliff. 'Local convergence analysis of gradient descent ascent with finite timescale separation'. In: Proceedings of the International Conference on Learning Representation . 2021.

[FS99] Yoav Freund and Robert E. Schapire. 'Adaptive Game Playing Using Multiplicative Weights'. In: Games and Economic Behavior 29 (1999), pp. 79-103.

[Geo63] George B. Dantzig. Linear Programming and Extensions . Princeton University Press, 1963.

[GGM08] Geoffrey J Gordon, Amy Greenwald, and Casey Marks. 'No-regret learning in convex games'. In: Proceedings of the 25th international conference on Machine learning . 2008, pp. 360-367.

[GJ03] Amy Greenwald and Amir Jafari. 'A general class of no-regret learning algorithms and game-theoretic equilibria'. In: Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings . Springer. 2003, pp. 2-12.

[Gli52] Irving L Glicksberg. 'A further generalization of the Kakutani fixed theorem, with application to Nash equilibrium points'. In: Proceedings of the American Mathematical Society 3.1 (1952), pp. 170-174.

[GLZ18] Xiand Gao, Xiaobo Li, and Shuzhong Zhang. 'Online learning with non-convex losses and non-stationary regret'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2018, pp. 235-243.

[GZL23] Ziwei Guan, Yi Zhou, and Yingbin Liang. 'Online Nonconvex Optimization with Limited Instantaneous Oracle Feedback'. In: The Thirty Sixth Annual Conference on Learning Theory . PMLR. 2023, pp. 3328-3355.

[Han57] James Hannan. 'Approximation to Bayes risk in repeated play'. In: Contributions to the Theory of Games 3 (1957), pp. 97-139.

[Hél+20] Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. 'Online non-convex optimization with imperfect feedback'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 17224-17235.

[HMC21a] Nadav Hallak, Panayotis Mertikopoulos, and Volkan Cevher. 'Regret minimization in stochastic non-convex learning via a proximal-gradient approach'. In: International Conference on Machine Learning . PMLR. 2021, pp. 4008-4017.

[HMC21b] Ya-Ping Hsieh, Panayotis Mertikopoulos, and Volkan Cevher. 'The limits of min-max optimization algorithms: Convergence to spurious non-critical sets'. In: International Conference on Machine Learning . PMLR. 2021, pp. 4337-4348.

[HSZ17] Elad Hazan, Karan Singh, and Cyril Zhang. 'Efficient regret minimization in nonconvex games'. In: International Conference on Machine Learning . PMLR. 2017, pp. 1433-1441.

[JNJ20] Chi Jin, Praneeth Netrapalli, and Michael Jordan. 'What is local optimality in nonconvex-nonconcave minimax optimization?' In: International conference on machine learning (ICML) . PMLR. 2020, pp. 4880-4889.

[Kar14] Samuel Karlin. Mathematical Methods and Theory in Games, Programming, and Economics: Volume 2: The Theory of Infinite Games . Elsevier, 2014.

[Kar59] Samuel Karlin. Mathematical methods and theory in games, programming, and economics: Volume II: the theory of infinite games . Vol. 2. Addision-Wesley, 1959.

[Kri+15] Walid Krichene, Maximilian Balandat, Claire Tomlin, and Alexandre Bayen. 'The hedge algorithm on a continuum'. In: International Conference on Machine Learning . PMLR. 2015, pp. 824-832.

[McK54] Lionel McKenzie. 'On equilibrium in Graham's model of world trade and other competitive systems'. In: Econometrica (1954), pp. 147-161.

[MK87] Katta G. Murty and Santosh N. Kabadi. 'Some NP-complete problems in quadratic and nonlinear programming'. en. In: Mathematical Programming 39.2 (June 1987), pp. 117-129.

[MM10] Odalric-Ambrym Maillard and Rémi Munos. 'Online learning in adversarial lipschitz environments'. In: Joint european conference on machine learning and knowledge discovery in databases . Springer. 2010, pp. 305-320.

[Mor+21a] Dustin Morrill, Ryan D'Orazio, Reca Sarfati, Marc Lanctot, James R Wright, Amy R Greenwald, and Michael Bowling. 'Hindsight and sequential rationality of correlated play'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 35. 6. 2021, pp. 5584-5594.

[Mor+21b] Dustin Morrill, Ryan D'Orazio, Marc Lanctot, James R Wright, Michael Bowling, and Amy R Greenwald. 'Efficient deviation types and learning for hindsight rationality in extensive-form games'. In: International Conference on Machine Learning . PMLR. 2021, pp. 7818-7828.

[MRS20] Eric Mazumdar, Lillian J Ratliff, and S Shankar Sastry. 'On gradient-based learning in continuous games'. In: SIAM Journal on Mathematics of Data Science 2.1 (2020), pp. 103-131.

[MV21] Oren Mangoubi and Nisheeth K Vishnoi. 'Greedy adversarial equilibrium: an efficient alternative to nonconvex-nonconcave min-max optimization'. In: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing . 2021, pp. 896-909.

[MZ19] Panayotis Mertikopoulos and Zhengyuan Zhou. 'Learning in games with continuous action sets and unknown payoff functions'. In: Mathematical Programming 173 (2019), pp. 465-507.

[Nas50] John F Nash Jr. 'Equilibrium points in n-person games'. In: Proceedings of the national academy of sciences 36.1 (1950), pp. 48-49.

[Neu28] J v. Neumann. 'Zur theorie der gesellschaftsspiele'. In: Mathematische annalen 100.1 (1928), pp. 295-320.

[Pet+22] Thomas Pethick, Puya Latafat, Panagiotis Patrinos, Olivier Fercoq, and Volkan Cevherå. 'Escaping limit cycles: Global convergence for constrained nonconvex-nonconcave minimax problems'. In: International Conference on Learning Representations (ICLR) . 2022.

[Pil+22] Georgios Piliouras, Mark Rowland, Shayegan Omidshafiei, Romuald Elie, Daniel Hennes, Jerome Connor, and Karl Tuyls. 'Evolutionary dynamics and phi-regret minimization in games'. In: Journal of Artificial Intelligence Research 74 (2022), pp. 11251158.

[PR24] Binghui Peng and Aviad Rubinstein. 'Fast swap regret minimization and applications to approximate correlated equilibria'. In: Proceedings of the 56th Annual ACM Symposium on Theory of Computing . 2024, pp. 1223-1234.

[RBS16] Lillian J Ratliff, Samuel A Burden, and S Shankar Sastry. 'On the characterization of local Nash equilibria in continuous games'. In: IEEE transactions on automatic control 61.8 (2016), pp. 2301-2307.

[Rob51] Julia Robinson. 'An iterative method of solving a game'. In: Annals of mathematics (1951), pp. 296-301.

[Ros65] J Ben Rosen. 'Existence and uniqueness of equilibrium points for concave n-person games'. In: Econometrica (1965), pp. 520-534.

[RS13] Sasha Rakhlin and Karthik Sridharan. 'Optimization, learning, and games with predictable sequences'. In: Advances in Neural Information Processing Systems (2013).

[RST11] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. 'Online learning: Beyond regret'. In: Proceedings of the 24th Annual Conference on Learning Theory . JMLR Workshop and Conference Proceedings. 2011, pp. 559-594.

[Sha24] Dravyansh Sharma. 'No Internal Regret with Non-convex Loss Functions'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 38. 13. 2024, pp. 1491914927.

[Sio58] Maurice Sion. 'On general minimax theorems.' In: Pacific J. Math. 8.4 (1958), pp. 171176.

[SL07] Gilles Stoltz and Gábor Lugosi. 'Learning correlated equilibria in games with compact sets of strategies'. In: Games and Economic Behavior 59.1 (2007), pp. 187-208.

[SMB22] Ziang Song, Song Mei, and Yu Bai. 'Sample-efficient learning of correlated equilibria in extensive-form games'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 4099-4110.

[SN20] Arun Sai Suggala and Praneeth Netrapalli. 'Online non-convex learning: Following the perturbed leader is optimal'. In: Algorithmic Learning Theory . PMLR. 2020, pp. 845861.

[Syr+15] Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. 'Fast convergence of regularized learning in games'. In: Advances in Neural Information Processing Systems (NeurIPS) (2015).

[VF08] Bernhard Von Stengel and Françoise Forges. 'Extensive-form correlated equilibrium: Definition and computational complexity'. In: Mathematics of Operations Research 33.4 (2008), pp. 1002-1022.

Contents

A Related Work

Non-Concave Games. An important special case of multi-player games are two-player zero-sum games, which are defined in terms of some function f : X × Y → ❘ that one of the two players say the one choosing x ∈ X , wants to minimize, while the other player, the one choosing y ∈ Y , wants to maximize. Finding Nash equilibrium in such games is tractable in the convex-concave setting, i.e. when f ( x, y ) is convex with respect to the minimizing player's strategy, x , and concave with respect to the maximizing player's strategy, y , but it is computationally intractable in the general nonconvex-nonconcave setting. Namely, a Nash equilibrium may not exist, and it is NP-hard to determine if one exists and, if so, find it. Moreover, in this case, stable limit points of gradientbased dynamics are not necessarily Nash equilibria, not even local Nash equilibria [DP18; MRS20]. Moreover, there are examples including the 'Polar Game' [Pet+22] and the 'Forsaken Matching Pennies' [HMC21b] showing that for GD / OG and many other no-regret learning algorithms in nonconvex-nonconcave min-max optimization, the last-iterate does not converge and even the averageiterate fails to be a stationary point. We emphasize that the convergence guarantees we provide for GD / OG in Section 4.1 and Section 4.3 holds for the empirical distribution of play, not the average-iterate or the last-iterate.

A line of work focuses on computing Nash equilibrium under additional structure in the game. This encompasses settings where the game satisfies the (weak) Minty variational inequality [MZ19; DDJ21; Pet+22; CZ23], or is sufficiently close to being bilinear [ALW21]. However, the study of universal solution concepts in the nonconvex-nonconcave setting is sparse. Daskalakis, Skoulakis, and Zampetakis [DSZ21] proved the existence and computational hardness of local Nash equilibrium. In a more recent work, [Das+23] proposes second-order algorithms with asymptotic convergence to local Nash equilibrium. Several works study sequential two-player zero-sum games with additional assumptions about the player who goes second. They propose equilibrium concepts such as local minimax points [JNJ20], differentiable Stackelberg equilibrium [FCR20], and greedy adversarial equilibrium [MV21]. Notably, local minimax points are stable limit points of Gradient-DescentAscent (GDA) dynamics [JNJ20; WZB20; FR21] while greedy adversarial equilibrium can be computed efficiently using second-order algorithms in the unconstrained setting [MV21]. In contrast to these studies, we focus on the more general case of multi-player non-concave games.

Local Equilibrium. To address the limitations associated with classical, global equilibrium concepts, a natural approach is to focus on developing equilibrium concepts that guarantee local stability instead. One definition of interest is the strict local Nash equilibrium, wherein each player's strategy corresponds to a local maximizer of their utility function, given the other players' strategies. Unfortunately, a strict local Nash equilibrium may not always exist, as demonstrated in Example 1. Furthermore, a weaker notion-the second-order local Nash equilibrium, where each player has no incentive to deviate based on the second-order Taylor expansion estimate of their utility, is also not guaranteed to exist as illustrated in Example 1. What's more, it is NP-hard to check whether a given strategy profile is a strict local Nash equilibrium or a second-order local Nash equilibrium, as implied by the result of Murty and Kabadi [MK87] and Ahmadi and Zhang [AZ22]. 6 Finally, one can consider local Nash equilibrium , a first-order stationary solution, which is guaranteed to exist [DSZ21]. Unlike non-convex optimization, where targeting first-order local optima sidesteps the intractability of global optima, this first-order local Nash equilibrium has been recently shown to be intractable, even in

two-player zero-sum non-concave games with joint feasibility constraints [DSZ21]. 7 See Table 1 for a summary of solution concepts in non-concave games.

Online Learning with Non-Convex Losses. A line of work has studied online learning against non-convex losses. To circumvent the computational intractability of this problem, various approaches have been pursued: some works assume a restricted set of non-convex loss functions [GLZ18], while others assume access to a sampling oracle [MM10; Kri+15] or access to an offline optimization oracle [AGH19; SN20; Hél+20] or a weaker notion of regret [HSZ17; AZF19; HMC21a; GZL23]. The work most closely related to ours is [HSZ17]. The authors propose a notion of w -smoothed local regret against non-convex losses, and they also define a local equilibrium concept for non-concave games. They use the idea of smoothing to average the loss functions in the previous w iterations and design algorithms with optimal w -smoothed local regret. The concept of regret they introduce suggests a local equilibrium concept. However, their local equilibrium concept is non-standard in that its local stability is not with respect to a distribution over strategy profiles sampled by this equilibrium concept. Moreover, the path to attaining this local equilibrium through decentralized learning dynamics remains unclear. The algorithms provided in [HSZ17; GZL23] require that every agent i experiences (over several rounds) the average utility function of the previous w iterates, denoted as F t i,w := 1 w ∑ w -1 ℓ =0 u t -ℓ i ( · , x t -ℓ -i ) . Implementing this imposes a significant coordination burden on the agents. In contrast, we focus on a natural concept of Φ( δ ) -equilibrium, which is incomparable to that of [HSZ17], and we also show that efficient convergence to this concept is achieved via decentralized gradient-based learning dynamics.

Φ -regret and Φ -equilibrium. The concept of Φ -regret and the associated Φ -equilibrium is introduced by Greenwald and Jafari [GJ03] and has been broadly investigated in the context of concave games [GJ03; SL07; GGM08; RST11; Pil+22; Ber+23; Dag+24; PR24] and extensiveform games [VF08; Mor+21a; Mor+21b; Far+22b; Bai+22; SMB22; AFS23; Zha+24]. The work of [Sha24] studies internal regret minimization against non-convex losses. To our knowledge, no efficient algorithm exists for the classes of Φ -equilibria we consider for non-concave games. Specifically, all of these algorithms, when applied to compute a ( ε, Φ( δ )) -equilibrium for a general δ -local strategy modification set Φ( δ ) (using Lemma 1), require running time exponential in either 1 /ε or the dimension d . In contrast, we show that for several natural choices of Φ( δ ) , ε -approximate Φ( δ ) -equilibrium can be computed efficiently, i.e. polynomial in 1 /ε and d , using simple algorithms.

B Additional Preliminaries: Solution Concepts in Non-Concave Games

We present definitions of several solution concepts in the literature as well as the existence and computational complexity of each solution concept.

Definition 4 (Nash Equilibrium) . In a continuous game, a strategy profile x ∈ ∏ n j =1 X j is a Nash equilibrium (NE) if and only if for every player i ∈ [ n ] ,

Definition 5 (Mixed Nash Equilibrium) . In a continuous game, a mixed strategy profile p ∈ ∏ n j =1 ∆( X j ) (here we denote ∆( X i ) as the set of probability measures over X i ) is a mixed Nash equilibrium (MNE) if and only if for every player i ∈ [ n ] ,

Definition 6 ((Coarse) Correlated Equilibrium) . In a continuous game, a distribution σ over joint strategy profiles Π n i =1 X i is a correlated equilibrium (CE) if and only if for all player i ∈ [ n ] ,

Similarly, a distribution σ over joint strategy profiles Π n i =1 X i is a coarse correlated equilibrium (CCE) if and only if for all player i ∈ [ n ] ,

Definition 7 (Strict Local Nash Equilibrium) . In a continuous game, a strategy profile x ∈ ∏ n j =1 X j is a strict local Nash equilibrium if and only if for every player i ∈ [ n ] , there exists δ > 0 such that

Definition 8 (Second-order Local Nash Equilibrium) . Consider a continuous game where each utility function u i ( x i , x -i ) is twice-differentiable with respect to x i for any fixed x -i . A strategy profile x ∈ ∏ n j =1 X j is a second-order local Nash equilibrium if and only if for every player i ∈ [ n ] , x i maximizes the second-order Taylor expansion of its utility functions at x i , or formally,

Existence Mixed Nash equilibria exist in continuous games, thus smooth games [Deb52; Gli52; Fan53]. By definition, an MNE is also a CE and a CCE. This also proves the existence of CE and CCE. In contrast, strict local Nash equilibria, second-order Nash equilibria, or (pure) Nash equilibria may not exist in a smooth non-concave game, as we show in the following example.

̸

Example 1. Consider a two-player zero-sum non-concave game: the action sets are X 1 = X 2 = [ -1 , 1] and the utility functions are u 1 ( x 1 , x 2 ) = -u 2 ( x 1 , x 2 ) = ( x 1 -x 2 ) 2 . Let x = ( x 1 , x 2 ) ∈ X 1 ×X 2 be any strategy profile: if x 1 = x 2 , then player 1 is not at a local maximizer; if x 1 = x 2 , then player 2 is not at a local maximizer. Thus x is not a strict local Nash equilibrium. Since the utility function is quadratic, we conclude that the game also has no second-order local Nash equilibrium.

Computational Complexity Consider a single-player smooth non-concave game with a quadratic utility function f : X → ❘ . The problem of finding a local maximizer of f can be reduced to the problem of computing a NE, a MNE, a CE, a CCE, a strict local Nash equilibrium, or a second-order local Nash equilibrium. Since computing a local maximizer or checking if a given point is a local maximizer is NP-hard [MK87], we know that the computational complexities of NE, MNE, CE, CCE, strict local Nash equilibria, and second-order local Nash equilibria are all NP-hard.

Representation Complexity Karlin [Kar59] present a two-player zero-sum non-concave game whose unique MNE has infinite support. Since in a two-player zero-sum game, the marginal distribution of a CE or a CCE is an MNE, it also implies that the representation complexity of any CE or CCE is infinite. We present the example in Karlin [Kar59] here for completeness and also prove that the game is Lipschitz and smooth.

Example 2 ([Kar59, Chapter 7.1, Example 3]) . We consider a two-player zero-sum game with action sets X 1 = X 2 = [0 , 1] . Let p and q be two distributions over [0 , 1] . The only requirement for p and q is that their cumulative distribution functions are not finite-step functions. For example, we can take p = q to be the uniform distribution.

Let µ n and ν n denote the n -th moments of p and q , respectively. Define the utility function

Claim 1. The game in Example 2 is 2 -Lipschitz and 6 -smooth, and ( p, q ) is its unique (mixed) Nash equilibrium.

Proof. Fix any y ∈ [0 , 1] , since | 1 2 n ( y n -ν n ) nx n -1 | ≤ n 2 n , the series of ∇ x u ( x, y ) is uniformly convergent. We have |∇ x u ( x, y ) | ≤ ∑ ∞ n =0 n 2 n ≤ 2 , y ∈ [0 , 1] . Similarly, we have |∇ 2 x u ( x, y ) | ≤ ∑ ∞ n =0 n 2 2 n ≤ 6 for all y ∈ [0 , 1] . By symmetry, we also have |∇ y ( x, y ) | ≤ 2 and |∇ 2 y ( x, y ) | ≤ 6 for all x, y ∈ [0 , 1] . Thus, the game is 2 -Lispchitz and 6 -smooth.

Since | 1 2 n ( x n -µ n )( y n -ν n ) | ≤ 1 2 n , the series of u ( x, y ) is absolutely and uniformly convergent. We have

In particular, ( p, q ) is a mixed Nash equilibrium, and the value of the game is 0 . Suppose ( p ′ , q ′ ) is also a mixed Nash equilibrium. Then ( p, q ′ ) is a mixed Nash equilibrium. Note that p supports on every point in [0 , 1] . As a consequence, we have

for all x ∈ [0 , 1] , where ν ′ n is the n -th moment of q ′ . Since the series vanished identically, the coefficients of each power of x must vanish. Thus ν ′ n = ν n and q ′ = q . Similarly, we have p ′ = p , and the mixed Nash equilibrium is unique.

C Rerget Bound for Hedge

Theorem 6 ([AHK12]) . In an N -expert problem, assume all the rewards are bounded, i.e., u t ∈ [ -M,M ] N , then the Hedge algorithm with step size η = min { 1 M , √ log N M √ T } has regret

D Proof of Lemma 1

Let { f t } t ∈ [ T ] be a sequence of non-convex L -smooth loss functions satisfying Assumption 1. Let { x t } t ∈ [ T ] be the iterates produced by A against { f t } t ∈ [ T ] . Then { x t } t ∈ [ T ] is also the iterates produced by A against a sequence of linear loss functions {⟨∇ f t ( x t ) , ·⟩} . For the latter, we know

Then using L -smoothness of { f t } and the fact that ∥ ϕ ( x ) -x ∥ ≤ δ for all ϕ ∈ Φ( δ ) , we get

This completes the proof of the first part.

Let each player i ∈ [ n ] employ algorithm A in a smooth game independently and produces iterates { x t } . The averaged joint strategy profile σ T that chooses x t uniformly at random from t ∈ [ T ] satisfies for any player i ∈ [ n ] ,

Thus σ T is a (max i ∈ [ n ] { Reg T Φ X i ( δ ) } · T -1 + δ 2 L 2 ) -approximate Φ( δ )) -equilibrium. This completes the proof of the second part.

E Missing Details in Section 4.1

E.1 Differences between External Regret and Φ X Proj -regret

In the following two examples, we show that Φ X Proj ( δ ) -regret is incomparable with external regret for convex loss functions . A sequence of actions may suffer high Reg T but low Reg T Proj ,δ (Example 3), and vise versa (Example 4).

Example 3. Let f 1 ( x ) = f 2 ( x ) = | x | for x ∈ X = [ -1 , 1] . Then the Φ X Proj ( δ ) -regret of the sequence { x 1 = 1 2 , x 2 = -1 2 } for any δ ∈ (0 , 1 2 ) is 0 . However, the external regret of the same sequence is 1 . By repeating the construction for T 2 times, we conclude that there exists a sequence of actions with Reg T Proj ,δ = 0 and Reg T = T 2 for all T ≥ 2 .

Example 4. Let f 1 ( x ) = -2 x and f 2 ( x ) = x for x ∈ X = [ -1 , 1] . Then the Φ X Proj ( δ ) -regret of the sequence { x 1 = 1 2 , x 2 = 0 } for any δ ∈ (0 , 1 2 ) is δ . However, the external regret of the same sequence is 0 . By repeating the construction for T 2 times, we conclude that there exists a sequence of actions with Reg T Proj ,δ = δT 2 and Reg T = 0 for all T ≥ 2 .

At a high level, the external regret competes against a fixed action, whereas Φ X Proj ( δ ) -regret is more akin to the notion of dynamic regret , competing with a sequence of varying actions. When the environment is stationary, i.e., f t = f (Example 3), a sequence of actions that are far away from the global minimum must suffer high regret, but may produce low Φ X Proj ( δ ) -regret since the change to the cumulative loss caused by a fixed-direction deviation could be neutralized across different actions in the sequence. In contrast, in a non-stationary (dynamic) environment (Example 4), every fixed action performs poorly, and a sequence of actions could suffer low regret against a fixed action but the Φ X Proj ( δ ) -regret that competes with a fixed-direction deviation could be large. Nevertheless, despite these differences between the two notions of regret as shown above, they are compatible for convex loss functions: our main results in this section provide algorithms that minimize external regret and Φ X Proj ( δ ) -regret simultaneously.

E.2 Proof of Theorem 3

Proof. Let us denote v ∈ B d ( δ ) a fixed deviation and define p t = Π X [ x t -v ] . By standard analysis of GD [Zin03] (see also the proof of [Bub+15, Theorem 3.2] ), we have

where the last step uses ∥ x 1 -p 1 ∥ ≤ δ and ∥∇ f t ( x t ) ∥ ≤ G . Here the terms ∥ x t +1 -p t +1 ∥ 2 -∥ x t +1 -p t ∥ 2 do not telescope, and we further relax them in the following key step.

Key Step: We relax the first term as:

where in the last inequality we use the fact that p t +1 is the projection of x t +1 -v onto X and p t is in X . Now we get a telescoping term 2 ⟨ p t -p t +1 , u ⟩ and a negative term -∥ p t -p t +1 ∥ 2 . The negative term is useful for improving the regret analysis in the game setting, but we ignore it for now. Combining the two inequalities above, we have

Since the above holds for any v with ∥ v ∥ ≤ δ , it also upper bounds Reg Proj ,δ

E.3 Lower bounds for Φ X Proj -Regret

Theorem 7 (Lower bound for Φ X Proj ( δ ) -regret against convex losses) . For any T ≥ 1 , D X > 0 , 0 < δ ≤ D X , and G ≥ 0 , there exists a distribution D on G -Lipschitz linear loss functions f 1 , . . . , f T over X = [ -D X , D X ] such that for any online algorithm, its Φ X Proj ( δ ) -regret on the loss sequence satisfies E D [Reg T Proj ,δ ] = Ω( δG √ T ) .

√

Remark 2. A keen reader may notice that the Ω( Gδ T ) lower bound in Theorem 7 does not match the O ( G √ δD X T ) upper bound in Theorem 3, especially when D X ≫ δ . A natural question is: which of them is tight? We conjecture that the lower bound is tight. In fact, for the special case where the feasible set X is a box , we obtain a D X -independent bound O ( d 1 4 Gδ √ T ) using a modified version of GD, which is tight when d = 1 . See Appendix J for a detailed discussion.

This lower bound suggests that GD achieves near-optimal Φ X Proj ( δ ) -regret for convex losses. For L -smooth non-convex loss functions, we provide another Ω( δ 2 LT ) lower bound for algorithms that satisfy the linear span assumption. The linear span assumption states that the algorithm produces x t +1 ∈ { Π X [ ∑ i ∈ [ t ] a i · x i + b i · ∇ f i ( x i )] : a i , b i ∈ ❘ , ∀ i ∈ [ t ] } as essentially the linear combination of the previous iterates and their gradients. Many online algorithms such as online gradient descent and optimistic gradient satisfy the linear span assumption. Combining with Lemma 1, this lower bound suggests that GD attains nearly optimal Φ X Proj ( δ ) -regret, even in the non-convex setting, among a natural family of gradient-based algorithms.

Proposition 1 (Lower bound for Φ X Proj ( δ ) -regret against non-convex losses) . For any T ≥ 1 , δ ∈ (0 , 1) , and L ≥ 0 , there exists a sequence of L -Lipschitz and L -smooth non-convex loss functions f 1 , . . . , f T on X = [ -1 , 1] such that for any algorithm that satisfies the linear span assumption, its Φ X Proj ( δ ) -regret on the loss sequence is Reg T Proj ,δ ≥ δ 2 LT 2 .

E.3.1 Proof of Theorem 7

Our proof technique comes from the standard one used in multi-armed bandits [Aue+02, Theorem 5.1]. Suppose that f t ( x ) = g t x . We construct two possible environments. In the first environment, g t = G with probability 1+ ε 2 and g t = -G with probability 1 -ε 2 ; in the second environment, g t = G with probability 1 -ε 2 and g t = -G with probability 1+ ε 2 . We use E i and P i to denote the expectation and probability measure under environment i , respectively, for i = 1 , 2 . Suppose that the true environment is uniformly chosen from one of these two environments. Below, we show that the expected regret of the learner is at least Ω( δG √ T ) .

Define N + = ∑ T t =1 I { x t ≥ 0 } be the number of times x t is non-negative, and define f 1: T = ( f 1 , . . . , f T ) . Then we have

In the first environment, we consider the regret with respect to v = δ . Then we have

where in the last inequality we use the fact that if x t ≥ 0 then x t -Π X [ x t -δ ] = x t -( x t -δ ) = δ because D ≥ δ . In the second environment, we consider the regret with respect to v = -δ . Then similarly, we have

Summing up the two inequalities, we get

Choosing ε = 1 √ (16 ln 2) T , we can lower bound the last expression by Ω( δG T ) . The theorem is proven by noticing that 1 2 ( E 1 [ Reg T Proj ,δ ] + E 2 [ Reg T Proj ,δ ]) is the expected regret of the learner.

E.3.2 Proof of Proposition 1

Proof. Consider f : [ -1 , 1] → ❘ such that f ( x ) = -L 2 x 2 and let f t = f for all t ∈ [ T ] . Then any first-order methods that satisfy the linear span assumption with initial point x 1 = 0 will produce x t = 0 for all t ∈ [ T ] . The Φ X Proj ( δ ) -regret is thus ∑ T t =1 ( f (0) -f ( δ )) = δ 2 LT 2 .

E.4 Improved Φ X Proj ( δ ) -Regret in the Game Setting and Proof of Theorem 9

Any online algorithm suffers an Ω( √ T ) Φ X Proj ( δ ) -regret even against linear loss functions by Theorem 7. This lower bound, however, holds only in the adversarial setting. In this section, we show an improved O ( T 1 4 ) individual Φ X Proj ( δ ) -regret bound under a slightly stronger smoothness assumption (Assumption 2) in the game setting, where players interact with each other using the same algorithm, previous results show improved external regret [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a]. This assumption is naturally satisfied by finite normal-form games and is also made for results about concave games [Far+22a].

Assumption 2. For any player i ∈ [ n ] , the utility u i ( x ) satisfies ∥∇ x i u i ( x ) -∇ x i u i ( x ′ ) ∥ ≤ L ∥ x -x ′ ∥ for all x, x ′ ∈ X .

We study the Optimistic Gradient (OG) algorithm [RS13], an optimistic variant of GD that has been shown to have improved individual external regret guarantee in the game setting [Syr+15]. The OG algorithm initializes w 0 ∈ X arbitrarily and g 0 = 0 . In each step t ≥ 1 , the algorithm plays x t , receives feedback g t := ∇ f t ( x t ) , and updates w t , as follows:

We show that OG has O ( T ) Φ X Proj ( δ ) -regret in the adversarial setting and fast O ( T 1 / 4 ) Φ X Proj ( δ ) -regret and convergence to approximate Φ X Proj ( δ ) -equilibrium in games.

Theorem 8 (Adversarial Regret Bound for OG) . Let δ > 0 and T ∈ N . For convex and G -Lipschitz loss functions { f t : X → ❘ } t ∈ [ T ] , the Φ X Proj ( δ ) -regret of (OG) with step size η > 0 is

Choosing step size η = √ δD X 2 G √ T , we have Reg T Proj ,δ ≤ 4 G √ δD X T .

Theorem 9 (Improved Individual Φ X Proj ( δ ) -Regret of OG in the Game Setting) . In a G -Lipschitz L -smooth (in the sense of Assumption 2) game, when all players employ OG with step size η > 0 , then for each player i , δ > 0 , and T ≥ 1 , their individual Φ X i Proj ( δ ) -regret denoted as Reg T,i Proj ,δ is Reg T,i Proj ,δ ≤ δD η + ηG 2 +3 nL 2 G 2 η 3 T. Choosing η = min { ( δD/ ( nL 2 G 2 T )) 1 4 , ( δD ) 1 2 /G } , we have Reg T,i Proj ,δ ≤ 4( δD ) 3 4 ( nL 2 G 2 T ) 1 4 + 2 √ δDG . Furthermore, for any δ > 0 and any ε > 0 , their empirical distribution of played strategy profiles converges to an ( ε + δ 2 L 2 ) -approximate Φ Proj ( δ ) -equilibrium in O (1 /ε 4 3 ) iterations.

E.4.1 Proof of Theorem 8

Proof. Fix any deviation v that is bounded by δ . Let us define p 0 = w 0 and p t = Π X [ x t -v ] . Following standard analysis of OG [RS13], we have

Now we apply a similar analysis from Theorem 3 to upper bound the term ∥ w t -1 -p t ∥ 2 -∥ w t -1 -p t -1 ∥ 2 :

where in the last-inequality we use ⟨ p t -1 -p t , x t -u -p t ⟩ ≤ 0 since p t = Π X [ x t -v ] and X is a compact convex set; we also use 2 ⟨ a, b ⟩ -b 2 ≤ a 2 . In the analysis above, unlike the analysis of GD where we drop the negative term -∥ p t -p t -1 ∥ 2 , we use -∥ p t -p t -1 ∥ 2 to get a term ∥ x t -w t -1 ∥ 2 which can be canceled by the last term in (3).

Now we combine the above two inequalities. Since the term ∥ x t -w t -1 ∥ 2 cancels out and 2 ⟨ p t -1 -p t , v ⟩ telescopes, we get

E.4.2 Proof of Theorem 9

In the analysis of Theorem 8 for the adversarial setting, the term ∥ g t -g t -1 ∥ 2 can be as large as 4 G 2 . In the game setting where every player i employs OG, g t i ,i.e., -∇ x i u i ( x ) , depends on other players' action x t -i . Note that the change of the players' actions ∥ x t -x t -1 ∥ 2 is only O ( η 2 ) . Such stability of the updates leads to an improved upper bound on ∥ g t i -g t -1 i ∥ 2 and hence also an improved O ( T 1 4 ) Φ X Proj ( δ ) -regret for the player.

Proof. Let us fix any player i ∈ [ n ] in the smooth game. In every step t , player i 's loss function f t : X i → ❘ is ⟨-∇ x i u i ( x t ) , ·⟩ determined by their utility function u i and all players' actions x t . Therefore, their gradient feedback is g t = -∇ x i u i ( x t ) . For all t ≥ 2 , we have

where we use L -smoothness of the utility function u i in the first inequality; we use the update rule of OG and the fact that gradients are bounded by G in the last inequality.

Applying the above inequality to the regret bound obtained in Theorem 8, the individual Φ X Proj ( δ ) -regret of player i is upper bounded by

Choosing η = min { ( δD/ ( nL 2 G 2 T )) 1 4 , ( δD ) 1 2 /G } , we have Reg T,i Proj ,δ ≤ 4( δD ) 3 4 ( nL 2 G 2 T ) 1 4 + 2 √ δDG . Using Lemma 1, we have the empirical distribution of played strategy profiles converge to an ( ε + δ 2 L 2 ) -approximate Φ Proj ( δ )) -equilibrium in O (1 /ε 4 3 ) iterations.

F Missing Details in Section 4.2

Algorithm 3: Conv(Φ) -regret minimization for Lipschitz smooth non-concave rewards

Input: x 1 ∈ X , K ≥ 2 , a no-external-regret algorithm R Φ against linear reward over ∆(Φ)

Output: A Conv(Φ) -regret minimization algorithm over X

1 function NEXTSTRATEGY()

2

p t ← R Φ .NEXTSTRATEGY(). Note that p t is a distribution over Φ .

3

4

x

k

ϕ

return

p

x

t

t

(

x

k

-

1

)

, for all

2

k

K

uniformly at random from

5 function OBSERVEREWARD ( ∇ x u t ( x t ))

6

a linear reward over

7

u

t

Φ

(

·

)

R

Φ

.OBSERVEREWARD(

u

(

·

F.1 Proof of Theorem 4

Proof Sketch. We adopt the framework in [SL07; GGM08] (as described in Section 3) with two main modifications. First, we utilize the L -smoothness of the utilities to transform the problem of external regret over ∆(Φ) against non-concave rewards into a linear optimization problem. Second, we use the technique of 'fixed point in expectation" [Zha+24] to circumvent the intractable problem of finding a fixed point.

Proof. For a sequence of strategies { x t } t ∈ [ T ] , its Conv(Φ) -regret is

t

Φ

∆(Φ)

)

).

with

{

x

1

, . . . , x

t

Φ

(

ϕ

) =

u

K

}

⟨∇

.

x

t

u

(

t

x

)

, ϕ

(

x

t

)

-

x

t

⟩

for all

ϕ

∈

Φ

.

Bounding External Regret over ∆(Φ) We can define a new reward function f t ( p ) := u t ( ϕ p ( x t )) over p ∈ ∆(Φ) . Since u t is non-concave, the reward f t is also non-concave and it is computational intractable to minimize external regret. We use locality to avoid computational barrier. Here we use the fact that Φ = Φ( δ ) contains only δ -local strategy modifications. Then by L -smoothness of u t , we know for any p ∈ ∆(Φ)

Thus we can approximate the non-concave optimization problem by a linear optimization problem over ∆(Φ) with only second-order error δ 2 L 2 . Here we use the notation a = b ± c to mean b -c ≤ a ≤ b + c .

We can then instantiate the external regret R Φ as the Hedge algorithm over reward f t ( p ) = ∑ ϕ ∈ Φ p ( ϕ ) ⟨∇ u t ( x t ) , ϕ ( x t ) -x t ⟩ and get

where we use the fact that ⟨∇ u t ( x t ) , ϕ ( x t ) -x t ⟩ ≤ ∥∇ u t ( x t ) ∥ · ∥ ϕ ( x t ) -x t ∥ ≤ Gδ .

Bounding error due to sampling from a fixed point in expectation We choose x 1 as an arbitrary point in X . Then we recursively apply ϕ p t to get

We denote µ t = Uniform { x k : 1 ≤ k ≤ K } . Then the strategy x t ∼ µ t is sampled from µ t . We have that µ t is an approximate fixed-point in expectation / stationary distribution in the sense that

Thanks to the boundedness of u t , we can use Hoeffding-Azuma's inequality to conclude that

for any ε > 0 . Combining the above with ε = √ 8 T log(1 /β ) and K = T , we get with probability at least 1 -β ,

Convergence to Φ -equilibrium If all players in a non-concave continuous game employ Algorithm 1, then we know for each player i , with probability 1 -β n , its Φ X i -regret is upper bounded by

By a union bound over all n players, we get with probability 1 -β , every player i 's Φ X i -regret is upper bounded by 8 √ T ( Gδ √ log | Φ + X i | + √ log( n/β )) + δ 2 LT . Now by Theorem 1, we know the empirical distribution of strategy profiles played forms an ( ε + δ 2 L ) -approximate Φ = Π n i =1 Φ X i -equilibrium, as long as T ≥ 128( G 2 δ 2 log | Φ X i | +log( n/β )) ε 2 iterations.

G Missing details in Section 4.3

We introduce a natural set of local strategy modifications and the corresponding local equilibrium notion. Given any set of (possibly non-local) strategy modifications Ψ = { ψ : X → X} , we define a set of local strategy modifications as follows: for δ ≤ D X and λ ∈ [0 , 1] , each strategy modification ϕ λ,ψ interpolates the input strategy x with the modified strategy ψ ( x ) : formally,

Note that for any ψ ∈ Ψ and λ ≤ δ D X , we have ∥ ϕ λ,ψ ( x ) -x ∥ = λ ∥ x -ψ ( x ) ∥ ≤ δ , respecting the locality constraint. The induced Φ X Int , Ψ ( δ ) -regret can be written as Reg T Int , Ψ ,δ := max ψ ∈ Ψ ,λ ≤ δ D X ∑ T t =1 ( f t ( x t ) -f t ((1 -λ ) x t + λψ ( x t ))) . We now define the corresponding Φ Int , Ψ ( δ ) -equilibrium.

Definition 9. Define Φ Int , Ψ ( δ ) = Π n j =1 Φ X j Int , Ψ j ( δ ) . In a continuous game, a distribution σ over strategy profiles is an ( ε -approximate Φ Int , Ψ ( δ ) -equilibrium if and only if for all player i ∈ [ n ] ,

Intuitively speaking, when a correlation device recommends strategies to players according to an ε -approximate Φ Int , Ψ ( δ ) -equilibrium, no player can increase their utility by more than ε through a local deviation by interpolating with a (possibly global) strategy modification ψ ∈ Ψ . The richness of Ψ determines the incentive guarantee provided by an ε -approximate Φ Int , Ψ ( δ ) -equilibriumas well as its computational complexity. When we choose Ψ to be the set of all possible strategy modifications, the corresponding notion of local equilibrium-limiting the gain of a player by interpolating with any strategy-resembles that of a correlated equilibrium .

Computation of ε -approximate Φ Int , Ψ ( δ ) -Equilibrium. By Lemma 1, we know computing an ε -approximate Φ Int , Ψ ( δ ) -equilibrium reduces to minimizing Φ X Int , Ψ ( δ ) -regret against convex loss functions. We show that minimizing Φ X Int , Ψ ( δ ) -regret against convex loss functions further reduces to Ψ -regret minimization against linear loss functions.

Theorem 10. Let A be an algorithm with Ψ -regret Reg T Ψ ( G,D X ) for linear and G -Lipschitz loss functions over X . Then, for any δ > 0 , the Φ X Int , Ψ ( δ ) -regret of A for convex and G -Lipschitz loss functions over X is at most δ D X · [Reg T Ψ ( G,D X )] + .

Proof. By definition and convexity of f t , we get

𝑥

𝜙

!"#$,&

(𝑥)

𝜙

'()*,&

(𝑥)

-𝑣

Note that when f t is linear, the reduction is without loss. Thus, any worst-case Ω( r ( T )) -lower bound for Ψ -regret implies a Ω( δ D X · r ( T )) lower bound for Φ Int , Ψ ( δ ) -regret. Moreover, for any set Ψ that admits efficient Ψ -regret minimization algorithms such as swap transformations over the simplex and more generally any set such that (i) all modifications in the set can be represented as linear transformations in some finite-dimensional space and (ii) fixed point computation can be carried out efficiently for any linear transformations [GGM08], we also get an efficient algorithm for computing an ε -approximate Φ Int , Ψ ( δ ) -equilibrium in the first-order stationary regime.

CCE-like Instantiation In the special case where Ψ contains only constant strategy modifications (i.e. ψ ( x ) = x ∗ for all x ), we get a coarse correlated equilibrium (CCE)-like instantiation of local equilibrium, which limits the gain by interpolating with any fixed strategy. We denote the resulting set of local strategy modification simply as Φ X Int . We can apply any no-external regret algorithm for efficient Φ X Int -regret minimization and computation of ε -approximate Φ Int ( δ ) -equilibrium in the first-order stationary regime as summarized in Theorem 5.

The above Φ X Int ( δ ) -regret bound of O ( √ T ) is derived for the adversarial setting. In the game setting, where each player employs the same algorithm, players may have substantially lower external regret [Syr+15; CP20; DFG21; Ana+22a; Ana+22b; Far+22a] but we need a slightly stronger smoothness assumption than Assumption 1. This assumption is naturally satisfied by finite normalform games and is also made for results about concave games [Far+22a]. Using Assumption 2 and Lemma 1, the no-regret learning dynamics of [Far+22a] that guarantees O (log T ) individual external regret in concave games can be applied to smooth non-concave games so that the individual Φ X Int ( δ ) -regret of each player is at most O (log T ) + δ 2 LT 2 . This gives an algorithm with faster ˜ O (1 /ε ) convergence to an ( ε + δ 2 L 2 ) -approximate Φ Int ( δ ) -equilibrium than GD.

H Beam-Search Local Strategy Modifications and Local Equilibria

In Section 4.1 and Section 4.3, we have shown that GD achieves near-optimal performance for both Φ X Int ( δ ) -regret and Φ X Proj ( δ ) -regret. In this section, we introduce another natural set of local strategy modifications, Φ X Beam ( δ ) , which is similar to Φ X Proj ( δ ) . Specifically, the set Φ X Beam ( δ ) contains deviations that try to move as far as possible in a fixed direction (see Figure 2 for an illustration of the difference between ϕ Beam ,v ( x ) and ϕ Proj ,v ( x ) ):

It is clear that ∥ ϕ Beam ,v ( x ) -x ∥ ≤ ∥ v ∥ ≤ δ . We can similarly derive the notion of Φ X Beam -regret and ( ε, Φ Beam ( δ )) -equilibrium. Surprisingly, we show that GD suffers linear Φ X Beam ( δ ) -regret (proof deferred to Appendix H.1).

Theorem 11. For any δ, η < 1 2 and T ≥ 1 , there exists a sequence of linear loss functions { f t : X ⊆ [0 , 1] 2 → ❘ } t ∈ [ T ] such that GD with step size η suffers Ω( δT ) Φ X Beam ( δ ) -regret.

H.1 Proof of Theorem 11

Let X ⊂ R 2 be a triangle region with vertices A = (0 , 0) , B = (1 , 1) , C = ( δ, 0) . Consider v = ( -δ, 0) . The initial point is x 1 = (0 , 0) .

The adversary will choose ℓ t adaptively so that x t remains on the boundary of X and cycles clockwise (i.e., A →··· → B →··· → C →··· → A →··· ). To achieve this, the adversary will repeat the following three phases:

Keep choosing ℓ t = u - - → BA ( u - - → BA denotes the unit vector in the direction of - - → BA ) until x t +1 reaches B .

Keep choosing ℓ t = u - - → CB until x t +1 reaches C .

Keep choosing ℓ t = u -→ AC until x t +1 reaches A .

In Phase 1, x t ∈ AB . By the choice of v = ( -δ, 0) , we have x t -ϕ v ( x t ) = ( -δ (1 -x t, 1 ) , 0) , and the instantaneous regret is δ (1 -x t, 1 ) √ 2 ≥ 0 .

In Phase 2, x t ∈ BC . By the choice of v = ( -δ, 0) , we have x t -ϕ v ( x t ) = (0 , 0) , and the instantaneous regret is 0 .

In Phase 3, x t ∈ CA . By the choice of v = ( -δ, 0) , we have x t -ϕ v ( x t ) = ( -δ + x t, 1 , 0) , and the instantaneous regret is -δ + x t, 1 ≤ 0 .

In each cycle, the number of rounds in Phase 1 is of order Θ( √ 2 η ) , the number of rounds in Phase 2 is between O ( 1 η ) and O ( √ 2 η ) , the number of rounds in Phase 3 is of order Θ( δ η ) .

Therefore, the cumulative regret in each cycle is roughly

On the other hand, the number of cycles is no less than T √ 2 η + √ 2 η + δ η = Θ( ηT ) . Overall, the cumulative regret is at least 0 . 5 δ -0 . 5 δ 2 × Θ( ηT ) = Θ( δT ) as long as δ < 0 . 5 .

η

I Hardness in the Global Regime

In the first-order stationary regime δ ≤ √ 2 ε/L , ( ε, δ ) -local Nash equilibrium is intractable, and we have shown polynomial-time algorithms for computing the weaker notions of ε -approximate Φ Int ( δ )) -equilibrium and ε -approximate Φ Proj ( δ )) -equilibrium. A natural question is whether correlation enables efficient computation of ε -approximate Φ( δ )) -equilibrium when δ is in the global regime, i.e., δ = Ω( √ d ) . In this section, we prove both computational hardness and a query complexity lower bound for both notions in the global regime

To prove the lower bound results, we only require a single-player game. The problem of computing an ε -approximate Φ( δ ) -equilibrium becomes: given scalars ε, δ, G, L > 0 and a polynomial-time Turing machine C f evaluating a G -Lipschitz and L -smooth function f : [0 , 1] d → [0 , 1] and its gradient ∇ f : [0 , 1] d → ❘ d , we are asked to output a distribution σ that is an ε -approximate Φ( δ ) -equilibrium or ⊥ if such equilibrium does not exist.

Hardness of finding ε -approximate Φ X Int ( δ ) -equilibria in the global regime When δ = √ d , which equals to the diameter D of [0 , 1] d , then the problem of finding an ε -approximate Φ X Int ( δ ) -equilibrium is equivalent to finding a ( ε, δ ) -local minimum of f : assume σ is an ε -approximate Φ X Int ( δ ) -equilibrium of f , then there exists x ∈ [0 , 1] d in the support of σ such that

Then hardness of finding an ε -approximate Φ X Int ( δ ) -equilibrium follows from hardness of finding a ( ε, δ ) -local minimum of f [DSZ21]. The following Theorem is a corollary of Theorem 10.3 and 10.4 in [DSZ21].

Theorem 12 (Hardness of finding ε -approximate Φ X Int ( δ ) -equilibria in the global regime) . In the worst case, the following two holds.

Computing an ε -approximate Φ X Int ( δ ) -equilibrium for a game on X = [0 , 1] d with G = √ d , L = d , ε ≤ 1 24 , δ = √ d is NP-hard.

Ω(2 d /d ) value/gradient queries are needed to determine an ε -approximate Φ X Int ( δ ) -equilibrium for a game on X = [0 , 1] d with G = Θ( d 15 ) , L = Θ( d 22 ) , ε < 1 , δ = √ d .

Hardness of finding ε -approximate Φ X Proj ( δ ) -equilibria in the global regime

Theorem 13 (Hardness of of finding ε -approximate Φ X Proj ( δ ) -equilibria in the global regime) . In the worst case, the following two holds.

Computing an ε -approximate Φ X Proj ( δ ) -equilibrium for a game on X = [0 , 1] d with G = Θ( d 15 ) , L = Θ( d 22 ) , ε < 1 , δ = √ d is NP-hard.

Ω(2 d /d ) value/gradient queries are needed to determine an ε -approximate Φ X Proj ( δ ) -equilibrium for a game on X = [0 , 1] d with G = Θ( d 15 ) , L = Θ( d 22 ) , ε < 1 , δ = √ d .

The hardness of computing ε -approximate Φ X Proj ( δ ) -equilibrium also implies a lower bound on Φ X Proj ( δ ) -regret in the global regime.

Corollary 2 (Lower bound of Φ X Proj ( δ ) -regret against non-convex functions) . In the worst case, the Φ X Proj ( δ ) -regret of any online algorithm is at least Ω(2 d /d, T ) even for loss functions f : [0 , 1] d → [0 , 1] with G,L = poly( d ) and δ = √ d .

The proofs of Theorem 13 and Corollary 2 can be found in the next two sections.

I.1 Proof of Theorem 13

We will reduce the problem of finding an ε -approximate Φ X Proj ( δ ) -equilibrium in smooth games to finding a satisfying assignment of a boolean function, which is NP-complete.

Fact 1. Given only black-box access to a boolean formula ϕ : { 0 , 1 } d → { 0 , 1 } , at least Ω(2 d ) queries are needed in order to determine whether ϕ admits a satisfying assignment x ∗ such that ϕ ( x ∗ ) = 1 . The term black-box access refers to the fact that the clauses of the formula are not given, and the only way to determine whether a specific boolean assignment is satisfying is by querying the specific binary string. Moreover, the problem of finding a satisfying assignment of a general boolean function is NP-hard.

We revisit the construction of the hard instance in the proof of [DSZ21, Theorem 10.4] and use its specific structures. Given black-box access to a boolean formula ϕ as described in Fact 1, following [DSZ21], we construct the function f ϕ ( x ) : [0 , 1] d → [0 , 1] as follows:

for each corner v ∈ V = { 0 , 1 } d of the [0 , 1] d hypercube, we set f ϕ ( x ) = 1 -ϕ ( x ) .

for the rest of the points x ∈ [0 , 1] d /V , we set f ϕ ( x ) = ∑ v ∈ V P v ( x ) · f ϕ ( v ) where P v ( x ) are non-negative coefficients defined in [DSZ21, Definition 8.9].

The function f ϕ satisfies the following properties:

if ϕ is not satisfiable, then f ϕ ( x ) = 1 for all x ∈ [0 , 1] d since f ϕ ( v ) = 1 for all v ∈ V ; if ϕ has a satisfying assignment v ∗ , then f ϕ ( v ∗ ) = 0 .

f ϕ is Θ( d 12 ) -Lipschitz and Θ( d 25 ) -smooth.

for any point x ∈ [0 , 1] d , the set V ( x ) := { v ∈ V : P v ( x ) = 0 } has cardinality at most d +1 while ∑ v ∈ V P v ( x ) = 1 ; any value / gradient query of f ϕ can be simulated by d +1 queries on ϕ .

̸

In the case there exists a satisfying argument v ∗ , then f ϕ ( v ∗ ) = 0 . Define the deviation e so that e [ i ] = 1 if v ∗ [ i ] = 0 and e [ i ] = -1 if v ∗ [ i ] = 1 . It is clear that ∥ e ∥ = √ d = δ . By properties of projection on [0 , 1] d , for any x ∈ [0 , 1] d , we have Π [0 , 1] n [ x -v ] = v ∗ . Then any ε -approximate Φ X Proj ( δ ) -equilibrium σ must include some x ∗ ∈ X with f ϕ ( x ∗ ) < 1 in the support, since ε < 1 . In case there exists an algorithm A that computes an ε -approximate Φ X Proj ( δ ) -equilibrium, A must have queried some x ∗ with f ϕ ( x ∗ ) < 1 . Since f ϕ ( x ∗ ) = ∑ v ∈ V ( x ∗ ) P v ( x ∗ ) f ϕ ( v ) < 1 , there exists ˆ v ∈ V ( x ∗ ) such that f ϕ (ˆ v ) = 0 . Since | V ( x ∗ ) | ≤ d +1 , it takes addition d +1 queries to find ˆ v with f ϕ (ˆ v ) = 0 . By Fact 1 and the fact that we can simulate every value/gradient query of f ϕ by d +1 queries on ϕ , A makes at least Ω(2 d /d ) value/gradient queries.

Suppose there exists an algorithm B that outputs an ε -approximate Φ X Proj ( δ ) -equilibrium σ in time T ( B ) for ε < 1 and δ = √ d . We construct another algorithm C for SAT that terminates in time T ( B ) · poly( d ) . C : (1) given a boolean formula ϕ , construct f ϕ as described above; (2) run B and get output σ (3) check the support of σ to find v ∈ { 0 , 1 } d such that f ϕ ( v ) = 0 ; (3) if finds v ∈ { 0 , 1 } d such that f ϕ ( v ) = 0 , then ϕ is satisfiable, otherwise ϕ is not satisfiable. Since we can evaluate f ϕ and ∇ f ϕ in poly( d ) time and the support of σ is smaller than T ( B ) , the algorithm C terminates in time O ( T ( B ) · poly( d )) . The above gives a polynomial time reduction from SAT to finding an ε -approximate Φ X Proj ( δ ) -equilibrium and proves the NP-hardness of the latter problem.

I.2 Proof of Corollary 2

Let ϕ : { 0 , 1 } d → { 0 , 1 } be a boolean formula and define f ϕ : [0 , 1] d → [0 , 1] the same as that in Theorem 13. We know f ϕ is Θ(poly( d )) -Lipschitz and Θ(poly( d )) -smooth. Now we let the adversary pick f ϕ each time. For any T ≤ O (2 d /d ) , in case there exists an online learning algorithm with Reg T Proj ,δ < T 2 , then σ := 1 T ∑ T t =1 1 x t is an ( 1 2 , δ ) -equilibrium. Applying Theorem 13 and the fact that in this case, Reg T Proj ,δ is non-decreasing with respect to T concludes the proof.

J Removing the D dependence for Φ X Proj -regret

For the regime δ ≤ D X which we are more interested in, the lower bound in Theorem 7 is Ω( Gδ √ T ) while the upper bound in Theorem 3 is O ( G √ δD X T ) . They are not tight especially when D X ≫ δ . A natural question is: which of them is the tight bound? We conjecture that the lower bound is tight. In fact, for the special case where the feasible set X is a box , we have a way to obtain a D X -independent bound O ( d 1 4 Gδ √ T ) , which is tight when d = 1 . Below, we first describe the improved strategy in 1 -dimension. Then we show how to extend it to the d -dimensional box setting.

J.1 One-Dimensional Case

In one-dimension, we assume that X = [ a, b ] for some b -a ≥ 2 δ (if b -a ≤ 2 δ , then our original bound in Theorem 3 is already of order Gδ √ T ). We first investigate the case where f t ( x ) is a linear function, i.e., f t ( x ) = g t x for some g t ∈ [ -G,G ] . The key idea is that we will only select x t from the two intervals [ a, a + δ ] and [ b -δ, b ] , and never play x t ∈ ( a + δ, b -δ ) . To achieve so, we concatenate these two intervals, and run an algorithm in this region whose diameter is only 2 δ . The key property we would like to show is that the regret is preserved in this modified problem.

More precisely, given the original feasible set X = [ a, b ] , we create a new feasible set Y = [ -δ, δ ] and apply our algorithm GD in this new feasible set. The loss function is kept as f t ( x ) = g t x . Whenever the algorithm for Y outputs y t ∈ [ -δ, 0] , we play x t = y t + a + δ in X ; whenever it outputs y t ∈ (0 , δ ] , we play x t = y t + b -δ . Below we show that the regret is the same in these two

problems. Notice that when y t ≤ 0 , we have for any v ∈ [ -δ, δ ] ,

Similarly, when y t > 0 , we can follow the same calculation and prove x t -Π X [ x t -v ] = y t -Π Y [ y t -v ] . Thus, the regret in the two problems:

are exactly the same for any v . Finally, observe that the diameter of Y is only of order O ( δ ) . Thus, the upper bound in Theorem 3 would give us an upper bound of O ( G √ δ · δT ) = O ( Gδ √ T ) .

For convex f t , we run the algorithm above with g t = ∇ f t ( x t ) . Then by convexity we have

so the regret in the modified problem (which is O ( Gδ √ T ) ) still serves as a regret upper bound for the original problem.

J.2 d -Dimensional Box Case

A d -dimensional box is of the form X = [ a 1 , b 1 ] × [ a 2 , b 2 ] ×··· × [ a d , b d ] . The box case is easy to deal with because we can decompose the regret into individual components in each dimension. Namely, we have

where we define X i = [ a i , b i ] , g t = ∇ f t ( x t ) , and use subscript i to indicate the i -th component of a vector. The last equality above is guaranteed by the box structure. This decomposition allows as to view the problem as d independent 1 -dimensional problems.

Nowwefollow the strategy described in Section J.1 to deal with individual dimensions (if b i -a i < 2 δ then we do not modify X i ; otherwise, we shrink X i to be of length 2 δ ). Applying the analysis of Theorem 3 to each dimension, we get

Choosing the optimal η = d 1 4 δ G √ T , we get the regret upper bound of order O ( d 1 4 Gδ √ T ) .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state the problem setting and our main contributions in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our results provide efficient uncoupled algorithms to compute ε -approximate Φ -equilibria in the first-order stationary regime. We leave the computational complexity of finding ε -approximate Φ -equilibria beyond the first-order stationary regime as an open question.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide assumptions and proofs for our theoretical results in the main body and the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not contain experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not contain experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not contain experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not contain experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not contain experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and the current paper conforms the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a purely theoretical paper and we do not see any immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a purely theoretical paper

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a purely theoretical paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This is a purely theoretical paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not contain experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not contain experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents new results about no-$\\Phi$-regret learning when the utilities are not concave. When $\\Phi$ is either finite or in a precise sense \"local\", it is shown that no-$\\Phi$-regret learning is possible, and hence that the corresponding notions of $\\Phi$-equilibria can be efficiently approximated at rate $\\text{poly}(1/\\epsilon)$.\n\nSTRENGTHS:\nThis presents some interesting and novel results in a regime (non-concave games) where positive results seem to be relatively difficult to come by. The paper was also very well written and easy to read. I vote to accept and only have some minor comments.\n\nWEAKNESSES:\nIt was useful to me to keep in mind that the regime $\\delta = O(\\epsilon)$ (where $O$ hides game/instance-dependent constants) is trivial, because no $\\delta$-local change can affect a Lipschitz utility function by more than $O(\\delta)$. It would be nice to explicitly state this somewhere.\n\nIn Table 1, it would be nice to distinguish the contributions of this paper from past contributions. Perhaps for each cell, include either the past citation or the theorem number in the current paper.\n\nMinor things and typos (not affecting score):\n* Table 1: effifient -> efficient\n* Missing close-parens: Algorithm 1, line 5; two in the botton half of page 18\n* Algorithm 2, line 3: form -> from\n\nQUESTIONS:\n1. Could you elaborate on Corollary 1? It is not obvious to me how that bound follows from Theorem 2. \n1. I find it interesting that the algorithms in this paper are presented as *randomized* algorithm outputting a *single* $x^t$ rather than a *determinisitic* algorithm outputting a *distribution* $\\mu^t \\in \\Delta(\\mathcal X)$ (as done by Peng and Rubinstein [2024], Dagan et al [2024], and Zhang et al [2024]). At least with your techniques, this difference seems to be an unavoidable consequence of the fact that the utilities $u^t$ are nonlinear, and therefore one cannot say $\\mathbb E_{\\phi \\sim p} u^t(\\phi(\\cdot)) = u^t(\\mathbb E_{\\phi \\sim p} \\phi(\\cdot))$. Do I understand this correctly?\n1. If I understood the previous point correctly: in Theorem 5, the nonlinearity of $u^t$ is a nonissue by assumption. So, is it possible to de-randomize that result by instead deterministically outputting the uniform distribution $\\mu^t := \\text{unif} \\\\{ x_1, \\dots, x_K \\\\}$?\n1. Is there an efficient algorithm for $\\Phi^\\mathcal{X}_\\text{Beam}(\\delta)$-regret minimization?\n1. Do the results of this paper have any novel implications for interesting special cases, such as nonconvex-concave zero-sum games, or simply the classical setting of concave no-regret learning? For example the results of Section 4 seem to have interesting (new?) implications for local (mixed) Nash equilibria in nonconvex-nonconcave zero-sum games. \n\nPapers cited here but not in the submission:\n* Dagan, Y., Daskalakis, C., Fishelson, M., & Golowich, N. (*STOC* 2024). From External to Swap Regret 2.0: An Efficient Reduction for Large Action Spaces.\n* Peng, B., & Rubinstein, A. (*STOC* 2024). Fast swap regret minimization and applications to approximate correlated equilibria.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a concept of $\\Phi$-equilibrium in non-concave games, which is a rarely discussed notion in the recent literature. This notion of equilibrium is defined over a set of strategy modifications, with specific definitions of $\\Phi$, $\\Phi$-equilibrium can recover commonly known notions such as CCEs, though this paper mostly focuses on more limiting choices of $\\Phi$. The paper shows that there exists efficient algorithms to find an $\\epsilon$-approximate $\\Phi$-equilibrium in tractable time for some specific families of set $\\Phi$.\n\nSTRENGTHS:\nAlthough some assumptions on families of set $\\Phi$ seem restrictive, it is known that without any assumptions, the task of finding e.g. CCE, even approximately and locally, is intractable. This paper provides a solid direction towards better understanding of algorithms in non-convex games. \n\nThis paper is quite self-contained, and section 1 explained the problem formulation such that the reviewer find easy to follow.\n\n\nFor the infinitely large $\\Phi$, the three families offered good insight and can potentially cover many existing strategy modification schemes in application.\n\nWEAKNESSES:\nThe structure of this paper is questionable. Section 1 alone takes 4 pages, and the remainder of this paper seems rushed and/or crammed. The authors did address the space constraint of this paper, but I believe the paper can be greatly improved in terms of better readability.\nFor instance, I believe it would be beneficial to address algorithm 1 in the main paper. And more space could be assigned to the finite $\\Phi$ setting, as a warm-up. The reviewer suggests that some paragraphs or even theorems in section 4 could be moved to the appendix.\n\nSome statements throughout the paper appear to be slightly repetitive and reiterates of previous sections, many statements can be shortened and more concise. For example, section 1.1 interweaves between introduction and contributions.\n\nAlthough this paper is quite high-level and technical, it would be reasonable to consider some empirical results to validate and strengthen the theoretical results such as the realistic complexity for the sampling procedure. This can perhaps be added in the appendix.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nThe authors adequately address the limitations of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the $\\Phi$-equlibrium in non-concave games and discuss when the $\\Phi$-equlibrium of a non-concave game can be learned in polynomial time. The theoretical results presented in this paper indicate that if the set $\\Phi$ is finite, then there exists an efficient uncoupled algorithm that converges to the corresponding  $\\Phi$-equilibria. Moreover, this work also considers the case when $\\Phi$ is not finite but consist of local modifications.\n\nSTRENGTHS:\nSignificance: This paper gives a non-trivial results on the $\\Phi$-equilibrium of non-concave games. It provides a simple condition to tell if $\\Phi$-equilbria can be learned in polynomial iterations. \n\nOriginality: There are no existing literature that have resolved the $\\Phi$-equilibrium problem. So, the result of this paper is new. This paper extends existing framework to consider infinite local strategy modifications, which also leads to new technical novelty. \n\nClarity: This work includes sufficient backgroun introduction and put adequate materials in the appendix for references. I feel easy to understand the main contribution of this work.\n\nWEAKNESSES:\nI believe this is a high quality paper and I give a clear accept. But I do have some confused points. I put them in the next sections.\n\nQUESTIONS:\n1. I find some papers showing that for Markov games, the CE or CCE can be learned in polynomial time. For example: *Jin, Chi, et al. \"V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL.\" arXiv preprint arXiv:2110.14555 (2021).* From my understanding, when setting their horizon $H=1$, the Markov game seems to be a classical game. Is there any difference between this setting and yours? \n\n2. Is there any applications of $\\Phi$-equilibrium? For example, when do we need to efficiently solve some  $\\Phi$-equilibrium? \n\n3. The definition of $\\Phi$-equilibrium looks really similar to the Correlated Equilibrium (CE) in your Definition 6. Is CE as the same as $\\Phi$-equilibrium if $\\Phi$ is choosen as the whole space?\n\nLIMITATIONS:\nThis is a purely theoretical work so there is no any negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper studies nonconcave games and $\Phi$-equilibrium. When $\Phi$ is finite, the paper showed that there exists an uncoupled learning algorithm that can efficiently find the equilibria. Under certain classes of strategy modification, online gradient descent can also approximate $\Phi$-equilibria when $\Phi$ is infinite.

STRENGTHS:
I am not an expert in non-concave games and $\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses

WEAKNESSES:
I am not an expert in non-concave games and $\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses

QUESTIONS:
I am not an expert in non-concave games and $\Phi$ equilibria. Thus I suggest referring to other reviewer for the strength and weaknesses

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12