Title: Uncoupled and Convergent Learning in Monotone Games under Bandit Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: Game theory

online learning
Abstract: We study the problem of no-regret learning algorithms for general monotone and smooth games and their last-iterate convergence properties. Specifically, we investigate the problem under bandit feedback and strongly uncoupled dynamics, which allows modular development of the multi-player system that applies to a wide range of real applications. We propose a mirror-descent-based algorithm, which converges in $O(T^{-1/4})$ and is also no-regret. The result is achieved by a dedicated use of two regularizations and the analysis of the fixed point thereof. The convergence rate is further improved to $O(T^{-1/2})$ in the case of strongly monotone games.
Motivated by practical tasks where the game evolves over time, the algorithm is extended to time-varying monotone games. We provide the first non-asymptotic result in converging monotone games and give improved results for equilibrium tracking games.

Uncoupled and Convergent Learning in Monotone Games under Bandit Feedback

Abstract

We study the problem of no-regret learning algorithms for general monotone and smooth games and their last-iterate convergence properties. Specifically, we investigate the problem under bandit feedback and strongly uncoupled dynamics, which allows modular development of the multi-player system that applies to a wide range of real applications. We propose a mirror-descent-based algorithm, which converges in O ( T -1 / 4 ) under Bregman divergence and is also no-regret, where the choice of Bregman divergence is determined by the convexity of the game. The result is achieved by the dedicated use of two regularizations and the analysis of the fixed point thereof. The convergence rate is further improved to O ( T -1 / 2 ) in the case of strongly monotone games. Motivated by practical tasks where the game evolves over time, the algorithm is extended to time-varying monotone games. We provide the first non-asymptotic result in converging monotone games and give improved results for equilibrium tracking games.

1 Introduction

We consider multi-player online learning in games. In this problem, the cost function for each player is unknown to the player, and they need to learn to play the game through repeated interaction with other players. We focus on a class of monotone and smooth games, which was first introduced by Rosen (1965). This encapsulates a wide array of common games, such as two-player zero-sum games, convex-concave games, and zero-sum polymatrix games (Bregman and Fokin, 1987). Our goal is to find algorithms that solve the problem under bandit feedback and strongly uncoupled dynamics. Within this context, each player can only access information regarding the cost function associated with their chosen actions without prior insight into their counterparts. This allows modular development of the multi-player system in real applications and leverages existing single-agent learning algorithms for reuse.

Many works have focused on the time-average convergence to a Nash equilibrium on learning in monotone games (Even-Dar et al., 2009; Syrgkanis et al., 2015; Farina et al., 2022). However, these works only guarantee the convergence of the time average of the joint action profile. Such

convergence properties are less appealing because while the trajectories of the players converge in the time-average sense, they may still exhibit cycling (Mertikopoulos et al., 2018). This jeopardizes the practical use of such algorithms.

Popular no-regret algorithms such as mirror descent have demonstrated convergence in the last iterate within specific scenarios, such as two-player zero-sum games (Cai et al., 2023) and strongly monotone games (Bravo et al., 2018; Drusvyatskiy et al., 2022; Lin et al., 2021). Yet convergence to a Nash equilibrium in monotone and smooth games is not available unless one assumes exact gradient feedback and coordination of players (Cai et al., 2022; Cai and Zheng, 2023). It remains open as to whether a no-regret algorithm can efficiently converge to a Nash equilibrium in monotone games with bandit feedback and strongly uncoupled dynamics. In this paper, we investigate the pivotal question:

How fast can no-regret algorithms converge (in the last iterate) to a Nash equilibrium in general monotone and smooth games with bandit feedback and strongly uncoupled dynamics?

In this work, we present a mirror-descent-based algorithm designed to converge to a Nash equilibrium in monotone and smooth games. Our algorithm is uncoupled, convergent and applicable to the general monotone and smooth game setting. Motivated by real applications, where many games are also time-varying, we extend our study to encompass time-varying monotone games. This justifies that our algorithm could be deployed in both stationary and non-stationary tasks.

We achieve state-of-the-art results in both monotone games and time-varying monotone games.

In monotone and smooth games:

-Under bandit feedback and strongly uncoupled dynamics, we show our algorithm achieves a last-iterate convergence rate of O ( T -1 / 4 ) .

-In cases where the game exhibits strong monotonicity, our result improves to O ( T -1 / 2 ) , matching the current best available convergence rates for strongly monotone games (Drusvyatskiy et al., 2022; Lin et al., 2021).

-Our algorithm is no regret, albeit players may be self-interested. The individual regret is at most O ( T 3 / 4 ) in monotone games and at most O ( T 1 / 2 ) in strongly monotone games.

In time-varying monotone and smooth games:

-If the game eventually converges to a static state within a time frame of O ( T α ) , our algorithm achieves convergence in O ( T -1 / 4+ α ) .

-If the game does not converge but experiences gradual changes in a Nash equilibrium that evolves in O ( T ϕ ) , our algorithm exhibits convergence rates of O ( max { T 2 ϕ / 3 -2 / 3 , T (4 ϕ +5) 2 / 72 -9 / 8 }) . The algorithm outperforms best available results of T ϕ / 5 -1 / 5 by Duvocelle et al. (2023) and T ϕ / 3 -2 / 3 by Yan et al. (2023).

Table 1 and Table 2 summarize our results and the results of previous works. We remark that, compared to previous results, which are typically obtained with the /lscript 2 norm or gap function being the convergence metric, we use the Bregman divergence induced by the algorithm's regularizer to measure the convergence. A suitable regularizer is chosen based on the curvature of the game, and the choice of our algorithm's regularizer will affect the meaning of convergence. We remark that the different convergence notions do not imply each other in general. It is only in the special case of a strongly monotone game that our results can be obtained with convergence in /lscript 2 norm, and convergence in the gap function implies convergence in /lscript 2 norm if the equilibrium is unique.

2 Related Works

Monotone games. The convergence of monotone games has been studied in a significant line of research. For a strongly monotone game under exact gradient feedback, the linear last-iterate convergence rate is known (Tseng, 1995; Liang and Stokes, 2019; Zhou et al., 2020). Under noisy gradient feedback, Jordan et al. (2023) showed a last-iterate convergence rate of O ( T -1 ) . Under bandit feedback, Bervoets et al. (2020) proposed an algorithm that asymptotically converges to the equilibrium if it is unique. Bravo et al. (2018) subsequently introduced an algorithm with a last-iterate convergence rate of O ( T -1 / 3 ) , while also ensuring the no-regret property. Later works (Lin et al.,

2021) further improved the last-iterate convergence rate to O ( T -1 / 2 ) under bandit feedback using the self-concordant barrier function. Jordan et al. (2023) gave a result of the same rate, but with the additional assumption that the Jacobian of each player's gradient is Lipschitz continuous. In the case of bandit but noisy feedback (with a zero-mean noise), Lin et al. (2021) showed that the convergence rate is still O ( T -1 / 2 ) .

For monotone but not strongly monotone games, Mertikopoulos and Zhou (2019) leveraged the dual averaging algorithm to demonstrate an asymptotic convergence rate under noisy gradient feedback. With access to the exact gradient information, Cai and Zheng (2023) gave a last-iterate convergence rate of O ( T -1 ) . In the context of bandit feedback, Tatarenko and Kamgarpour (2019) proposed an algorithm that asymptotically converges to a Nash equilibrium.

Time-varying monotone games. Motivated by real-world applications such as Cournot competition, where multiple firms supply goods to the market, and pricing is subject to fluctuations due to factors like weather, holidays, and politics. Duvocelle et al. (2023) studied the strongly monotone game under a time-varying cost function. When the game converges to a static state, they propose an algorithm that achieves asymptotic convergence under bandit feedback. Assuming the cost function varies O ( T φ ) across a horizon T , Duvocelle et al. (2023) provided an algorithm that attains a convergence rate of O ( T φ / 5 -1 / 5 ) under bandit feedback. Subsequent work of Yan et al. (2023) further improved this rate to O ( T φ / 3 -2 / 3 ) under exact gradient feedback.

3 Preliminaries

We consider a multi-player game with n players, with the set of players denoted as N . Each player i takes action on a compact and convex set X i ⊆ R d of d dimensions, and has cost function c i ( x i , x -i ) , where x i ∈ X i is the action of the i -th player and x -i ∈ ∏ j ∈ [ n ] ,j = i X j is the action of all other players. We assume the radius of X i is bounded, i.e., ‖ x -x ′ ‖ ≤ B, ∀ x, x ′ ∈ X i . Without loss of generality, we further assume c i ( x ) ∈ [0 , 1] .

/negationslash

In this work, we study a class of monotone continuous games, where the gradient of the cost functions is monotone, and the cost functions are continuous (Assumption 3.1). Games that satisfy this assumption include convex-concave games, convex potential games, extensive form games, Cournot competition, and splittable routing games. A discussion of these games is available in Section 3.1. Note that the class of monotone continuous games is commonly studied in the literature (Lin et al., 2021; Farina et al., 2022).

Assumption 3.1. For all player i ∈ N , the cost function c i ( x i , x -i ) is continuous, differentiable, convex, and /lscript i -smooth in x i . Further, c i has bounded gradient | ∇ i c i ( x ) | ≤ G and the gradient F ( x ) = [ ∇ i c i ( x )] i ∈ N is a monotone operator, i.e., ( F ( x ) -F ( y )) /latticetop ( x -y ) ≥ 0 , ∀ x, y .

A common solution concept in the game is the Nash equilibrium, which is a state of dynamics where no player can reduce its cost by unilaterally changing its action. Our aim is to learn a Nash equilibrium x ∗ ∈ ∏ i X i of the game. Formally, the Nash equilibrium is defined as follows.

For notational convenience, we denote L = ∑ i ∈ N /lscript i .

Definition 3.1 (Nash equilibrium) . An action x ∗ ∈ ∏ i X i is a Nash equilibrium if c i ( x ∗ ) ≤ c i ( x i , x ∗ -i ) , ∀ x i ∈ X i , i ∈ N .

When the game satisfies Assumption 3.1, and is with a compact action set, it is known that it must admit at least one Nash equilibrium (Debreu, 1952).

3.1 Examples of Monotone Continuous Games

A wide range of monotone games are captured by Assumption 3.1, and we now present a few classic examples. We include more examples in the appendix.

Example 3.1 (convex-concave game) . Consider a two-player convex-concave game, where the objective function is c 1 ( x 1 , x 2 ) = f ( x 1 , x 2 ) , c 2 ( x 1 , x 2 ) = -f ( x 1 , x 2 ) . It is immediate that if f is continuous, differentiable, smooth, convex in x 1 , concave in x 2 , then the game satisfies Assumption 3.1. Examples are rock, paper, scissors, and chicken games.

Example 3.2 (Cournot competition) . In the Cournot oligopoly model, there is a finite set of N firms, where firm i supplies the market with a quantity x i ∈ [0 , C i ] of some good and C i is the firm's production capacity. The good is priced as a decreasing function P ( x tot ) = a -bx tot , where

x tot = ∑ N i =1 x i is the total number of goods supplied to the market, and a, b > 0 are positive constants. The cost of firm i is then given by c i ( x i , x -i ) = d i x i -x i P ( x tot ) , where d i is the cost of producing one unit of good. This is the associated production cost minus the total revenue from producing x i units of goods. It is clear that c i is continuous and differentiable, and Bravo et al. (2018) showed that c i has a positive definite and bounded Hessian (i.e., convex and smooth).

Example 3.3 (Splittable routing game) . In a splittable routing game, each player directs a flow, denoted as f i , from a source to a destination in an undirected graph G = ( V, E ) . Each edge e ∈ E is linked to a latency function, represented as /lscript e ( f ) , which denotes the latency cost of the flow passing through the edge. The strategies available to player i are the various ways of dividing or "splitting" the flow f i into distinct paths connecting the source and the destination. With some restrictions on the latency function, the game satisfies Assumption 3.1 (Roughgarden and Schoppmann, 2015).

3.2 Bandit Feedback and Strongly Uncoupled Dynamic

In this work, we focus on learning under bandit feedback and strongly uncoupled dynamics. The bandit feedback setting restricts each player to only observe the cost function c i ( x i , x -i ) with respect to the action taken x i . The strongly uncoupled learning dynamic (Daskalakis et al., 2011) means players do not have prior knowledge of the cost function or the action space of other players and can only keep track of a constant amount of historical information. As the bandit feedback and strongly uncoupled dynamic only require each player to access information of its own, this allows for modular development of the multi-player system, by reusing existing single-agent learning algorithms.

4 Algorithm

Our algorithm builds upon the renowned mirror-descent algorithm. The mirror descent algorithm generalizes the gradient descent by extending first-order optimization to constrained decision spaces by leveraging Bregman divergences for updates. For an agent with decision set X ⊆ R d , mirror map h : X → R (a strictly convex, differentiable function), and step size η t > 0 at iteration t , the mirror descent update takes the form x t +1 = arg min x ∈ X { η t 〈∇ f t ( x t ) , x 〉 + D p ( x, x t ) } , where f t : X → R denotes the agent's loss (or negative payoff) at iteration t , and D p ( x, x t ) = p ( x ) -p ( x t ) -〈∇ p ( x t ) , x -x t 〉 is the Bregman divergence induced by p . We also remark that each player may choose their own p i , instead of agreeing on the same p , and thus the overall dynamic is still uncoupled. The efficacy of online mirror-descent in finding a Nash equilibrium has been demonstrated under full information, and in both linear and strongly monotone games, with extensive investigations into its last-iterate convergence investigated in Cen et al. (2021); Lin et al. (2021); Cai et al. (2023); Duvocelle et al. (2023).

Our algorithm differs from classic online mirror descent approaches by making use of two regularizers: A self-concordant barrier regularizer h to build an efficient Ellipsoidal gradient estimator and contest the bandit feedback; and a regularizer p to accommodate monotone (and not strongly monotone) games. Similar use of two regularizers has also been investigated (Lin et al., 2021). However, their method used the Euclidean norm regularization, which cannot be extended to our setting.

Regularizers. Let h be a ν -self-concordant barrier function (Definition 4.1), p be a convex function with µI /precedesequal ∇ 2 p ( x ) /precedesequal ζ I , ζ > 0 , µ ≥ 0 . Let D p denote the Bregman divergence induced by p . We choose p such that for any x i , x ′ i ∈ X i , D p ( x i , x ′ i ) ≤ C p < ∞ , and for some κ > 0 , c i ( x i , x -i ) -κ p ( x i ) is convex. Note that when c i is convex but not linear, we can always find such p when the action set is bounded. Intuitively, this is to interpolate a function p that possesses less curvature than all c i . We will discuss the necessary modification to the algorithm when c i is linear in Section 5.3.

Definition 4.1. A function h : int ( X ) ↦→ R is a ν -self concordant barrier for a closed convex set X ⊆ R n , where int ( X ) is an interior of X , if 1) h is three times continuously differentiable; 2) h ( x ) → ∞ if x → ∂ X , where ∂ X is a boundary of X ; 3) for ∀ x ∈ int( X ) and ∀ λ ∈ R n , we have ∣ ∣ ∇ 3 h ( x )[ λ , λ , λ ] ∣ ∣ ≤ 2 ( λ /latticetop ∇ 2 h ( x ) λ ) 3 / 2 and ∣ ∣ ∇ h ( x ) /latticetop λ ∣ ∣ ≤ √ ν ( λ /latticetop ∇ 2 h ( x ) λ ) 1 / 2 where ∇ 3 h ( x ) [ λ 1 , λ 2 , λ 3 ] = ∂ 3 ∂ t 1 ∂ t 2 ∂ t 3 h ( x + t 1 λ 1 + t 2 λ 2 + t 3 λ 3 ) ∣ ∣ ∣ t 1 = t 2 = t 3 =0 .

It is shown that any closed convex domain of R d has a self-concordant barrier (Lee and Yue, 2021). We also remark that adding a linear function to h does not change the self-concordant property, as the third derivative of any linear function is zero; therefore, the addition of the linear term does not affect the third derivative of sum of the functions.

Ellipsoidal gradient estimator. As our algorithm operates under bandit feedback and strongly uncoupled dynamics, we would need to design a gradient estimator while only using costs for the individual player.

Let S d , B d be the d -dimensional unit sphere and the d -dimensional unit ball, respectively. Our algorithm estimates the gradient using the following ellipsoidal estimator:

where z t i is uniformly independently sampled from S d (before receiving the feedback) and δ t , η t ∈ [0 , 1] are tunable parameters.

/negationslash

One can show that ˆ g t i is an unbiased estimate of the gradient of a smoothed cost function ˆ c i ( x t ) = E w t i ∼ B d E z t -i ∼ Π j = i S d [ c i ( x t i + A t i w t i , ˆ x t -i )] . When p is strongly convex, one can upper bound ‖∇ i ˆ c i ( x ) -∇ i c i ( x ) ‖ by the maximum eigenvalue of A t i and it suffices to take δ t = 1 , which recovers the results in Lin et al. (2021). However, when p is convex and not strongly convex, one would need to carefully tune δ t to control the bias from estimating the smoothed cost function. This ellipsoidal gradient estimator was first introduced by Abernethy et al. (2008) for the case of c i being linear, and was then extended by Hazan and Levy (2014) to the case of strongly convex costs. In learning for games, the ellipsoidal estimator was used in the case of strongly monotone games (Bravo et al., 2018; Lin et al., 2021).

Based on the ellipsoidal gradient estimator, we present our uncoupled and convergent algorithm for monotone games under bandit feedback.

Implementation. Note that solving Equation (1) is equivalent to solving a convex but potentially non-smooth optimization problem. Certain sets X ⊆ R d , including the cases when X is the strategy space of a normal-form game or an extensive-form game, can be solved by the proximal Newton algorithm provably in O (log 2 (1 / /epsilon1 )) iterations (Farina et al., 2022). When such guarantees are not required, one could use other optimization methods to solve (1). Our experiment section provides more details.

The choice of p and h is game-dependent. For example, when c i ( x ) = x 2 and the action set is on the positive half line, we can use the negative log function as our self-concordant barrier function h and take p ( x ) = x -log(1 + x ) with κ ≤ 2 .

5 No-regret Convergence to a Nash equilibrium

In this section, we present our main results on the last-iterate convergence to a Nash equilibrium. We show that Algorithm 1 converges to a Nash equilibrium in monotone, strongly monotone, and linear games. Such convergence is no-regret, meaning that the individual regret of each player is sublinear.

For notational simplicity, we present the results in a perfect bandit feedback model, where player i observes exactly c i ( x t ) . The discussion of noisy bandit feedback, where player i observes c i ( x t ) + /epsilon1 t i , with /epsilon1 t i be a zero-mean noise, is deferred to the appendix (Theorem D.1).

5.1 Perfect Bandit Feedback

The following theorem describes the last-iterate convergence rate (in expectation) for convex and strongly convex loss under perfect bandit feedback.

inf ¯ x ∈ X ∗ ∑ i ∈ N D p (¯ x i , x i ) , where X ∗ is the set of Nash equilibrium. With Algorithm 1 and under Assumption 3.1, we have

We remark that if each player chooses different p i , we can obtain the same convergence guarantee under the metric E [∑ i ∈ N D p i ( X ∗ i , x T +1 i )] , as the analysis hold through decomposing p into p i .

In the case of monotone games, Bravo et al. (2018) showed an asymptotic convergence to a Nash equilibrium. To the best of our knowledge, Theorem 5.1 is the first result on the last-iterate convergence rate for monotone games. For strongly monotone games, Bravo et al. (2018) first gave a O ( T -1 / 3 ) last-iterate convergence rate, which was later improved to O ( T -1 / 2 ) by Lin et al. (2021).

We note that the choice of the regularizer p determines the convergence metrics D p ( · , · ) and therefore affects the meaning of convergence. In the most extreme case, where p is linear, such metrics can be vacuous. Therefore, it is important to choose an appropriate p to recover a Nash equilibrium. Under Assumption 3.1, this metric may imply an /epsilon1 -Nash equilibrium. While these measures are meaningful for comparison, we remark that they may not be directly comparable to the (total) gap function used in some prior works. Specifically, neither of the two measures upper bounds the other.

While we defer the proof to the appendix, we discuss the main ideas for deriving the results. By the update rule, we can obtain the inequality for an arbitrary ω i :

When the game is strongly monotone, we can directly use strongly monotonicity and take p to be the Euclidean norm to obtain a recursive relation similar to

We then sum the recursive inequality and leverage the combination of two regularizations, which gives us an upper bound on η T κ ( T +1) ∑ i ∈ N D p ( ω i , x T +1 i ) . Now it suffices to properly choose a suitable ω i such that both the first term ∑ i ∈ N D h ( ω i , x 1 i ) and the the cumulative sum of ∑ T t =1 ∑ i ∈ N η t 〈∇ i c i ( ω ) , ω i -x t i 〉 are bounded. When ω i is a Nash equilibrium x ∗ i , the later term can be upper bounded trivially using the monotonicity of c i , while it does not imply a bounded first term. Therefore, we set ω i = x ∗ i when the first term can be bounded. Otherwise, we set it to a close enough point to x ∗ i , such that the first term can be bounded and the later term is bounded through a more careful calculation.

This amounts to applying this recursion and upper-bounding the residual terms individually to obtain a last-iterate convergence. However, when the game is monotone but not strongly monotone, we will need a different approach. Notice that G = ∇ c i -∇ p is a monotone operator. Using the property of Bregman divergence, we have 〈 G ( x ) -G ( x ′ ) , x ′ -x 〉 ≤ -∑ i ∈ N ( D p ( x i , x ′ i ) + D p ( x ′ i , x i )) .

High probability result. In the case of a strongly monotone game, our results show that the O ( T -1 / 4 ) last-iterate convergence rate holds with high probability. This is the first high-probability result for last-iterate convergence in strongly monotone games.

Theorem 5.2. Define ∑ i ∈ N D p ( X ∗ i , x i ) as inf ¯ x ∈ X ∗ ∑ i ∈ N D p (¯ x i , x i ) , where X ∗ is the set of Nash equilibrium. With a probability of at least 1 -log( T ) δ , δ ≤ e -1 , and with Algorithm 1 and under Assumption 3.1, we have ∑ i ∈ N D p ( X ∗ i , x T +1 i ) ≤ O ( nd ν log( T ) κ √ T + nd ζ B T + nBL κ √ T + ndC p √ T + nd log( T ) κ √ T + dBL log( T ) κ µ √ T + nBd 2 log 2 (1 / δ ) log( T ) κ min { √ µ,µ } √ T ) .

5.2 Individual Low Regret

Beyond the fast convergence to a Nash equilibrium, our algorithm also ensures each player has a sublinear regret when playing against other players. The sublinear regret is a desirable property as the players could be self-interested in general, and want to ensure their return even when other players are not adhering to the protocol. The low regret property remains true for players that are potentially adversarial, despite the convergence to a Nash equilibrium no longer holds in that case.

For player i , and a sequence of actions { ˆ x t i } T t =1 , define the individual regret as the cumulative expected difference between the costs received and the cost of playing the hindsight optimal action. That is, ∑ T t =1 E [ c i ( ˆ x t i , x t -i ) -c i ( ω i , x t -i )] , where { x t -i } T t =1 is a fixed sequence of actions of other players. The following theorem shows a guarantee of the individual regret of each player.

sequence of { x t -i } T t =1 , and with Algorithm 1, we have

Our result matches the √ T regret bound for strongly monotone games (Lin et al., 2021), but applies to monotone games as well.

Implication on social welfare. By designing the algorithm to be no-regret, we can also show that the social welfare attained by the algorithm also converges to the optimal value.

The social welfare for a joint action x is defined as SW( x ) = ∑ i ∈ N c i ( x ) . We let OPT = min x SW( x ) to denote the optimal social welfare.

Definition 5.1 (Roughgarden 2015; Syrgkanis et al. 2015) . Fix C 1 > 0 , C 2 < 1 . A game is ( C 1 , C 2 ) -smooth if there exists a strategy x ′ , such that for any x ∈ ∏ i ∈ N X i , ∑ i ∈ N c i ( x ′ i , x -i ) ≤ C 1 · OPT+ C 2 · SW( x ) .

The following proposition confirms that the social welfare converges to optimum on average.

Proposition 5.1. With η t = 1 2 dt 3 / 4 , δ t = 1 t 1 / 4 , and suppose every player employ Algorithm 1, we have 1 T ∑ T t =1 E [SW(ˆ x )] = O ( C 1 OPT (1 -C 2 ) + n ν d log( T ) (1 -C 2 ) T 1 / 4 + √ nB ∑ i ∈ N /lscript i (1 -C 2 ) T 1 / 4 ) for a ( C 1 , C 2 ) -smooth game.

5.3 Special Case: Linear Cost Function

When c i is linear, there does not exist a p that is convex while making c i -κ p strictly convex. Algorithm 1 therefore does not apply to the linear case. This coincides with our intuition that the landscape c i does not provide enough curvature information for the algorithm to utilize.

To extend the algorithm to the linear case, we modify line 6 of Algorithm 1 as

The idea is to first show the convergence of x T to a game with the cost c i ( x ) + τ p ( x ) . With this regularized game, we choose p to be a strongly convex function and measure convergence in terms of the gap function 〈 c i ( x ) , x i -x ∗ 〉 . By carefully controlling τ , we obtain the following result.

Similar regularization techniques have been used in the analysis of the zero-sum game (Cen et al., 2021; Cai et al., 2023). Our result matches the last-iterate convergence for zero-sum matrix game (Cai et al., 2023), which is a class of games with linear cost functions. However, our result is more general as it applies to multi-player linear games with convex and compact action sets (while previous works only apply to a simplex action set). It remains open how games with linear cost functions could be effectively learned and whether the convergence rate could be improved.

6 Application to Time-varying Game

In this section, we further apply Algorithm 1 to games that evolve over time. A time-varying game G t is a game where the cost function c t i ( · ) , i ∈ N depends on t . The game G t is not revealed to the players before choosing their actions x t . We assume that G t satisfies Assumption 3.1 for every t .

Such evolving games have applications in Kelly's auction and power control, where the cost function may change as time-dependent values change, such as channel gains. While the changes of G t can be random, we discuss two cases here, 1) when G t converges to a static game G in o ( T ) time, and 2) when the variation path of some Nash equilibrium, ∑ T t =1 ‖ x t +1 , ∗ i -x t, ∗ i ‖ is bounded in o ( T ) .

Converging monotone game. Let G t denote the game formed by the costs { c t i ( · ) } i ∈ N , and G be the game formed by the costs { c i ( · ) } i ∈ N . Suppose G t converges to G , and let X ∗ be the set of Nash equilibrium of the game G . The cost function c t i converges to some cost function c i in o ( T ) time. The following theorem shows the last iterate convergence to x ∗ .

Theorem 6.1. With ∑ T t =1 ∑ i ∈ N max x ‖∇ i c i ( x ) -∇ i c t i ( x ) ‖ 2 = T α , take η t = 1 2 dt 3 / 4 , δ t = 1 t 1 / 4 . Define ∑ i ∈ N D p ( X ∗ i , x i ) as inf ¯ x ∈ X ∗ ∑ i ∈ N D p (¯ x i , x i ) , where X ∗ is the set of Nash equilibrium of game G . Under Algorithm 1 and under Assumption 3.1, we have E [∑ i ∈ N D p ( X ∗ i , x T +1 i )] ≤ O ( nd ν log( T ) κ T 1 / 4 + n ζ dB T 3 / 4 + nBL κ √ T + ndC p T 1 / 4 + nd log( T ) κ T 1 / 4 + √ nB 2 L log( T ) κ T 1 / 4 + B T 1 / 4 -α ) .

For monotone games, Duvocelle et al. (2023) showed an asymptotic last-iterate convergence rate. To the best of our knowledge, Theorem 6.1 is the first last-iterate convergence rate for the class of converging monotone game.

Evolving game and equilibrium tracking. We now discuss the case where G t does not necessarily converge to a game G , but the cumulative changes of some equilibrium are bounded. We use the variation path V i ( T ) = ∑ t ∈ [ T ] ∥ ∥ ∥ x t +1 , ∗ i -x t, ∗ i ∥ ∥ ∥ to track the cumulative changes of equilibrium. In this setting, the last-iterate convergence is not applicable, and the convergence is measured in terms of the average gap. Because of this, the algorithm is slightly modified and updates with x t +1 i = argmin x i ∈ X i { η t 〈 x i , ˆ g t i 〉 + D h ( x i , x t i ) } .

Theorem 6.2. Assume V i ( T ) ≤ T ϕ , ϕ ∈ [0 , 1] . Take η t = 1 2 d · t -1 -ϕ 3 , δ t = 1 t 1 / 2 . Under Algorithm 1 and under Assumption 3.1, we have 1 T ∑ T t =1 ∑ i ∈ N 〈 ∇ i c t i ( ˆ x t i , ˆ x t -i ) , ˆ x t i -x t, ∗ i 〉 = ˜ O ( n ν d + Ln 3 / 2 B 2 + nG T 2(1 -ϕ ) 3 + n T 9 8 -(4 ϕ +5) 2 72 ) .

In the case of a strongly monotone game, Duvocelle et al. (2023) gave a result of T ϕ / 5 -1 / 5 and Yan et al. (2023) gave a result of T ϕ / 3 -2 / 3 . In comparison, Theorem 6.2 extends the study to monotone games, and improves the result to O ( max { T 2 ϕ / 3 -2 / 3 , T (4 ϕ +5) 2 / 72 -9 / 8 }) .

Cournot competition

Zero-sum matrixgame

Convex-concave minimax game

3.50

Ours (with estimated gradient)

Ours

Ours

50

OMD with gradient

3.25

OMD with gradient

4.50

OMD with gradient

40

GD with gradient

Payoff for min player

3.00

GD with gradient

GD with estimated gradient

min player

4.25

GD with estimated gradient

GD with gradient

Average cost

2.75

4.00

30

2.50

Payoff for n

3.75

20

2.25

3.50

10

2.00

3.25

0

1.75

3.00

1.50

2.75

0

20

40

60

80

100

0

25

50

75

100

125

150

175

200

0

20

40

60

80

100

Timesteps

Timesteps

Timesteps

7 Experiment

In this section, we provide a numerical evaluation of our proposed algorithm in three static games. We repeat each experiment with 50 different random seeds. We ran all experiments with a 10-core CPU, with 32 GB memory. We set η t = 1 √ t +1 , and δ t = 0 . 001 for our algorithm.

We present the results of the following example games described below. More results with other parameters can be found in the Appendix K.

Cournot competition. In this Cournot duopoly model, n players compete with constant marginal costs, each having individual constant price intercepts and slopes. We model the game with 5 players, where the margin cost is 40 , price intercept is [30 , 50 , 30 , 50 , 30] , and the price slope is [50 , 30 , 50 , 30 , 50] . We have included results of other choices of intercepts and slopes in the appendix.

Zero-sum matrix game. In this zero-sum matrix game, the two players aim to solve the bilinear problem min x max y x /latticetop Ay . We set this matrix A to be [[1 , 2] , [3 , 4]] . We have included results of other choices of A in the appendix.

Monotone zero-sum matrix game. In this monotone version of the zero-sum matrix game, we regularize the game by the regularizer x 2 + y 2 .

Algorithm 1 is evaluated against two baseline methods: online mirror descent and gradient descent, with exact gradient, or estimated gradient (bandit feedback). We set the learning rate η to be 0 . 01 in both zero-sum matrix games and monotone zero-sum matrix games and 0 . 09 in Cournot competition.

Figure 1 summarizes our experimental findings, where our algorithm attains comparable performance to online mirror descent and gradient descent with full information. This demonstrates the fast convergence of our algorithm as suggested by the theoretical results. We also compare our algorithm to gradient descent with an estimated gradient, using the same ellipsoidal gradient estimator, for a fairer comparison. Our algorithms are shown to be more robust to partial information (bandit feedback), which again validates our theoretically fast convergence results. All the codes can be found at https://github.com/jingdong00/monotone_games .

8 Conclusion

In this work, we present a mirror-descent-based algorithm that converges in O ( T -1 / 4 ) in general monotone and smooth games under bandit feedback and strongly uncoupled dynamics. Our algorithm is no-regret, and the result can be improved to O ( T -1 / 2 ) in the case of strongly-monotone games. To our best knowledge, this is the first uncoupled and convergent algorithm in general monotone games under bandit feedback. We then extend our results to time-varying monotone games and present the first result of O ( T -1 / 4 ) for converging games and the improved result of O ( max { T 2 ϕ / 3 -2 / 3 , T (4 ϕ +5) 2 / 72 -9 / 8 } ) for equilibrium tracking. We further verify the effectiveness of our algorithm with empirical evaluations.

References

Abernethy, J., Hazan, E., and Rakhlin, A. (2008). Competing in the dark: An efficient algorithm for bandit linear optimization. In Conference on Learning Theory .

Bartlett, P., Dani, V ., Hayes, T., Kakade, S., Rakhlin, A., and Tewari, A. (2008). High-probability regret bounds for bandit online linear optimization. In Conference on Learning Theory .

Bauschke, H. H., Bolte, J., and Teboulle, M. (2017). A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications. Mathematics of Operations Research , 42(2):330-348.

Bervoets, S., Bravo, M., and Faure, M. (2020). Learning with minimal information in continuous games. Theoretical Economics , 15(4):1471-1508.

Bravo, M., Leslie, D., and Mertikopoulos, P. (2018). Bandit learning in concave n-person games. Advances in Neural Information Processing Systems .

Bregman, L. and Fokin, I. (1987). Methods of determining equilibrium situations in zero-sum polymatrix games. Optimizatsia , 40(57):70-82.

Cai, Y., Luo, H., Wei, C.-Y., and Zheng, W. (2023). Uncoupled and convergent learning in two-player zero-sum markov games. arXiv preprint arXiv:2303.02738 .

Cai, Y., Oikonomou, A., and Zheng, W. (2022). Finite-time last-iterate convergence for learning in multi-player games. Advances in Neural Information Processing Systems .

Cai, Y. and Zheng, W. (2023). Doubly optimal no-regret learning in monotone games. In International Conference on Machine Learning .

Cen, S., Wei, Y., and Chi, Y. (2021). Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems .

Chen, P.-A. and Lu, C.-J. (2016). Generalized mirror descents in congestion games. Artificial Intelligence , 241:217-243.

Daskalakis, C., Deckelbaum, A., and Kim, A. (2011). Near-optimal no-regret algorithms for zero-sum games. In Symposium on Discrete Algorithms .

Debreu, G. (1952). A social equilibrium existence theorem. Proceedings of the National Academy of Sciences , 38(10):886-893.

Drusvyatskiy, D., Fazel, M., and Ratliff, L. J. (2022). Improved rates for derivative free gradient play in strongly monotone games. In Conference on Decision and Control . IEEE.

Duvocelle, B., Mertikopoulos, P., Staudigl, M., and Vermeulen, D. (2023). Multiagent online learning in time-varying games. Mathematics of Operations Research , 48(2):914-941.

Even-Dar, E., Mansour, Y., and Nadav, U. (2009). On the convergence of regret minimization dynamics in concave games. In Symposium on Theory of computing .

Farina, G., Anagnostides, I., Luo, H., Lee, C.-W., Kroer, C., and Sandholm, T. (2022). Near-optimal no-regret learning dynamics for general convex games. Advances in Neural Information Processing Systems .

Hazan, E. and Levy, K. (2014). Bandit convex optimization: Towards tight bounds. Advances in Neural Information Processing Systems .

Jordan, M. I., Lin, T., and Zhou, Z. (2023). Adaptive, doubly optimal no-regret learning in strongly monotone and exp-concave games with gradient feedback. arXiv:2310.14085.

Koller, D., Megiddo, N., and Von Stengel, B. (1996). Efficient computation of equilibria for extensive two-person games. Games and Economic Behavior , 14(2):247-259.

Lee, Y. T. and Yue, M.-C. (2021). Universal barrier is n-self-concordant. Mathematics of Operations Research , 46(3):1129-1148.

Liang, T. and Stokes, J. (2019). Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In International Conference on Artificial Intelligence and Statistics , pages 907-915.

Lin, T., Zhou, Z., Ba, W., and Zhang, J. (2021). Doubly optimal no-regret online learning in strongly monotone games with bandit feedback. arXiv preprint arXiv:2112.02856 .

Mertikopoulos, P., Papadimitriou, C., and Piliouras, G. (2018). Cycles in adversarial regularized learning. In Proceedings of the twenty-ninth annual ACM-SIAM symposium on discrete algorithms .

Mertikopoulos, P. and Zhou, Z. (2019). Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173:465-507.

Rosen, J. B. (1965). Existence and uniqueness of equilibrium points for concave n-person games. Econometrica: Journal of the Econometric Society , pages 520-534.

Roughgarden, T. (2015). Intrinsic robustness of the price of anarchy. Journal of the ACM (JACM) , 62(5):1-42.

Roughgarden, T. and Schoppmann, F. (2015). Local smoothness and the price of anarchy in splittable congestion games. Journal of Economic Theory , 156:317-342.

Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. (2015). Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems .

Tatarenko, T. and Kamgarpour, M. (2019). Learning nash equilibria in monotone games. In IEEE 58th Conference on Decision and Control (CDC) . IEEE.

Tseng, P. (1995). On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics , 60(1-2):237-252.

Yan, Y.-H., Zhao, P., and Zhou, Z.-H. (2023). Fast rates in time-varying strongly monotone games. In International Conference on Machine Learning . PMLR.

Zhou, Z., Mertikopoulos, P., Bambos, N., Boyd, S. P., and Glynn, P. W. (2020). On the convergence of mirror descent beyond stochastic convex programming. SIAM Journal on Optimization , 30(1):687716.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately and clearly state the main theoretical claims made and discuss the main contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper clearly states the assumption made and gives examples of when the assumptions are satisfied. The paper also states clearly of the experimental settings, the computational resources needed, etc.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are clearly stated, and all proofs are included in the appendix. All theorems and lemmas are properly referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All implementation details are given in the experiment section. Code will be released upon acceptance of this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will provide open access to the code and data upon acceptance of this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are described in the experiment settings in detail to reproduce the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, all figures for the experiments are shown with shading the one standard deviation error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All computational resources used have been stated in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I have reviewed the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is foundational and theoretical research without particular application or deployment.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is theoretical with no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The experimental environment used in this paper is synthetic.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The experiment environments have been described in detail in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article presents a set of mirror descent algorithms for oblivious learners in continuous action monotone games. These learners only take actions and receive rewards - they do not receive gradient information or any information about actions or rewards of other players. The authors consider each of static games, converging games, and drifting games, giving state of the art convergence guarantees in each case, in terms of the last iterate convergence. A customary cursory experimental demonstration is provided.\n\nSTRENGTHS AND WEAKNESSES:\nThe article achieves better results than have yet been published, with a O(1/T^0.25) results in monotone games, improving to O(1/T^0.5) for strictly monotone games, with the obvious modifications for converging and drifting games. These are strong results, showing a high quality paper.\nThere is a lot squeezed in to one paper here. This is both a strength and a weakness. It means there is a lot of progress, but is quite difficult to follow all that is attempted to be presented, even for a somewhat expert reader (I have published closely related papers and currently supervise PhD students on this topic). Clarity is somewhat weak as a result.\nWhile the algorithm is somewhat standard, the key innovation which allows these new results is the introduction of the assumption in lines 152-158. Two different regularizers allows better control of the learning rates. However this also brings a weakness - it is far from clear how an oblivious learning could choose there p regularizer, since it depends quite strongly on the particular form of the game's cost functions.\n\nQUESTIONS:\nLines 152-158, and Lines 193-195. Please provide some discussion at least flagging up the issue of how to choose p when learners are supposed to be quite oblivious. Ideally some realistice examples would be given where the reader could see how p could be chosen (I don't count c(x)=x^2).\n\nLine 155: “convex but not linear”. Do we just mean “strictly convex”? I’m concerned that a function c_i that is linear in part of the function space but strictly convex elsewhere might not admit a kappa and admissible p while staying convex.\n\nLines 160-168 repeat the conditions on h twice over.\n\nLine 174. Previous works have had to work to ensure the actually-played \\hat{x}_i^t are in the feasible set. I suspect that the ellipsoidal estimator, combined with the barrier function, does this automatically? Please can you just note this fact - it is not clear from what is presented in the article.\n\nAlgorithm 1, Inputs line. Have I missed something? There a constant kappa that has appeared out of nowhere. What is this and what effect does it have on anything?\n\nAlgorithm 1, line 4. This would read much more nicely if it started with \"sample z\", before then defining \\hat{x}, then receiving the feedback (ie the order in which it happens in the code).\n\nLine 207, and other similar theorem statements. The convergence is given in terms of the sum of the Bregman divergences with respect to the game-dependent, user-chosen p regularizer. In contrast, most results in the literature that are given with respect to Bregman divergences are with respect to a non-game-dependent regularizer such as the h in this article. Some discussion of this would be valuable.\n\nLine 214, the term \"a fixed point given\" is confusing - it could be interpreted that omega is a fixed point of the dynamic (which it is not). I think \"omega_i is any strategy\" would suffice?\n\nSection 5 generally. If mu = 0 it is far from clear to me that there is a unique Nash equilibrium. Yet your results implicitly assume there is. Please can you clarify or fix this.\n\nLine 228. “While it does not imply a bounded first term”. The first term in question is a Bregman divergence with respect to barrier regulariser h. The only way the first term could be “unbounded” for fixed \\omega is if \\omega was on the boundary. Since you want \\omega to be a Nash equilibrium, you ought to say when a Nash equilibrium is on the boundary, \\omega has to be carefully chosen. Boundary Nash are often distinguished in the literature but you make no mention of this.\n\nTheorems 5.2 and 5.3. kappa does not appear in these results as stated. It vanishes some way through the proofs. You need to decide if it should be dropped from Theorem 5.1, or included in these two theorems.\n\nLine 271. I think the only change to algorithm 1 is to replace kappa with tau. And now we specify tau. If I am correct, then this feels like a peculiar way to present it. Why not just specify that now you take kappa=1/T^{1/6} and carry on. Also, within this theorem, you do not specify what delta_t should be. And your results are in terms of G, whereas you have defined G_p in the first line of the statement.\n\nLines 295-297. These appear to claim both that Duvocelle provided a result and that you are the first to provide the same result. Please clarify.\n\nLine 300. By writing down x_i^{t,*} you once again assume a unique equilibrium at each time point. I don't believe conditions have been specified which ensure the uniques of the equilibrium. Please clarify.\n\nLine 303. The key innovation of the paper, the inclusion of the D_p regularizer, is now not included in the the mirror map. Is that deliberate? Please explain why omitting this is now the right thing to do.\n\np9, some better discussion of the results would be extremely helpful. In particular, are the plots showing what we would expect to see (i.e. what is the expected outcome of each of the competitor algorithms, and is this what we see). Is the higher initial variance from your algorithm to be expected? Is the non-convergence of GD with estimate expected? Being consistent with your color scheme across the plots would be really helpful too - green is GD with gradient on the outside plots, but GD with estimated gradient on the middle plot...\n\nLines 324 and 325 seem to contradict line 312?\n\nI have not read all of the proofs in great detail. They look to be about right, but there is a lot of algebra and a lot of repetition. They are not particularly nicely presented - the reader has to do a lot of their own work to follow from one line to the next. I did notice the following problems:\n\nLine 552: \"Similar to Theorem 5.1\" is a little too much of a leap for me. Theorem 5.1 starts by referring directly to Lemma J.1. Lemma J.1 is entirely based on fixed cost functions. I expect that if I were to replace the c's in Lemma J.1 with c_t's then the difference would be the B\\sum\\Delta^t term. But a reader should not have to work through the proof of Lemma J.1 to be confident that all the other terms (except the last 2) in Lemma J.1 are not dependent on c being fixed. I would have thought a Lemma J.1 in terms of possibly evolving cost functions, that could then be specialized by fixing them, would be more useful.\n\nLine 614. Lemma J.3 is for G-\\nabla p being monotone. But here you state (essentially) that G-p is monotone. I believe that, since p is strictly convex, the result may hold anyway. But as written the logic is incorrect I think. Note also that Lemma J.3 is not quite what is needed - it has a single term on the LHS whereas this result has a sum on the LHS (see also comments on Lemma J.3).\n\nLine 615. You should really offer some more help to the reader than “Therefore”. Perhaps “Summing  [Eqn no for line 613] and using [eqn no for line 614] to replace the third term on the right hand side, we can then drop the non-negative D_p(x,omega) term to get” or something. Of course, dropping out the D_p from the inequality in Line 614 (while explaining that you do so) would mean less going on here.\n\nLine 616. Technically, you are summing over t, not over T. There’s a lot going on here too – these three lines took me half an hour to work out! Some more help to the reader would be very valuable. Perhaps “Summing over T, cancelling both sum_{t,i} D_h and sum_{t,i} tD_p terms, and discarding the non-negative D_h(omega,x^{T+1}, we have”. You’re also missing an eta_1 multiplier on the second term of the RHS.\n\nLine 621, and Lemma J.6, need to be significantly clearer. In the lemma, pi is undefined, as is {\\cal X}_\\epsilon. Both need to be defined. Then Line 621 needs to justify why the claimed result is a consequence of the lemma. Using the same words as the lemma would help (i.e. I think that x_i^1 is a center and I think it is implicitly claiming that omega\\in {\\cal X}_{1-1/\\sqrt{T}}, but it is not clear how this fits together or why we have a bound of nu\\log(T).\n\nLine 622, last two sub-lines, you have sum eta_t terms outside of the sum over t, which I think should be inside.\n\nLine 629. I think, since f changes with both x_i^t and t you should probably include these as arguments?\n\nLines 630-631. This needs a lot more explanation. I don’t believe it has been stated that D_p or D_h are self-concordant barriers, so why does adding the linear term make f a self-concordant barrier?\n\nLine 632. This should be an inequality, not an equality.\n\nLine 634 there is a mistake. A_i^t is defined on p5 to be (\\nabla^2 f(x_i^t))^{-1/2}, but here it is taken to be (\\nabla^2 f(x_i^t))^{-1}. Additionally, the second line of the equations is, I think, never used?\n\nLine 635. You take x_i^{t+1} to be argmin f(x_i^t). But it should be argmin_{x_i} f_{x_i^t,t}(x_i) (i.e. the x_i^t and t are fixed arguments, whereas it’s the argument x_i of f that you maximise over). Also, there is a mistake here – the kappa term from algorithm 1 is not part of f, so this armin is not correct.\n\nLine 639. Is p a function from the combined action space X, or from individual action spaces X_i? The first line here implies the former, but the second the latter. There’s also, within the lemma, no obvious reason why the nabla_p (x) – nabla_p(x’) relates to a sum over i of different D_p(x_i,x_i’) terms. I would advise simply making the lemma more specific to the use case in the article.\n\nLIMITATIONS:\nThe article should definitely discuss the limitations from needing to know the cost functions well enough to be able to select an appropriate p. Other than that, I think any limitations are well specified in the results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of no-regret learning in smooth monotone games with bandit feedback, focusing on developing uncoupled algorithms with non-asymptotic last-iterate convergence rates. The main contribution of the paper is that they proposed a mirror descent-based no-regret online learning algorithm that achieves $O(T^{-1/4})$ last-iterate convergence rate with respect to a problem-dependent Bregman divergence between the iterate and the Nash equilibria. When the loss function is linear (i.e., in two-player zero-sum games), they propose another algorithm with a $O(T^{-1/6})$ last-iterate convergence rate. They also extend the results for time-varying monotone games and present convergence rate results under several bounded variation assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: This paper tackles an important question on uncoupled learning in smooth monotone games with bandit feedback, for which existing results are limited to either strongly monotone games, zero-sum games, or only asymptotic convergence rates. \n\nWeakness: Although this paper studies an interesting and important question, several of its claims do not seem correct to me, which significantly undermines its contribution. In short, the main result of $O(T^{-1/4})$ last-iterate convergence rates does not hold for the gap function/other standard convergence measures, and the given guarantee could be vacuous. I think a major revision is need to address these issues. Detailed comments below.\n1. The claimed $O(T^{-1/4})$ last-iterate convergence rates for general smooth monotone games is not for standard convergence measures such as the gap function, the total gap function, the KL-divergence (or Euclidean distance) between the iterate and a Nash equilibrium. Instead, the proximity to a Nash equilibrium $x*$ is measured by $D_p(x^t, x^{*})$, where $p$ is a **game-dependent** convex function. Specifically, in lines 154-155, the function $p$ is chosen so that $c - \\kappa p$ is convex. This is problematic since if we choose $p$ to be a linear function (which is convex and bounded over a compact set), then the corresponding Bregman divergence $D_p(x, x')$ is zero for any two points, making the convergence rate vacuous. Line 193-195 says *\"when $c_i(x) = x^2$, $\\dots$, we can $\\dots$ take $p = x$\"*. For this choice of $p$, $D_p(x, x') = 0$ for any two points, and the guarantees in Theorems 5.1, 5.2, 6.1 are vacuous.\n2. Continuing on the last point, where the authors give an example where they choose $p = x$ to be a linear function. The sentence on Line 263 *\"when $c_i$ is linear, there does exist a $p$ that is convex while making $c_i - \\kappa p$ convex.\"* makes no sense since we can choose $p$ to be a linear function and the resulting function $c_i - \\kappa p$ is linear, thus convex. \n3. The choice of $p$ also gives rise to a minor concern on the uncoupledness of the algorithm. Typically, when the players are playing an unknown game, it is common to assume they know some (upper bounds of) game parameters, such as the smoothness. For example, as long as the agents know they are playing a $L$-smooth monotone game (the specific game instance remains unknown), they can choose their step size and achieve convergence in any $L$-smooth monotone game. But to implement Algorithm 1 in the paper, the agents must also agree on a function $p$ which depends on the unknown game instance, and the choice of $p$ must be different even in the class of $L$-smooth monotone games (and even if they know $L$). This issue affects the uncoupledness of the algorithm.\n\nQUESTIONS:\n1. Does the $O(T^{-1/4})$ convergence rates hold for the gap function or the total gap function? Or in other words, is the $T$-th iterate an  $O(T^{-1/4})$-approximate Nash equilibrium of the game?\n2. What is the requirement of choosing $p$ in Algorithm 1? In the current presentation, it seems that one can choose $p$ to be linear (as discussed in the weakness above), but then the guarantee on $D_p(x, x^*)$ becomes vacuous?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of no-regret learning in general monotone problems under bandit feedback. The main result is a mirror descent-based algorithm that attains faster rates in strongly monotone and monotone problems. The results are also extended to time-varying problems, improving again over the existing rates. Experiments are also provided to support some of the theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\nOn the positive side, the paper studies a well-motivated and important problem. While there has been much work studying last-iterate convergence in games, most existing results apply under full feedback. This paper is part of a recent line of work that tries to extend those results to the bandit feedback setting. There are many challenges in extending those results to the bandit feedback setting. To my knowledge, the results reported by the paper provide significant improvements over the current state of the art. Given the significance of this problem, I believe this is a valuable contribution. The technical approach is also non-trivial and contains some interesting ideas. Although the main ideas appear to be known from prior work, I believe that the way those prior techniques are used in combination in this setting is new. Moreover, the writing overall is reasonable and the paper goes a good job at motivating the key ideas.\n\nOn the other hand, I have some concerns about the paper. First, there is a claim that I find very confusing. Table 1 claims that the rate obtained for zero-sum games is worse than general monotone problems. This doesn't make sense since (two-player) zero-sum games are monotone. The only way to reconcile this is that the main result for the monotone case requires further assumptions that are hidden. It is essential that the authors can address this confusion. Also, the statements of the theorems are not rigorous enough. They do not clearly state the main assumptions and most of the parameters that appear in the statements are not defined or specified in that context. I expect to see formal, self-contained statements.\n\nQUESTIONS:\nI have two main questions:\n\n- Can the authors explain the incongruity highlighted above related to monotone problems and zero-sum games? Is it the case that the result for monotone problems requires stronger assumptions? \n- Can the authors provide the precise statements for the theorems? The key assumptions were not clear even after reading the appendix.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers monotone games in a setting where the players\noperate independently (uncoupled), and observe their costs/payoffs but\nnot the gradient of the cost function (bandit feedback). The class of\nmonotone games include convex-concave games, Cournot competition, and\nsplittable routing games.\n\nIn particular, they analyse the last-iterate distance to the Nash\nequilibrium of an algorithm based on mirror descent. Given that only\nbandit feedback is available, the algorithm uses an ellipsoidal\ngradient estimator.\n\nIn overall, they prove convergence O(1/T^1/2) for strongly monotone\ngames, O(1/T^1/4) for monotone games, and O(1/T^1/6) for linear games\n(results are also available for time-varying games). According to the\nauthors, this is the first last-iterate convergence results available\nfor monotone games. Previous results have either been asymptotic, or\nhave made stronger assumptions about the games.\n\nThere is also an experimental comparison between their algorithm and\nOMD and GD.\n\nSTRENGTHS AND WEAKNESSES:\nIn overall, this is a highly significant contribution that deserves to\nbe published.\n\nI must admit that the paper is outside my core area of expertise. I\nhave therefore not had the opportunity to read every proof in detail.\nHowever, I have some minor remarks below.\n\nThe experimental study is rather light touch, and only uses 5\ndifferent random seeds. The authors should consider more independent\nruns to take into account variance. It would also be interesting to\nsee how the behaviour of the algorithm scales with increasing\ndimension d (currently only very low dimensions are explored). The\nexperimental results should be discussed more in-depth and contrasted\nwith the theoretical results.\n\nQUESTIONS:\nTo follow the analysis, it would be helpful if the paper provides a\nvery brief introduction to mirror descent.\n\nAlgorithm 1, line 4. You could state that z_i^t is sampled u.a.r. from\nthe d-Sphere, and before receiving the bandit feedback.\n\nAlgorithm 1, line 6. It would be useful with a comment about the\nhardness of solving the optimisation problem in Eq (1).\n\nAs far as I can see, there is a typo in the statement of Lemma\nJ.2. Instead of an identity, the result should be an upper bound. In\nparticular, the Holder inequality is stated as an equality. This\nchange would not have any impact on the overall result since Lemma J.2\nis always applied as an upper bound.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}