Title: Boosting Perturbed Gradient Ascent for Last-Iterate Convergence in Games
Venue: NeurIPS 2024
Accepted: no
Keywords: Last-Iterate Convergence

Learning in Games

Noisy Feedback
Abstract: This paper introduces a payoff perturbation technique, introducing a strong convexity to players' payoff functions in games. This technique is specifically designed for first-order methods to achieve last-iterate convergence in games where the gradient of the payoff functions is monotone in the strategy profile space, potentially containing additive noise. Although perturbation is known to facilitate the convergence of learning algorithms, the magnitude of perturbation requires careful adjustment to ensure last-iterate convergence. Previous studies have proposed a scheme in which the magnitude is determined by the distance from an anchoring or reference strategy, which is periodically re-initialized. In response, this paper proposes Gradient Ascent with Boosting Payoff Perturbation, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme. This innovation empowers us to provide faster last-iterate convergence rates against the existing payoff perturbed algorithms, even in the presence of additive noise.

Abstract

This paper introduces a payoff perturbation technique, introducing a strong convexity to players' payoff functions in games. This technique is specifically designed for first-order methods to achieve last-iterate convergence in games where the gradient of the payoff functions is monotone in the strategy profile space, potentially containing additive noise. Although perturbation is known to facilitate the convergence of learning algorithms, the magnitude of perturbation requires careful adjustment to ensure last-iterate convergence. Previous studies have proposed a scheme in which the magnitude is determined by the distance from an anchoring or reference strategy, which is periodically re-initialized. In response, this paper proposes Gradient Ascent with Boosting Payoff Perturbation, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme. This innovation empowers us to provide faster last-iterate convergence rates against the existing payoff perturbed algorithms, even in the presence of additive noise.

1 Introduction

This study considers online learning in monotone games, where the gradient of the payoff function is monotone in the strategy profile space. Monotone games encompassed diverse well-studied games as special instances, such as concave-convex games, zero-sum polymatrix games [Cai and Daskalakis, 2011, Cai et al., 2016], λ -cocoercive games [Lin et al., 2020], and Cournot competition [Bravo et al., 2018]. Due to their wide-ranging applications, there has been growing interest in developing learning algorithms to compute Nash equilibria in monotone games.

Typical learning algorithms such as Gradient Ascent [Zinkevich, 2003] and Multiplicative Weights Update [Bailey and Piliouras, 2018] have been extensively studied and shown to converge to equilibria in an average-iterate sense, which is termed average-iterate convergence . However, averaging the strategies can be undesirable because it can lead to additional memory or computational costs in the context of training Generative Adversarial Networks [Goodfellow et al., 2014] and preference-based fine-tuning of large language models [Munos et al., 2023, Swamy et al., 2024]. In contrast, last-iterate convergence , in which the updated strategy profile itself converges to a Nash equilibrium, has emerged as a stronger notion than average-iterate convergence.

Payoff-perturbed algorithms have recently been regaining attention in this context [Sokota et al., 2023, Liu et al., 2023]. Payoff perturbation is a classical technique, e.g., [Facchinei and Pang, 2003] and introduces a strongly convex penalty to the players' payoff functions to stabilize learning, which leads to convergence to approximate equilibria, not only in the full feedback setting where the perfect gradient vector of the payoff function can be used to update strategies, but also in the noisy feedback setting where the gradient vector is contaminated by noise.

However, to ensure convergence toward a Nash equilibrium of the underlying game, the magnitude 36 of perturbation requires careful adjustment. As a remedy, it is adjusted by the distance from an 37 anchoring or reference strategy. Koshal et al. [2010] and Tatarenko and Kamgarpour [2019] simply 38 decay the magnitude in each iteration, and their methods asymptotically converge, since the perturbed 39 function gradually loses strong convexity. In response to this, recent studies [Perolat et al., 2021, Abe 40 et al., 2023, 2024] re-initialize the anchoring strategies periodically, or in a predefined interval, so 41 that they keep the perturbed function strongly convex and achieve non-asymptotic convergence. 42

43

44

45

46

47

48

We should also mention the optimistic family of learning algorithms, which incorporates recency bias and exhibits last-iterate convergence [Daskalakis et al., 2018, Daskalakis and Panageas, 2019, Mertikopoulos et al., 2019, Wei et al., 2021]. Unfortunately, the property has mainly been proven in the full feedback setting. Although it might empirically work with noisy feedback, the convergence is slower, as demonstrated in Section 6. The fast convergence in the noisy feedback setting is another reason why payoff-perturbed algorithms have been gaining renewed interest.

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

This paper, in particular, focuses on Adaptively Perturbed Mirror Descent (APMD) [Abe et al., 2024], which achieves ˜ O (1 / √ T ) 1 and ˜ O (1 /T 1 10 ) last-iterate convergence rates in the full/noisy feedback setting, respectively. The motivation of this study lies in improving the convergence rates of APMD. We propose an elegant one-line modification of APMD, which effectively accelerates convergence. In fact, we just add the difference between the current anchoring strategy and the initial anchoring strategy to the payoff perturbation function in APMD.

Our contributions are manifold. Firstly, we propose a novel payoff-perturbed learning algorithm named Gradient Ascent with Boosting Payoff Perturbation (GABP). This method incorporates a unique perturbation payoff function, enabling it to achieve faster convergence rates than APMD. Subsequently, we prove that GABP exhibits accelerated ˜ O (1 /T ) and ˜ O (1 /T 1 7 ) last-iterate convergence rates to a Nash equilibrium with full and noisy feedback, respectively. We further show that each player's individual regret is at most O ( (ln T ) 2 ) in the full feedback setting, provided all players play according to GABP. Finally, through our experiments, we demonstrate the competitive or superior performance of GABP over Optimistic Gradient Ascent [Daskalakis et al., 2018, Wei et al., 2021] and APMD in concave-convex games, irrespective of the presence of noise.

2 Preliminaries

Monotone games. In this study, we focus on a continuous multi-player game, which is denoted as ( [ N ] , ( X i ) i ∈ [ N ] , ( v i ) i ∈ [ N ] ) . [ N ] = { 1 , 2 , · · · , N } denotes the set of N players. Each player i ∈ [ N ] chooses a strategy π i from a d i -dimensional compact convex strategy space X i , and we write X = ∏ i ∈ [ N ] X i . Each player i aims to maximize her payoff function v i : X → R , which is differentiable on X . We denote π -i ∈ ∏ j = i X j as the strategies of all players except player i , and π = ( π i ) i ∈ [ N ] ∈ X as the strategy profile . This paper particularly studies learning in smooth monotone games , where the gradient operator V ( · ) = ( ∇ π i v i ( · )) i ∈ [ N ] of the payoff functions is monotone: ∀ π, π ′ ∈ X ,

̸

and L -Lipschitz for L > 0

where ∥ · ∥ denotes the ℓ 2 -norm.

Many common and well-studied games, such as concave-convex games, zero-sum polymatrix games [Cai et al., 2016], λ -cocoercive games [Lin et al., 2020], and Cournot competition [Bravo et al., 2018], are included in the class of monotone games.

Example 2.1 (Concave-Convex Games) . Consider a game defined by ( { 1 , 2 } , ( X 1 , X 2 ) , ( v, -v )) , where v : X 1 × X 2 → R . In this game, player 1 wishes to maximize v , while player 2 aims to minimize v . If v is concave in x 1 ∈ X 1 and convex in x 2 ∈ X 2 , the game is called a concave-convex game or minimax optimization problem, and it is not hard to see that this game is a special case of monotone games.

Nash equilibrium and gap function. A Nash equilibrium [Nash, 1951] is a widely used solution 83 concept for a game, which is a strategy profile where no player can gain by changing her own strategy. 84 Formally, a strategy profile π ∗ ∈ X is called a Nash equilibrium, if and only if π ∗ satisfies the 85 following condition: 86

We define the set of all Nash equilibria to be Π ∗ . It has been shown that there exists at least one Nash 87 equilibrium [Debreu, 1952] for any smooth monotone games. 88

To quantify the proximity to Nash equilibrium for a given strategy profile π ∈ X , we use the gap 89 function , which is defined as: 90

Additionally, we use another measure of proximity to Nash equilibrium, referred to as the tangent residual . This measure is defined as:

where N X ( π ) = { ( a i ) i ∈ [ N ] ∈ ∏ N i =1 R d i | ∑ N i =1 ⟨ a i , π ′ i -π i ⟩ ≤ 0 , ∀ π ′ ∈ X} is the normal cone of π ∈ X . It is easy to see that GAP( π ) ≥ 0 (resp. r tan ( π ) ≥ 0 ) for any π ∈ X , and the equality holds if and only if π is a Nash equilibrium. Defining D := sup π,π ′ ∈X ∥ π -π ′ ∥ as the diameter of X , the gap function for any given strategy profile π ∈ X is upper bounded by its tangent residual.

Lemma 2.2 (Lemma 2 of Cai et al. [2022a]) . For any π ∈ X , we have:

The gap function and the tangent residual are standard measures of proximity to Nash equilibrium; e.g., it has been used in Cai and Zheng [2023], Abe et al. [2024].

Problem setting. This study focuses on the online learning setting in which the following process repeats from iterations t = 1 to T : (i) Each player i ∈ [ N ] chooses her strategy π t i ∈ X i , based on previously observed feedback; (ii) Each player i receives the (noisy) gradient vector ̂ ∇ π i v i ( π t ) as feedback. This study examines two feedback models: full feedback and noisy feedback . In the full feedback setting, each player observes the perfect gradient vector ̂ ∇ π i v i ( π t ) = ∇ π i v i ( π t ) . In the noisy feedback setting, each player's gradient feedback ∇ π i v i ( π t ) is contaminated by an additive noise vector ξ t i , i.e., ̂ ∇ π i v i ( π t ) = ∇ π i v i ( π t ) + ξ t i , where ξ t i ∈ R d i . Throughout the paper, we assume that ξ t i is the zero-mean and bounded-variance noise vector at each iteration t .

Adaptively perturbed Mirror Descent. To facilitate the convergence in the online learning setting, recent studies have utilized a payoff perturbation technique, where payoff functions are perturbed by strongly convex functions [Sokota et al., 2023, Liu et al., 2023, Abe et al., 2022]. However, while the addition of these strongly convex functions leads learning algorithms to converge to a stationary point, this stationary point may be significantly distant from a Nash equilibrium. Therefore, the magnitude of perturbation requires careful adjustment. Perolat et al. [2021], Abe et al. [2023, 2024] have introduced a scheme in which the magnitude is determined by the distance (or divergence function) from an anchoring strategy σ i , which is periodically re-initialized. Specifically, Adaptively Perturbed Mirror Descent (APMD) [Abe et al., 2024] perturbs each player's payoff function by a strongly convex divergence function G ( π i , σ i ) : X i ×X i → [0 , ∞ ) , where the anchoring strategy σ i is periodically replaced by the current strategy π t i every predefined iterations T σ .

Let us define σ k ( t ) i as the anchoring strategy after k ( t ) updates. Since σ i is overwritten every T σ iterations, we can write k ( t ) = ⌊ ( t -1) /T σ ⌋ +1 and σ k ( t ) i = π T σ ( k ( t ) -1)+1 i . Except for the payoff perturbation and the update of the anchor strategy, APMD updates each player i 's strategy in the same way as standard Mirror Descent algorithms:

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

Algorithm 1 GABP for player i .

Require: Learning rates { η t } t ≥ 0 , perturbation strength µ , update interval T σ , initial strategy π 1 i

1:

2:

k

σ

1

, τ

0

1

1

i i 3: for t = 1 , 2 , · · · , T do

π

4: Receive the gradient feedback ̂ ∇ π i v i ( π t )

5: Update the strategy by

6:

7:

8:

τ

τ

=

k

+1

T

σ

then

k

9:

σ

k

i

10: end if

11: end for

where η t is the learning rate at iteration t , µ ∈ (0 , ∞ ) is the perturbation strength , and D ψ ( π i , π ′ i ) = 123 ψ ( π i ) -ψ ( π ′ i ) -⟨∇ ψ ( π ′ i ) , π i -π ′ i ⟩ as the Bregman divergence associated with a strictly convex 124 function ψ : X i → R . When both G and D ψ is set to the squared ℓ 2 -distance, this algorithm can be 125 equivalently written as: 126

127

128

129

130

131

132

133

134

135

We refer to this version of APMD as Adaptively Perturbed Gradient Ascent (APGA). Abe et al. [2024] have shown that APGA exhibits the convergence rates of ˜ O (1 / √ T ) and ˜ O (1 /T 1 10 ) with full and noisy feedback, respectively.

3 Gradient ascent with boosting payoff perturbation

This section proposes an accelerated version of APGA, Gradient Ascent with Boosting Payoff Perturbation (GABP). The pseudo-code of GABP is outlined in Algorithm 1. In order to obtain faster last-iterate convergence rates compared to APGA, GABP introduces a novel payoff perturbation term in addition to APGA's original payoff perturbation term, µ ( π t i -σ k ( t ) i ) . Formally, GABP updates each player's strategy as follows:

The term ( ∗ ) is our proposed additional perturbation term. It shrinks as k ( t ) , the number of updates 136 of σ k ( t ) i , increases. 137

For a more intuitive explanation of the proposed perturbation term, we present the following update 138 rule, which is equivalent to (4): 139

From this formula, it appears that GABP replaces the reference strategy σ k ( t ) i for the perturbation 140 term in (3) of APGA with k ( t ) σ k ( t ) i + σ 1 i k ( t )+1 . As a result, the anchoring strategy in GABP evolves more 141 gradually than in APGA, leading to further stabilization of the learning dynamics. There is a tradeoff 142

+1

, τ

π

t

+1

i

τ

if

0

between the shrinking speed of the term (*) and the stabilizing impact on the last-iterate convergence 143 rate of GABP. The shrinking speed of 1 / ( k ( t ) + 1) achieves a faster convergence rate, and we believe 144 that this represents the optimal balance for this trade-off. Although one might think that the term ( ∗ ) 145 is closely related to Accelerated Optimistic Gradient (AOG) [Cai and Zheng, 2023], we discuss the 146 detail in Appendix F to be concise and avoid a complicated explanation. 147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

4 Last-iterate convergence rates

This section provides the last-iterate convergence rates of GABP in the full/noisy feedback setting, respectively.

4.1 Full feedback setting

First, we demonstrate the last-iterate convergence rate of GABP with full feedback where each player receives the perfect gradient vector as feedback at each iteration t , i.e., ̂ ∇ π i v i ( π t ) = ∇ π i v i ( π t ) . Theorem 4.1 shows that the last-iterate strategy profile π T updated by GABP converges to a Nash equilibrium with an ˜ O (1 /T ) rate in the full feedback setting.

Theorem 4.1. If we use the constant learning rate η t = η ∈ (0 , µ ( L + µ ) 2 ) and the constant perturbation strength µ > 0 , and set T σ = c · max(1 , 6 ln 3( T +1) ln(1+ ηµ ) ) for some constant c ≥ 1 , then the strategy π t updated by GABP satisfies for any t ∈ { 2 , 3 , · · · , T +1 } :

√

This rate is significantly faster than APGA's rate of ˜ O (1 / T ) . Moreover, it is a competitive rate compared to the previous state-of-the-art rate of O (1 /T ) [Yoon and Ryu, 2021, Cai and Zheng, 2023]. Note that the rate in Theorem 4.1 holds for any constant perturbation strength µ > 0 .

4.1.1 Proof sketch of Theorem 4.1

To derive the bound of the gap function GAP( π t ) , it is sufficient to derive that of r tan ( π t ) due to Lemma 2.2. This section provides the proof sketch of Theorem 4.1. The complete proof is placed in Appendix B.

(1) Decomposition of the tangent residual of the last-iterate strategy profile. From the firstorder optimality condition for π t , we can see that V ( π t -1 ) -µ ( π t -1 -k ( t -1) σ k ( t -1) + σ 1 k ( t -1)+1 ) -1 η ( π t -π t -1 ) ∈ N X ( π t ) . Therefore, from the triangle inequality and L -smoothness (2) of the gradient operator, the tangent residual r tan ( π t ) can be bounded as:

Let us define the stationary point π µ,σ k ( t ) , which satisfies the following condition: ∀ i ∈ [ N ] , 170

where ˆ σ k ( t ) i = k ( t ) σ k ( t ) i + σ 1 i k ( t )+1 . We will show that π t converges to the stationary point π µ,σ k ( t ) at an exponential rate later. By using π µ,σ k ( t ) and applying the triangle inequality to ∥ π t -π t -1 ∥ , we decompose the term of O ( ∥ π t -π t -1 ∥ ) into O ( ∥ π t -π µ,σ k ( t -1) ∥ ) and O ( ∥ π µ,σ k ( t -1) -π t -1 ∥ ) .

171

172

173

Similarly, the term of O ( ∥ π t -1 -σ k ( t -1) ∥ ) is decomposed into O ( ∥ π t -1 -π µ,σ k ( t ) -1 ∥ ) and 174 O ( ∥ π µ,σ k ( t ) -1 -σ k ( t -1) ∥ ) . Then, the tangent residual is bounded as follows: 175

Therefore, it is enough to derive the convergence rate on ∥ π µ,σ k ( t ) -1 -π t ∥ and ∥ π µ,σ k ( t -1) -σ k ( t -1) ∥ . 176

177

178

179

180

181

(2) Convergence rate of π t to the stationary point π µ,σ k ( t ) . Using the strong convexity of the perturbation payoff function, µ 2 ∥ x -ˆ σ k ( t ) i ∥ 2 , we show that π t converges to π µ,σ k ( t ) exponentially fast (in Lemma B.1). That is, we have for any t ≥ 1 :

Since the first and second terms of the right-hand side of (5) are bounded by the distance between the stationary point and the anchoring strategy by using (6), we have:

(3) Potential function for bounding the distance between π µ,σ k ( t ) -1 and σ k ( t ) -1 . To derive the 182 upper bound on ∥ ∥ ∥ π µ,σ k ( t -1) -σ k ( t -1) ∥ ∥ ∥ , we define the following potential function P k ( t ) : 183

By some algebra, we can see that P k ( t ) is approximately non-increasing (in Lemma B.3). That is, we 184 have for any t ≥ 1 such that k ( t ) ≥ 2 : 185

Using (6) again, it is easy to show that ∥ ∥ ∥ π µ,σ k ( t ) -σ k ( t )+1 ∥ ∥ ∥ + ∥ ∥ ∥ π µ,σ k ( t ) -1 -σ k ( t ) ∥ ∥ ∥ ≤ O ( 1 ( k ( t )+1) 3 ) 186 for a sufficiently large T σ . Therefore, under the assumption that T σ ≥ Ω(ln T ) , by telescoping of (8) 187 and some algebra, we can derive the following upper bound on ∥ ∥ ∥ π µ,σ k ( t ) -σ k ( t ) ∥ ∥ ∥ (in Lemma B.2): 188

(4) Putting it all together: last-iterate convergence rate of π t . By combining (7) and (9), we get r tan ( π t ) ≤ O ( 1 k ( t -1)+1 ) . Therefore, since k ( t ) = ⌊ t -1 T σ ⌋ +1 , it holds that r tan ( π t ) ≤ O ( T σ t + T σ -2 ) . Finally, taking T σ = Θ(ln T ) , we have:

The upper bound on the gap function is immediately obtained since we have Lemma 2.2.

4.2 Noisy feedback setting

Next, we establish the last-iterate convergence rate in the noisy feedback setting, where each player i observes a noisy gradient vector contaminated by an additive noise vector ξ t i ∈ R d i : ̂ ∇ π i v i ( π t ) + ξ t i . We assume that the noisy vector ξ t i is zero-mean and its variance is bounded. Formally, defining the sigma-algebra generated by the history of the observations as F t := σ ( ( ̂ ∇ π i v i ( π 1 )) i ∈ [ N ] , . . . , ( ̂ ∇ π i v i ( π t -1 )) i ∈ [ N ] ) , ∀ t ≥ 1 , the noisy vector ξ t i is assumed to satisfy the following conditions:

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

Assumption 4.2. For all t ≥ 1 and i ∈ [ N ] , the noise vector ξ t i satisfies the following properties: (a) Zero-mean: E [ ξ t i |F t ] = (0 , · · · , 0) ⊤ ; (b) Bounded variance: E [ ∥ ξ t i ∥ 2 |F t ] ≤ C 2 with some constant C > 0 .

Assumption 4.2 is standard in online learning in games with noisy feedback [Mertikopoulos and Zhou, 2019, Hsieh et al., 2019, Abe et al., 2024] and stochastic optimization [Nemirovski et al., 2009, Nedi´ c and Lee, 2014]. Under Assumption 4.2 and a decreasing learning rate sequence η t , we can obtain a faster last convergence rate ˜ O (1 /T 1 7 ) than the convergence rate ˜ O (1 /T 1 10 ) of APGA.

Theorem 4.3. Let κ = µ 2 , θ = 3 µ 2 +8 L 2 2 µ . Suppose that Assumption 4.2 holds and V ( π ) ≤ ζ for any π ∈ X . We also assume that T σ is set to satisfy T σ = c · max( T 6 7 , 1) for some constant c ≥ 1 . If we use the constant perturbation strength µ > 0 and the decreasing learning rate sequence η t = 1 κ ( t -T σ ( k ( t ) -1))+2 θ , then the strategy π T +1 satisfies:

Note that the non-increasing property, as described in (8), of the potential function holds regardless 211 of the presence of noise. This implies that a proof technique similar to the one used with the potential 212 function in the full feedback setting can also be applied in the noisy feedback setting. The detailed 213 proof can be found in Appendix C. 214

215

216

217

218

219

220

5 Individual regret bound

In this section, we present an upper bound on an individual regret for each player. Specifically, we examine two performance measures in our study: the external regret and the dynamic regret [Zinkevich, 2003]. The external regret is a conventional measure in online learning. In online learning in games, the external regret for player i is defined as the gap between the player's realized cumulative payoff and the cumulative payoff of the best fixed strategy in hindsight:

The dynamics regret is a much stronger performance metric, which is given by: 221

We show in Theorem 5.1 that the individual regret is at most O ( (ln T ) 2 ) if each player i ∈ [ N ] plays according to GABP in the full feedback setting:

Theorem 5.1. In the same setup of Theorem 4.1, we have for any player i ∈ [ N ] and T ≥ 3 :

√

This regret bound is significantly superior to the O ( T ) regret bound of Optimistic Gradient Ascent, and it is slightly inferior to the O (ln T ) regret bound of AOG [Cai and Zheng, 2023]. The proof is given in Appendix D.

6 Experiments

222

223

224

225

226

227

228

In this section, we present the empirical results of our GABP, comparing its performance with 229 Adaptively Perturbed Gradient Ascent (APGA) [Abe et al., 2024] and Optimistic Gradient Ascent 230 (OGA) [Daskalakis et al., 2018, Wei et al., 2021]. We conduct experiments on two classes of concave231 convex games. The first experiment is carried out on random payoff games, which are two-player 232 zero-sum normal-form games with payoff matrices of size d . In this game, each player's strategy 233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

10 0

10 1

10 2

10 3

Iterations

-3

-2

-1

log10 (GAP(

))

10 0

10 1

10 2

10 3

10 4

Iterations

-1.5

-1.0

-0.5

OGA

APGA

GABP

10 0

10 1

10 2

10 3

10 4

10 5

Iterations

-2

-1

0

log10

(

r

(

))

Hard Concave-Convex (Full)

10 0

10 1

10 2

10 3

10 4

10 5

Iterations

-0.5

0.0

0.5

Hard Concave-Convex (Noisy)

0.0

0.2

0.4

0.6

0.8

1.0

Iterations

×10 5

0.00

0.25

0.50

0.75

1.00

DynamicReg

×10 7

Hard Concave-Convex (Full)

0.0

0.2

0.4

0.6

0.8

1.0

Iterations

×10 5

0.0

0.5

1.0

1.5

2.0

×10 7

Hard Concave-Convex (Noisy)

OGA

APGA

GABP

space is represented by the d -dimensional probability simplex, i.e., X 1 = X 2 = ∆ d . All entries of the payoff matrix are drawn independently from a uniform distribution over the interval [ -1 , 1] . We set d = 50 and the initial strategies are set to π 1 1 = π 1 2 = 1 d 1 . The second instance is a hard concave-convex game [Ouyang and Xu, 2021], formulated as the following max-min optimization problem: max x ∈X 1 min y ∈X 2 f ( x, y ) , where f ( x, y ) = -1 2 x ⊤ Hx + h ⊤ x + ⟨ Ax -b, y ⟩ . Following the setup in Cai and Zheng [2023], we choose X 1 = X 2 = [ -200 , 200] d with d = 100 . The precise terms of H ∈ R d × d , A ∈ R d × d , b ∈ R d , and h ∈ R d are provided in Appendix E.2. All algorithms are executed with initial strategies π 1 1 = π 1 2 = 1 n 1 . The detailed hyperparameters of the algorithms, tuned for best performance, are shown in Table 1 in Appendix E.3.

The numerical results of the random payoff game and the hard concave-convex game are shown in Figure 1. Both the full feedback and noisy feedback experiments in the random payoff game were conducted with 50 different random seeds, which corresponds to using 50 different payoff matrices. For experiments on the hard concave-convex game with noisy feedback, we use 10 different random seeds. We assume that the noise vector ξ t i is generated from the multivariate Gaussian distribution N (0 , 0 . 1 2 I ) in an i.i.d. manner for both games. We observe that GABP exhibits competitive or faster performance over APGA and OGA in all experiments.

Figure 2 illustrates the dynamic regret in the hard concave-convex game. GABP exhibits lower regret than APGA and OGA in both settings, demonstrating its efficiency and robustness. Note that APGA and OGA exhibit almost identical trajectories with full feedback, with their plots overlapping completely.

7 Related literature

No-regret learning algorithms have been extensively studied with the intent of achieving key objectives 255 such as average-iterate convergence or last-iterate convergence. Recently, learning algorithms 256 introducing optimism [Rakhlin and Sridharan, 2013a,b], such as optimistic Follow the Regularized 257 Leader [Shalev-Shwartz and Singer, 2006] and optimistic Mirror Descent [Zhou et al., 2017, Hsieh 258 et al., 2021], have been introduced to admit last-iterate convergence in a broad spectrum of game 259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

settings. These optimistic algorithms with full feedback have been shown to achieve last-iterate convergence in various classes of games, including bilinear games [Daskalakis et al., 2018, Daskalakis and Panageas, 2019, Liang and Stokes, 2019, de Montbrun and Renault, 2022], cocoercive games [Lin et al., 2020], and saddle point problems [Daskalakis and Panageas, 2018, Mertikopoulos et al., 2019, Golowich et al., 2020b, Wei et al., 2021, Lei et al., 2021, Yoon and Ryu, 2021, Lee and Kim, 2021, Cevher et al., 2023]. Recent studies have provided finite convergence rates for monotone games [Golowich et al., 2020a, Cai et al., 2022a,b, Gorbunov et al., 2022, Cai and Zheng, 2023].

Compared to the full feedback setting, there are significant challenges in learning with noisy feedback. For example, a learning algorithm must estimate the gradient from feedback that is contaminated by noise. Despite the challenge, a vast literature has successfully achieved last-iterate convergence with noisy feedback in specific classes of games, including potential games [Cohen et al., 2017], strongly monotone games [Giannou et al., 2021b,a], and two-player zero-sum games [Abe et al., 2023]. These results have often leveraged unique structures of their payoff functions, such as strict (or strong) monotonicity [Bravo et al., 2018, Kannan and Shanbhag, 2019, Hsieh et al., 2019, Anagnostides and Panageas, 2022] and strict variational stability [Mertikopoulos et al., 2019, Azizian et al., 2021, Mertikopoulos and Zhou, 2019, Mertikopoulos et al., 2022]. Without these restrictions, convergence is mainly demonstrated in an asymptotic manner, with no quantification of the rate [Koshal et al., 2010, 2013, Yousefian et al., 2017, Tatarenko and Kamgarpour, 2019, Hsieh et al., 2020, 2022, Abe et al., 2023]. Consequently, an exceedingly large number of iterations might be necessary to reach an equilibrium.

There have been several studies focusing on payoff-regularized learning, where each player's payoff or utility function is perturbed or regularized via strongly convex functions [Cen et al., 2021, 2023, Pattathil et al., 2023]. Previous studies have successfully achieved convergence to stationary points, which are approximate equilibria. For instance, Sokota et al. [2023] have demonstrated that their perturbed mirror descent algorithm converges to a quantal response equilibrium [McKelvey and Palfrey, 1995, 1998]. Similar results have been obtained with the Boltzmann Q-learning dynamics [Tuyls et al., 2006] and penalty-regularized dynamics [Coucheney et al., 2015] in continuous-time settings [Leslie and Collins, 2005, Abe et al., 2022, Hussain et al., 2023]. To ensure convergence toward a Nash equilibrium of the underlying game, the magnitude of perturbation requires careful adjustment. Several learning algorithms have been proposed to gradually reduce the perturbation strength µ in response to this [Bernasconi et al., 2022, Liu et al., 2023, Cai et al., 2023]. These include well-studied methods such as iterative Tikhonov regularization [Facchinei and Pang, 2003, Koshal et al., 2010, Tatarenko and Kamgarpour, 2019]. Alternatively, Perolat et al. [2021] and Abe et al. [2023] have employed a payoff perturbation scheme, where the magnitude of perturbation is determined by the distance from an anchoring strategy, which is periodically re-initialized by the current strategy. Recently, Abe et al. [2024] have established ˜ O (1 / √ T ) and ˜ O (1 /T 1 10 ) last-iterate convergence rates for the payoff perturbation scheme in the full/noisy feedback setting, respectively. Our algorithm achieves faster ˜ O (1 /T ) and ˜ O (1 /T 1 7 ) last-iterate convergence rates by modifying the periodically re-initializing anchoring strategy scheme so that the anchoring strategy evolves more gradually.

8 Conclusion

This study proposes a novel payoff-perturbed algorithm, Gradient Ascent with Boosting Payoff Perturbation, which achieves ˜ O (1 /T ) and ˜ O (1 /T 1 7 ) last-iterate convergence rates in monotone games with full/noisy feedback, respectively. Extending our results in settings where each player only observes bandit feedback is an intriguing and challenging future direction.

References

Kenshi Abe, Mitsuki Sakamoto, and Atsushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In UAI , pages 1-10, 2022.

Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-iterate convergence with full and noisy feedback in two-player zero-sum games. In AISTATS , pages 7999-8028, 2023.

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Atsushi Iwasaki. Adaptively perturbed mirror descent for learning in games. In ICML , 2024.

Ioannis Anagnostides and Ioannis Panageas. Frequency-domain representation of first-order methods: A simple and robust framework of analysis. In SOSA , pages 131-160, 2022.

Waïss Azizian, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. The last-iterate convergence rate of optimistic mirror descent in stochastic variational inequalities. In COLT , pages 326-358, 2021.

James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Economics and Computation , pages 321-338, 2018.

Martino Bernasconi, Alberto Marchesi, and Francesco Trovò. Last-iterate convergence to tremblinghand perfect equilibria. arXiv preprint arXiv:2208.08238 , 2022.

Mario Bravo, David Leslie, and Panayotis Mertikopoulos. Bandit learning in concave N-person games. In NeurIPS , pages 5666-5676, 2018.

Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. In SODA , pages 217-234, 2011.

Yang Cai and Weiqiang Zheng. Doubly optimal no-regret learning in monotone games. In ICML , pages 3507-3524, 2023.

Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Mathematics of Operations Research , 41(2):648-655, 2016.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In NeurIPS , pages 33904-33919, 2022a.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Tight last-iterate convergence of the extragradient method for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228 , 2022b.

Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Uncoupled and convergent learning in two-player zero-sum markov games with bandit feedback. In NeurIPS , pages 36364-36406, 2023.

Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. In NeurIPS , pages 27952-27964, 2021.

Shicong Cen, Yuejie Chi, Simon S Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum Markov games. In ICLR , 2023.

Volkan Cevher, Georgios Piliouras, Ryann Sim, and Stratis Skoulakis. Min-max optimization made simple: Approximating the proximal point method via contraction maps. In Symposium on Simplicity in Algorithms (SOSA) , pages 192-206, 2023.

Johanne Cohen, Amélie Héliou, and Panayotis Mertikopoulos. Learning with bandit feedback in potential games. In NeurIPS , pages 6372-6381, 2017.

Pierre Coucheney, Bruno Gaujal, and Panayotis Mertikopoulos. Penalty-regulated dynamics and robust learning procedures in games. Mathematics of Operations Research , 40(3):611-633, 2015.

Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. In NeurIPS , pages 9256-9266, 2018.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In ITCS , pages 27:1-27:18, 2019.

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism. In ICLR , 2018.

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

Étienne de Montbrun and Jérôme Renault. Convergence of optimistic gradient descent ascent in bilinear games. arXiv preprint arXiv:2208.03085 , 2022.

Gerard Debreu. A social equilibrium existence theorem. Proceedings of the National Academy of Sciences , 38(10):886-893, 1952.

Francisco Facchinei and Jong-Shi Pang. Finite-dimensional variational inequalities and complementarity problems . Springer, 2003.

Angeliki Giannou, Emmanouil Vasileios Vlatakis-Gkaragkounis, and Panayotis Mertikopoulos. Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information. In COLT , pages 2147-2148, 2021a.

Angeliki Giannou, Emmanouil-Vasileios Vlatakis-Gkaragkounis, and Panayotis Mertikopoulos. On the rate of convergence of regularized learning in games: From bandits and uncertainty to optimism and beyond. In NeurIPS , pages 22655-22666, 2021b.

Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. In NeurIPS , pages 20766-20778, 2020a.

Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. In COLT , pages 1758-1784, 2020b.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , pages 2672-2680, 2014.

Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In NeurIPS , pages 21858-21870, 2022.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In NeurIPS , pages 6938-6948, 2019.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling. In NeurIPS , pages 16223-16234, 2020.

Yu-Guan Hsieh, Kimon Antonakopoulos, and Panayotis Mertikopoulos. Adaptive learning in continuous games: Optimal regret bounds and convergence to Nash equilibrium. In COLT , pages 2388-2422, 2021.

Yu-Guan Hsieh, Kimon Antonakopoulos, Volkan Cevher, and Panayotis Mertikopoulos. No-regret learning in games with noisy feedback: Faster rates and adaptivity via learning rate separation. In NeurIPS , pages 6544-6556, 2022.

Aamal Abbas Hussain, Francesco Belardinelli, and Georgios Piliouras. Asymptotic convergence and performance of multi-agent Q-learning dynamics. arXiv preprint arXiv:2301.09619 , 2023.

Aswin Kannan and Uday V. Shanbhag. Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants. Computational Optimization and Applications , 74(3):779-820, 2019.

Jayash Koshal, Angelia Nedi´ c, and Uday V Shanbhag. Single timescale regularized stochastic approximation schemes for monotone nash games under uncertainty. In CDC , pages 231-236. IEEE, 2010.

Jayash Koshal, Angelia Nedic, and Uday V. Shanbhag. Regularized iterative stochastic approximation methods for stochastic variational inequality problems. IEEE Transactions on Automatic Control , 58(3):594-609, 2013.

Sucheol Lee and Donghwan Kim. Fast extra gradient methods for smooth structured nonconvexnonconcave minimax problems. In NeurIPS , pages 22588-22600, 2021.

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In AISTATS , pages 1441-1449, 2021.

David S Leslie and Edmund J Collins. Individual q-learning in normal form games. SIAM Journal on Control and Optimization , 44(2):495-514, 2005.

Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In AISTATS , pages 907-915, 2019.

Tianyi Lin, Zhengyuan Zhou, Panayotis Mertikopoulos, and Michael Jordan. Finite-time last-iterate convergence for multi-agent learning in games. In ICML , pages 6161-6171, 2020.

Mingyang Liu, Asuman Ozdaglar, Tiancheng Yu, and Kaiqing Zhang. The power of regularization in solving extensive-form games. In ICLR , 2023.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6-38, 1995.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for extensive form games. Experimental economics , 1:9-41, 1998.

Panayotis Mertikopoulos and Zhengyuan Zhou. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1):465-507, 2019.

Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In ICLR , 2019.

Panayotis Mertikopoulos, Ya-Ping Hsieh, and Volkan Cevher. Learning in games from a stochastic approximation viewpoint. arXiv preprint arXiv:2206.03922 , 2022.

Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886 , 2023.

John Nash. Non-cooperative games. Annals of mathematics , pages 286-295, 1951.

Angelia Nedi´ c and Soomin Lee. On stochastic subgradient mirror-descent algorithm with weighted averaging. SIAM Journal on Optimization , 24(1):84-107, 2014.

Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization , 19(4):1574-1609, 2009.

Yuyuan Ouyang and Yangyang Xu. Lower complexity bounds of first-order methods for convexconcave bilinear saddle-point problems. Mathematical Programming , 185(1):1-35, 2021.

Sarath Pattathil, Kaiqing Zhang, and Asuman Ozdaglar. Symmetric (optimistic) natural policy gradient for multi-agent learning with parameter convergence. In AISTATS , pages 5641-5685, 2023.

Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From Poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In ICML , pages 8525-8535, 2021.

Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In COLT , pages 993-1019, 2013a.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In NeurIPS , pages 3066-3074, 2013b.

Shai Shalev-Shwartz and Yoram Singer. Convex repeated games and fenchel duality. Advances in neural information processing systems , 19, 2006.

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

Samuel Sokota, Ryan D'Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. In ICLR , 2023.

Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 , 2024.

Tatiana Tatarenko and Maryam Kamgarpour. Learning Nash equilibria in monotone games. In CDC , pages 3104-3109. IEEE, 2019.

Karl Tuyls, Pieter Jan Hoen, and Bram Vanschoenwinkel. An evolutionary dynamical analysis of multi-agent learning in iterated games. Autonomous Agents and Multi-Agent Systems , 12(1): 115-153, 2006.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In ICLR , 2021.

TaeHo Yoon and Ernest K Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O (1 /k 2 ) rate on squared gradient norm. In ICML , pages 12098-12109, 2021.

Farzad Yousefian, Angelia Nedi´ c, and Uday V Shanbhag. On smoothing, regularization, and averaging in stochastic approximation methods for stochastic variational inequality problems. Mathematical Programming , 165:391-431, 2017.

Zhengyuan Zhou, Panayotis Mertikopoulos, Aris L Moustakas, Nicholas Bambos, and Peter Glynn. Mirror descent learning in continuous games. In CDC , pages 5776-5783. IEEE, 2017.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In ICML , pages 928-936, 2003.

A Broader impact

Our study can bring about a positive impact on society by contributing to the advancement of the 469 Game AI industry. However, as far as we can envision, there are no conceivable negative social 470 impacts. 471

472

473

B Proofs for Theorem 4.1

B.1 Proof of Theorem 4.1

Proof of Theorem 4.1. From the first-order optimality condition for π t , we have for any x ∈ X : 474

and then V ( π t -1 ) -µ ( π t -1 -k ( t -1) σ k ( t -1) + σ 1 k ( t -1)+1 ) -1 η ( π t -π t -1 ) ∈ N X ( π t ) . Thus, the tangent 475 residual for π t can be bounded as: 476

Letting us define 477

then we get by triangle inequality: 478

In terms of upper bound on ∥ ∥ ∥ π µ,σ k ( t -1) -π t ∥ ∥ ∥ and ∥ ∥ ∥ π µ,σ k ( t -1) -π t -1 ∥ ∥ ∥ , we introduce the following 479 lemma: 480

Lemma B.1. If we use the constant learning rate η t = η ∈ (0 , µ ( L + µ ) 2 ) , we have for any t ≥ 1 : 481

Combining (10) and Lemma B.1, we have: 482

Next, we derive the following upper bound on ∥ ∥ ∥ π µ,σ k ( t -1) -σ k ( t -1) ∥ ∥ ∥ : 483

Lemma B.2. If we set η t = η ∈ (0 , µ ( L + µ ) 2 ) and T σ ≥ max(1 , 6 ln 3( T +1) ln(1+ ηµ ) ) , we have for any t ≥ 1 : 484 ∥ ∥ ∥ π µ,σ k ( t ) -σ k ( t ) ∥ ∥ ∥ ≤ 8 D k ( t ) + 1 .

By combining (11) and Lemma B.2, we get: 485

Therefore, since k ( t ) = ⌊ t -1 T σ ⌋ +1 , it holds that: 486

Finally, taking T σ = c · max(1 , 6 ln 3( T +1) ln(1+ ηµ ) ) , we have: 487

488

B.2 Proof of Lemma B.1 489

Proof of Lemma B.1. First, we have for any three vectors a, b, c : 490

Thus, we have for any t ≥ 1 : 491

Here, let us define ˆ σ k ( t ) = k ( t ) σ k ( t ) + σ 1 k ( t )+1 . Then, from the first-order optimality condition for π t +1 , we 492 have for any t ≥ 1 : 493

Similarly, from the first-order optimality condition for π µ,σ k ( t ) , we get: 494

Combining (12), (13), and (14) yields: 495

where the second inequality follows from (1). From Cauchy-Schwarz inequality and Young's 496 inequality, the second term in the right-hand side of this inequality can be bounded by: 497

where the second inequality follow from (2), and the last inequality follows from the assumption that 498 η ≤ µ ( L + µ ) 2 . By combining (15) and (16), we get: 499

Thus, 500

501

502

Therefore, we have for any t ≥ 1 :

Furthermore, since k ( s ) = k ( t ) for s ∈ [( k ( t ) -1) T σ +1 , t ] , we have for such s that:

Therefore, by applying this inequality from t, t -1 , · · · , ( k ( t ) -1) T σ +1 , we get for any t ≥ 1 : 503

̸

Here, since k ( t ) = k ( t +1) when t satisfies that t = T σ ⌊ t T σ ⌋ , we have for such t that: 504

On the other hand, when t satisfies that t = T σ ⌊ t T σ ⌋ : 505

Therefore, we have for any t ≥ 1 such that t = T σ ⌊ t T σ ⌋ : 506

By combining (17), (18), and (19), we have for any t ≥ 1 : 507

508

B.3 Proof of Lemma B.2 509

Proof of Lemma B.2. First, we have for any Nash equilibrium π ∗ ∈ Π ∗ and t ≥ 1 such that k ( t ) ≥ 1 : 510

Here, the first-order optimality condition for π µ,σ k ( t ) : 511

where we use (1) and the fact that π ∗ is a Nash equilibrium. Combining these inequalities yields: 512

From Young's inequality, we have for any ρ 1 , ρ 2 > 0 : 513

By setting ρ 1 = 4 k ( t )+2 , ρ 2 = 4( k ( t )+1) k ( t )+2 , we obtain: 514

Here, we introduce the following lemma: 515

Lemma B.3. For any t ≥ 1 such that k ( t ) ≥ 2 , we have: 516

By combining (20) and Lemma B.3, we get: 517

Therefore, we have for any t ≥ 1 such that k ( t ) ≥ 2 : 518

By the definition of ˆ σ k ( t ) , 519

Therefore, from Cauchy-Schwarz inequality, we have: 520

Furthermore, from Lemma B.1, we have for any k ≥ 1 : 521

Combining (21) nad (22), we have for any t ≥ 1 such that k ( t ) ≥ 2 : 522

Therefore, since T σ ≥ max(1 , 6 ln 3( T +1) ln(1+ ηµ ) ) ⇒ ( 1 1+ ηµ ) T σ ≤ ( k ( t )+1) 3 (1+ ηµ ) Tσ ≤ 1 16 , we have for k ( t ) ≥ 2 : 523

and then: 524

On the other hand, for k ( t ) = 1 , we have: 525

In summary, for any t ≥ 1 , we have: 526

527

B.4 Proof of Lemma B.3 528

Proof of Lemma B.3. From the first-order optimality condition for π µ,σ k ( t ) , we have: 529

Similarly, from the first-order optimality condition for π µ,σ k ( t ) -1 , we have: 530

Summing up these inequalities, we get for any t ≥ 1 such that k ( t ) ≥ 2 : 531

Here, for any vectors a, b, c , it holds that: 532

Thus, we have: 533

Here, from the definition of ˆ σ k ( t ) , we have: 534

Combining (23) and (24) yields for any t ≥ 1 such that k ( t ) ≥ 2 : 535

Multiplying both sides by k ( t )( k ( t ) + 1) , we have: 536

537

C Proofs for Theorem 4.3 538

539

540

541

C.1 Proof of Theorem 4.3

Proof of Theorem 4.3. Let us define K := T T σ . We can decompose the gap function for π T +1 as follows:

where the last inequality follows from Lemma 2.2. From the first-order optimality condition for 542 π µ,σ K , we have for any x ∈ X : 543

and then V ( π µ,σ K ) -µ ( π µ,σ K -Kσ K + σ 1 K +1 ) ∈ N X ( π µ,σ K ) . Thus, the gap function for π T +1 can 544 be bounded by: 545

Taking its expectation yields: 546

Here, we derive the following upper bound on E [ ∥ ∥ ∥ π µ,σ k ( t ) -π t +1 ∥ ∥ ∥ 2 ] : 547

Lemma C.1. Let κ = µ 2 , θ = 3 µ 2 +8 L 2 2 µ . Suppose that Assumption 4.2 holds. If we set η t = 548 1 κ ( t -T σ ( k ( t ) -1))+2 θ , we have for any t ≥ 1 : 549

Setting t = T = KT σ , we can write k ( t ) = ⌊ KT σ -1 T σ ⌋ +1 = K . Therefore, from Lemma C.1, we 550 have: 551

On the other hand, in terms of E [∥ ∥ ∥ π µ,σ k ( t ) -σ k ( t ) ∥ ∥ ∥ ] , we introduce the following lemma: 552

Lemma C.2. If we set η t = 1 κ ( t -T σ ( k ( t ) -1))+2 θ and T σ ≥ max(1 , T 6 7 ) , we have for any t ≥ 1 : 553

By setting t = KT σ in this lemma, we get: 554

Combining (25), (26), and (27), we have: 555

where the second inequality follows from K = T T σ . Finally, since T σ = c · max(1 , T 6 7 ) , we have for 556 any T ≥ T σ : 557

Since T = T σ K , we have finally: 558

559

C.2 Proof of Lemma C.1 560

Proof of Lemma C.1. From the first-order optimality condition for π t +1 , we have for t ≥ 1 : 561

Combining (28), (12), and (14), we have: 562

where the third inequality follows from (1). From Cauchy-Schwarz inequality and Young's inequality, 563 the fourth term on the right-hand side of this inequality can be bounded by: 564

By combining (29) and (30), we have: 565

By taking the expectation conditioned on F t for both sides and using Assumption 4.2 (a) and (b), 566

Therefore, under the setting where η t = 1 κ ( t -T σ ( k ( t ) -1))+2 θ , we have for any t ≥ 1 : 567

Rearranging and taking the expectations, we get: 568

Therefore, 571

Here, we have: 572

Combining (31), (32), and the fact that π ( k ( t ) -1) T σ +1 = σ k ( t ) , we have: 573

574

C.3 Proof of Lemma C.2 575

Proof of Lemma C.2. First, from Lemma C.1, we have for any k ≥ 1 : 576

Moreover, by taking the expectation of (21), we have for any t ≥ 1 such that k ( t ) ≥ 2 : 577

Combining these inequalities, we get for any t ≥ 1 such that k ( t ) ≥ 2 : 578

Since T σ ≥ max(1 , T 6 7 ) ⇒ k ( t ) 3 √ T σ ≤ 1 , we have: 579

Since E [ X ] 2 ≤ E [ X 2 ] for any random variable X , we get: 580

581

Furthermore, for k ( t ) = 1 , we have: 582

Therefore, we have for any t ≥ 1 : 583

584

D Proof of Theorem 5.1 585

Proof of Theorem 5.1. By the definition of dynamic regret, we have: 586

Here, we introduce the following lemma: 587

Lemma D.1 (Lemma 2 of Cai et al. [2022a]) . For any π ∈ X , we have: 588

Therefore, we have: 589

Thus, from Theorem 4.1: 590

591

592

E Experimental details

E.1 Information on the computer resources 593

The experiments were conducted on macOS Sonoma 14.4.1 with Apple M2 Max and 32GB RAM. 594

E.2 Hard concave-convex game 595

Following the setup in Ouyang and Xu [2021], Cai and Zheng [2023], we choose 596

⊤

and H = 2 A A . 597

E.3 Hyperparameters 598

For each game, we carefully tuned the hyperparameters for each algorithm to ensure optimal perfor599 mance. The specific parameters for each game and setting are summarized in Table 1. 600

F Relationship with accelerated optimistic gradient algorithm 601

Our GABP bears some relation to Accelerated Optimistic Gradient (AOG) [Cai and Zheng, 2023], 602 which updates the strategy by: 603

This can be equivalently written as: 604

This means that AOG employs a convex combination tπ t i + π 1 i t +1 of the current strategy π t i and initial 605 strategy π 1 i as the proximal point in gradient ascent. However, our GABP diverges from AOG in that it 606 uses a convex combination k ( t ) σ k ( t ) i + σ 1 i k ( t )+1 of σ k ( t ) i and σ 1 i as the reference strategy for the perturbation 607 term. 608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions and scope of this study.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In 'Introduction' and 'Conclusion', we have reiterated the limitation of this study.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see the theoretical results and their proofs in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided descriptions of experimental setups in the experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included the experimental code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided descriptions of experimental setups in the experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please see Figures.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have shown the computer resources for this study in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed and followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have described the potential societal impacts of our study in Appendix A. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no such risks associated with the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This study does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies last-iterate convergence rates of online learning in monotone games. The main contribution is an algorithm called Gradient Ascent with Boosting Payoff Perturbation (GABP). The GABP algorithm achieves (1) $O(\\log T / T)$ last-iterate convergence with full gradient feedback, which is near-optimal; (2) and $O(1/T^{1/7})$ last-iterate convergence with noisy gradient feedback (the noise is zero-mean with bounded variance). The latter result improves prior results of $O(1/T^{1/10})$. Moreover, the GABP algorithm guarantees an individual dynamic regret of $O(\\log^2 T)$ under full gradient feedback, slightly worse than the state-of-the-art bound of $O(\\log T)$. This paper also contains numerical experiments on small game instances to demonstrate the effectiveness of GABP.\n\nSTRENGTHS:\nThe problem of last-iterate convergence rates of no-regret learning algorithms in monotone games is relevant and interesting. Most existing results focus on the full gradient feedback, while only a few provide concrete convergence rates under the noisy gradient or the bandit feedback. The proposed GABP algorithm has near-optimal $O(\\log T / T)$ last-iterate convergence rate under full gradient feedback. It also improves the convergence rates under noisy gradient feedback from $O(1/T^{1/10})$ to $O(1/T^{1/7})$. This is a solid contribution to learning in games, although the rate for the noisy gradient feedback setting may not be tight.\n\nWEAKNESSES:\n1. The proposed GABP algorithm does not achieve the optimal $O(1/T)$ last-iterate convergence rate under full gradient feedback. The $O(1/T^{1/7})$ last-iterate convergence rate is also not tight for the noisy feedback.\n2. The relationship between the proposed GABP algorithm and the AOG algorithm in [1] and the intuition behind the fast last-iterate convergence rates is not clearly discussed. These two algorithms are different (as shown in Appendix F) but share similar ideas. The anchoring term in both algorithms comes from the (implicit) Halpern iteration algorithm, which can not be run directly. The difference is that GABP views each step of Halpern iteration as a fixed point problem (Line 170) and uses an inner loop of $\\log (1/\\epsilon)$ steps to get an $\\epsilon$-approximation (this is called updating the reference strategy in the paper.); In contrast, AOG directly uses optimism to approximate the implicit update. This leads to GABP being a log factor slower than AOG in the full gradient setting. However, the approximating the fixed point approach is more robust in the noisy gradient setting due to strong monotonicity. Moreover, the potential function and the approximately non-increasing potential analysis are very similar to that used in [1]. If they are inspired by [1] then this should be acknowledged. \n\n[1] Doubly Optimal No-Regret Learning in Monotone Games, Cai and Zheng, ICML 2023.\n\nQUESTIONS:\n1. Could you explain Equation (5) and the related discussion? This inequality is not consistent with the text above. In particular, this inequality contains only $k(t-1)$, and the superscript $k(t)-1$ does not appear in the inequality. \n2. What is the intuition behind the $O(1/T^{1/7})$ convergence rate in the noisy gradient setting? In your opinion, what is the best possible rate that can be achieved by the current approach?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel algorithmic approach to enhance the convergence of first-order methods in the context of monotone games. The authors propose a payoff perturbation technique that introduces strong convexity to players' payoff functions, which is crucial for achieving last-iterate convergence. This technique is particularly designed to handle scenarios where the gradient of the payoff functions is monotone and potentially noisy. The paper presents a method called Gradient Ascent with Boosting Payoff Perturbation (GABP), which incorporates a unique perturbation into the payoff function and maintains a periodically re-initializing anchoring strategy. The authors demonstrate that GABP offers faster last-iterate convergence rates compared to existing algorithms, even in the presence of additive noise.\n\nSTRENGTHS:\nOriginality: The paper presents a unique perturbation technique that addresses the challenge of last-iterate convergence in monotone games. The proposed GABP algorithm is an innovative modification of Adaptively Perturbed Mirror Descent (APMD), offering improved convergence rates.\n\nQuality: The theoretical development is thorough, with rigorous proofs provided for the convergence rates of GABP in both full and noisy feedback settings. The paper also includes a detailed analysis of the algorithm's performance in terms of individual regret.\n\nClarity: The paper is well-organized, with clear explanations of the algorithm, theoretical results, and experimental setup. The use of pseudo-code for GABP aids in understanding the algorithm's implementation.\n\nSignificance: The work contributes to the field of online learning in games, providing a solution that is particularly relevant for applications such as Generative Adversarial Networks (GANs) and large language model fine-tuning, where last-iterate convergence is desirable.\n\nWEAKNESSES:\nExperimental Validation: While the paper provides empirical results, the experiments could be expanded to include a broader range of game types and noise levels to further validate the robustness and generalizability of GABP.\n\nComparison with State-of-the-Art: The paper compares GABP with APMD and Optimistic Gradient Ascent (OGA) but could benefit from a more comprehensive comparison with other existing methods in the literature to better situate its contributions.\n\nPractical Considerations: While the paper addresses the theoretical aspects of GABP, it could provide more insights into practical considerations, such as the implementation challenges and potential modifications needed for real-world applications.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on last-iterate convergence of  game dynamics. A payoff perturbation technique is proposed by adding strong convexity to players' payoff functions. Despite it is a well studied technqiue in learning in repeated games with first-order methods, especially in last-iterate convergence, a novel perturbation scheme introduced in this paper allows on to provide faster last-iterate convergence compared to previous works.\n\nSTRENGTHS:\nThis paper provides a relatively complete result containing last-iterate convergence rate of the proposed algorithm GABP in full feedback and noisy feedback. The faster rate of convergence is an improvement compared to existing works. Except for some weakness (will be stated later), the presentation of this paper is clear to understand. The authors have reviewed most related works to my best knowledge, so that the contributions claimed are easy to follow. Addition to theoretical works, this paper has provided experiments (sufficient in my opinion) showing the comparison of GABP and existing algorithms such as Adaptively perturbed gradient ascent and Optimistic gradient ascent.\n\nWEAKNESSES:\nOne obvious spot that should be added to improve the presentation is the following. The game considered in this paper is motivated by real-life examples. But the authors only give one example motivating monotone games. Part of contributions of the paper is claimed to be the study of two feedback models: full feedback and noisy feedback, but there is not specific examples and applications illustrating the importance of these settings. For sure readers can always find related works even just by googling the keywords, but providing concrete application scenes where the gradient of payoff can be achieved perfectly or only partially achievable gradients can be obtained is important, especially \"noisy feedback\" can be just a model of many cases.\n\nQUESTIONS:\nNoisy feedback has been studied in some previous works e.g. Mertikopoulos and Zhou, 2019, Hsieh et al. 2019. What is technical difference/challenge of this paper comparing to aforementioned ones?\n\nLIMITATIONS:\nThis is theory based paper, no potential negative impact will cause.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper studies first order methods to solve monotone games where the gradient of the payoff function is monotone in the strategy, along with additive noise. The authors introduce a payoff perturbation technique which introduces strong convexity to the to the payoff functions and thereby derive last iterate convergence rates.

STRENGTHS:
Overall the paper is well written and the method and results are interesting.

WEAKNESSES:
The authors should include a table which compares their paper with others in the literature. This would make it easier for the reader to place the results in context and see where improvements are made more easily.
(for example comparison to [Yoon and Ryu, 2021, Cai and Zheng, 2023] including constants)

QUESTIONS:
The idea of changing the anchoring slowly seems very interesting. How different is this approach from the two-scale GDA type algorithms that have been studies recently 
(For example, see Lin, Tianyi, Chi Jin, and Michael Jordan. "On gradient descent ascent for nonconvex-concave minimax problems." International Conference on Machine Learning. PMLR, 2020. and follow up papers).
Can a similar algorithm be derived from the updates of the algorithms proposed by the authors of this paper?

LIMITATIONS:
See Weaknesses and Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-29