Title: Adaptive Quasi-Newton and Anderson Acceleration Framework with Explicit Global Convergence Rates
Venue: NeurIPS 2023
Accepted: no
Keywords: Quasi Newton

Nonlinear acceleration

Accelerated

Convergence

Rate

Adaptive

Nonasymptotic
Abstract: Despite the impressive numerical performance of quasi-Newton and Anderson/nonlinear acceleration methods, their global convergence rates have remained elusive for over 50 years. This paper addresses this long-standing question by introducing a framework that derives novel and adaptive quasi-Newton or nonlinear/Anderson acceleration schemes. Under mild assumptions, the proposed iterative methods exhibit explicit, non-asymptotic convergence rates that blend those of gradient descent and Cubic Regularized Newton's method. Notably, these rates are achieved adaptively, as the method autonomously determines the optimal step size using a simple backtracking strategy. The proposed approach also includes an accelerated version that improves the convergence rate on convex functions. Numerical experiments demonstrate the efficiency of the proposed framework, even compared to a fine-tuned BFGS algorithm with line search.

Abstract

Despite the impressive numerical performance of quasi-Newton and Anderson/nonlinear acceleration methods, their global convergence rates have remained elusive for over 50 years. This paper addresses this long-standing question by introducing a framework that derives novel and adaptive quasi-Newton or nonlinear/Anderson acceleration schemes. Under mild assumptions, the proposed iterative methods exhibit explicit, non-asymptotic convergence rates that blend those of gradient descent and Cubic Regularized Newton's method. Notably, these rates are achieved adaptively, as the method autonomously determines the optimal step size using a simple backtracking strategy. The proposed approach also includes an accelerated version that improves the convergence rate on convex functions. Numerical experiments demonstrate the efficiency of the proposed framework, even compared to a fine-tuned BFGS algorithm with line search.

1 Introduction 13

Consider the problem of finding the minimizer x ⋆ of the unconstrained minimization problem 14

where d is the problem's dimension, and the function f has a Lipschitz continuous Hessian.

Assumption 1. The function f ( x ) has a Lipschitz continuous Hessian with a constant L ,

In this paper, ∥ . ∥ stands for the maximal singular value of a matrix and for the ℓ 2 norm for a vector. Many twice-differentiable problems like logistic or least-squares regression satisfy Assumption 1.

The Lipschitz continuity of the Hessian is crucial when analyzing second-order algorithms, as it extends the concept of smoothness to the second order. The groundbreaking work by Nesterov et al. [45] has sparked a renewed interest in second-order methods, revealing the remarkable convergence rate improvement of Newton's method on problems satisfying Assumption 1 when augmented with cubic regularization. For instance, if the problem is also convex, accelerated gradient descent typically achieves O ( 1 t 2 ) , while accelerated second-order methods achieve O ( 1 t 3 ) . Recent advancements have further pushed the boundaries, achieving even faster convergence rates of up to O ( 1 t 7 / 2 ) through the utilization of hybrid methods [42, 14] or direct acceleration of second-order methods [43, 27, 39].

Unfortunately, second-order methods may not always be feasible, particularly in high-dimensional 27 problems common in machine learning. The limitation is that exact second-order methods require 28 solving a linear system that involves the Hessian of the function f . This main limitation motivated 29 alternative approaches that balance the efficiency of second-order methods and the scalability of 30 first-order methods, such as inexact/subspace/stochastic techniques , nonlinear/Anderson acceleration , 31 and quasi-Newton methods. 32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

1.1 Contributions

Despite the impressive numerical performance of quasi-Newton methods and nonlinear acceleration schemes, there is currently no knowledge about their global explicit convergence rates. In fact, global convergence cannot be guaranteed without using either exact or Wolfe-line search techniques. This raises the following long-standing question that has remained unanswered for over 50 years :

What are the non-asymptotic global convergence rates of quasi-Newton and Anderson/nonlinear acceleration methods?

This paper provides a partial answer by introducing generic updates (see algorithms 1 to 3) that can be viewed as cubic-regularized quasi-Newton methods or regularized nonlinear acceleration schemes.

Under mild assumptions, the iterative methods constructed within the proposed framework (see algorithms 3 and 6) exhibit explicit, global and non-asymptotic convergence rates that interpolate the one of first order and second order methods (more details in appendix A):

Convergence rate on non-convex problems (Theorem 4): min i ∥∇ f ( x i ) ∥ ≤ O ( t -2 3 + t -1 3 ) ,

Convergence rate on (star-)convex problems (Theorems 5 and 6): f ( x t ) -f ⋆ ≤ O ( t -2 + t -1 ) ,

Accelerated rate on convex problems (Theorem 8): f ( x t ) -f ⋆ ≤ O ( t -3 + t -2 ) .

1.2 Related work

Inexact, subspace, and stochastic methods. Instead of explicitly computing the Hessian matrix and Newton's step, these methods compute an approximation using sampling [2], inexact Hessian computation [29, 19], or random subspaces [20, 31, 34]. By adopting a low-rank approximation for the Hessian, these approaches substantially reduce per-iteration costs without significantly compromising the convergence rate. The convergence speed in such cases often represents an interpolation between the rates observed in gradient descent methods and (cubic) Newton's method.

Nonlinear/Anderson acceleration. Nonlinear acceleration techniques, including Anderson acceleration [1], have a long standing history [3, 4, 28]. Driven by their promising empirical performance, they recently gained interest in their convergence analysis [61, 26, 60, 37, 66, 64, 69, 68, 53, 62, 63, 6, 57, 8, 54]. In essence, Anderson acceleration is an optimization technique that enhances convergence by extrapolating a sequence of iterates using a combination of previous gradients and corresponding iterates. Comprehensive reviews and analyses of these techniques can be found in notable sources such as [37, 7, 36, 35, 5, 17]. However, these methods do not generalize well outside quadratic minimization and their convergence rate can only be guaranteed asymptotically when using a line-search or regularization techniques [59, 65, 53].

Quasi-Newton methods. Quasi-Newton schemes are renowned for their exceptional efficiency in continuous optimization. These methods replace the exact Hessian matrix (or its inverse) in Newton's step with an approximation that is updated iteratively during the method's execution. The most widely used algorithms in this category include DFP [18, 25] and BFGS [58, 30, 24, 10, 9]. Most of the existing convergence results predominantly focus on the asymptotic super-linear rate of convergence [67, 32, 12, 11, 15, 22, 72, 70, 71]. However, recent research on quasi-Newton updates has unveiled explicit and non-asymptotic rates of convergence [49, 51, 50, 40, 41]. Nonetheless, these analyses suffer from several significant drawbacks, such as assuming an infinite memory size and/or requiring access to the Hessian matrix. These limitations fundamentally undermine the essence of quasi-Newton methods, which are typically designed to be Hessian-free and maintain low per-iteration cost through their low-memory requirement and low-rank structure.

Recently, Kamzolov et al. [38] introduced an adaptive regularization technique combined with 75 cubic regularization, with global, explicit (accelerated) convergence rates for any quasi-Newton 76 method. The method incorporates a backtracking line search on the secant inexactness inequality 77 that introduces a quadratic regularization. However, this algorithm relies on prior knowledge of the 78 Lipschitz constant specified in Assumption 1. Unfortunately, the paper does not provide an adaptive 79 method to find jointly the Lipschitz constant as well, as it is a priory too costly to know which 80 parameter to update. This aspect makes the method impractical in real-world scenarios. 81

Paper Organization Section 2 introduces the proposed novel generic updates and some essential 82 theoretical results. Section 3 presents the convergence analysis of the iterative algorithm, which 83 uses one of the proposed updates. Section 4 is dedicated to the accelerated version of the proposed 84 framework. Section 5 presents examples of methods generated by the proposed framework. 85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

2 Type-I and Type-II Step

This section first examines a remarkable property shared by quasi-Newton and Anderson acceleration: the sequence of iterates of these methods can be expressed as a combination of directions formed by previous iterates and the current gradient. Building upon this observation, section 2.1 investigates how to obtain second-order information without directly computing the Hessian of the function f by approximating the Hessian within the subspace formed by these directions. Subsequently, section 2.2 demonstrates how to utilize this approximation to establish an upper bound for the function f and its gradient norm ∥∇ f ( x ) ∥ . Minimizing these upper bounds, respectively, leads to a type-I and type-II method.

Motivation: what quasi-Newton and nonlinear acceleration schemes actually do? The BFGS update is a widely used quasi-Newton method for unconstrained optimization. It approximates the inverse Hessian matrix using updates based on previous gradients and iterates. The update reads

where H t is the approximation of the inverse Hessian at iteration t , h t is the step size, d t = x t -x t -1 is the step direction, g t = ∇ f ( x t ) - ∇ f ( x t -1 ) is the gradient difference. After unfolding the equation, the BFGS update can be seen as a combination of the d i 's and ∇ f ( x t ) ,

where P i are projection matrices in R d × d and α i are coefficients. Similar reasoning can be applied to other quasi-Newton formulas (see appendix B for more details).

This observation aligns with the principles of Anderson acceleration methods. Considering the same vectors d t and g t , Anderson acceleration updates x t +1 as:

where h t is the relaxation parameter, which can be seen as the step size of the method. As all x i 's belong to the span of previous gradients, the update is similar to (2), see appendix B for more details. This is not surprising, as it has been shown that Anderson acceleration can be viewed as a quasi-Newton method [23]. Some studies have explored the relationship between these two classes of optimization techniques and established strong connections in terms of their algorithmic behavior [23, 73, 56, 13].

Hence, quasi-Newton algorithms and nonlinear/Anderson acceleration methods utilize previous directions d i and the current gradient ∇ f ( x t ) in subsequent iterations. However, their convergence is guaranteed only if a line search is used, and their convergence speed is heavily dependent on H 0 (quasi-Newton) or h t (Anderson acceleration) [48].

2.1 Error Bounds on the Hessian-Vector Product Approximation by a Difference of Gradients

Consider the following d × N matrices that represent the algorithm's memory ,

For example, to mimic quasi-Newton techniques, the matrices Y and Z can be defined such that,

Motivated by (2), this paper studies the following update, defined as a linear combination of the previous directions d i ,

The objective is to determine the optimal coefficients α based on the information contained in the matrices defined in (3). Notably, the absence of the gradient in the update (4) distinguishes this

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

approach from (2), allowing for the development of an adaptive method that eliminates the need for an initial matrix H 0 (quasi-Newton methods) or a mixing parameter h t (Anderson acceleration).

Under assumption (1), the following bounds hold for all x, y, z, x + ∈ R d [45],

The accuracy of the estimation of the matrix ∇ 2 f ( x ) , depends on the error vector ε ,

The following Theorem 1 explicitly bounds the error of approximating ∇ 2 f ( x ) D by G .

Theorem 1. Let the function f satisfy Assumption 1. Let x + be defined as in (4) and the matrices D, G be defined as in (3) and vector ε as in (7). Then, for all w ∈ R d and α ∈ R N ,

Proof sketch and interpretation. The theorem states that the Hessian-vector product ∇ 2 f ( x )( y -z ) can be approximated by the difference of gradients ∇ f ( y ) - ∇ f ( z ) , providing a cost-effective approach to estimate ∇ 2 f without computing it. This property is the basis of quasi-Newton methods. The detailed proof can be found in appendix F. The main idea of the proof is as follows. From (5) with y = y i and z = z i , writing d i = y i -z i , and Assumption 1,

The first term in ε i bounds the error of (5), while the second comes from the distance between (5) and the current point x where the Hessian is estimated. Then, it suffices to combine the inequalities with coefficients α to obtain Theorem 1.

2.2 Type I and Type II Inequalities and Methods

In the literature, Type-I methods often refer to algorithms that aim to minimize the function value f ( x ) , while type-II methods minimize the gradient norm ∥∇ f ( x ) ∥ [23, 73, 13]. Applying the bounds (6) and (5) to the update in (4) yields the following Type-I and Type-II upper bounds, respectively.

Theorem 2. Let the function f satisfy Assumption 1. Let x + be defined as in (4), the matrices D, G be defined as in (3) and ε be defined as in (7). Then, for all α ∈ R N ,

The proof can be found in appendix F. Minimizing eqs. (10) and (11) leads to algorithms 1 and 2, respectively, whose constant L is replaced by a parameter M , found by backtracking line-search. A study of the (strong) link between these proposed algorithms and nonlinear/Anderson acceleration and quasi-Newton methods can be found in appendix B.

Solving the sub-problems In algorithms 1 and 2, the coefficients α are computed by solving a minimization sub-problem in O ( N 3 + Nd ) (see appendix C for more details). Usually, N is rather small (e.g. between 5 and 100 ); hence solving the subproblem is negligible compared to computing a new gradient ∇ f ( x ) . Here is the summary:

In algorithm 1 , the subproblem can be solved easily by a convex problem in two variables, which involves an eigenvalue decomposition of the matrix H ∈ R N × N [45].

In algorithm 2 , the subproblem can be cast into a linear-quadratic problem of O ( N ) variables and constraints that can be solved efficiently with SDP solvers (e.g., SDPT3 ).

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

Algorithm 1 Type-I Subroutine with Backtracking Line-search

Require: First-order oracle for f , matrices G, D , vector ε , iterate x , initial smoothness M

1: Initialize M ← 0 2

0 . M

2: do

3: M ← 2 M and H ← G T D + D T G 2 +I N M ∥ D ∥∥ ε ∥ 2

4:

α

⋆

arg min

α

5: x + ← x + Dα

1

2

Dα

6

6: while f ( x + ) ≥ f ( x ) + ∇ f ( x ) T Dα ⋆ + 1 2 [ α ⋆ ] T Hα ⋆ + M ∥ Dα ⋆ ∥ 3 6

7: return x + , M

Algorithm 2 Type-II Subroutine with Backtracking Line-search

Same as algorithm 1, but minimize and check the upper bound (11) instead of (10) on lines 4 and 6.

3 Iterative Type-I Method: Framework and Rates of Convergences

The rest of the paper analyzes the convergence rate of methods that use algorithm 1 as a subroutine; see algorithm 3. The analysis of methods that uses algorithm 2 is left for future work.

3.1 Main Assumptions and Design Requirements

This section lists the important assumptions on the function f . Some subsequent results require an upper bound on the radius of the sub-level set of f at f ( x 0 ) .

Assumption 2. The radius of the sub-level set { x : f ( x ) ≤ f ( x 0 ) } is bounded by R < ∞ .

To ensure the convergence toward f ( x ⋆ ) , some results require f to be star-convex or convex.

Assumption 3. The function f is star convex if, for all x ∈ R d and ∀ τ ∈ [0 , 1] ,

Assumption 4. The function f is convex if, for all y, z ∈ R d , f ( y ) ≥ f ( z ) + ∇ f ( z )( y -z ) .

The matrices Y, Z, D must meet some conditions listed below as "requirements" (see section 5 for details). All convergence results rely on one of these conditions on the projector onto span ( D ) ,

Requirement 1a. For all t , the projector P of the stochastic matrix D satisfies E [ P ] = I

Requirement 1b. For all t , the projector P t satisfies P t ∇ f ( x t ) = ∇ f ( x t )

t t t N d . .

The first condition guarantees that, in expectation, the matrix D t spans partially the gradient ∇ f ( x t ) , since E [ P t ∇ f ( x t )] = N d ∇ f ( x t ) . The second condition simply requires the possibility to move towards the current gradient when taking the step x + Dα . This condition resonates with the idea presented in (2), where the step x + -x combines previous directions and the current gradient ∇ f ( x t ) .

In addition, it is required that the norm of ∥ ε ∥ does not grow too quickly, hence the next assumption.

Requirement 2. For all t , the relative error ∥ ε t ∥ ∥ D t ∥ is bounded by δ .

The Requirement 2 is also non-restrictive, as it simply prevents taking secant equations at y i -z i and z i -x i too far apart. Most of the time, δ satisfies δ ≤ O ( R ) .

Finally, the condition number of the matrix D also has to be bounded.

Requirement 3. For all t , the matrix D t is full-column rank, which implies that D T t D t is invertible. In addition, its condition number κ D t def = √ ∥ D T t D t ∥∥ ( D T t D t ) -1 ∥ is bounded by κ .

The condition on the rank of D is not overly restrictive. In most practical scenarios, this condition is 180 typically satisfied without issue. However, the second condition might be hard to meet, but section 5 181 studies strategies that prevent κ D from exploding by taking orthogonal directions or pruning D . 182

f

(

x

) +

∇

f

(

x

)

T

Dα

+

α

T

Hα

+

M

∥

∥

3

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

Algorithm 3 Generic Iterative Type-I Methods

3.2 Rates of Convergence

When f satisfies Assumption 1, algorithm 3 ensures a minimal function decrease at each step. Theorem 3. Let f satisfy Assumption 1. Then, at each iteration t ≥ 0 , algorithm 3 achieves

Under some mild assumptions, algorithm 3 converges to a critical point for non-convex functions.

Theorem 4. Let f satisfy Assumption 1, and assume that f is bounded below by f ∗ . Let Requirements 1b to 3 hold, and M t ≥ M min . Then, algorithm 3 starting at x 0 with M 0 achieves

Going further, algorithm 3 converges to an optimum when the function is star-convex.

Theorem 5. Assume f satisfy Assumptions 1 to 3. Let Requirements 1b to 3 hold. Then, algorithm 3 starting at x 0 with M 0 achieves, for t ≥ 1 ,

Finally, the next theorem shows that when algorithm 3 uses a stochastic D that satisfies Requirement 1a, then f ( x t ) also converges in expectation to f ( x ⋆ ) when f is convex.

Theorem 6. Assume f satisfy Assumptions 1, 2 and 4. Let Requirements 1a, 2 and 3 hold. Then, in expectation over the matrices D i , algorithm 3 starting at x 0 with M 0 achieves, for t ≥ 1 ,

Interpretation The rates presented in Theorems 4 to 6 combine the ones of cubic regularized Newton's method and gradient descent (or coordinate descent, as in Theorem 6) for functions with Lipschitz-continuous Hessian. As C 1 , C 2 , and C 3 decrease, the rates approach those of cubic Newton.

The constants C 1 , C 2 , and C 3 quantify the error of approximating D ∇ 2 f ( x ) D by H in (10) into two terms. The first represents the error made by approximating ∇ 2 f ( x ) D by G , while the second describes the low-rank approximation of ∇ 2 f ( x ) in the subspace spanned by the columns of D . The approximation is more explicit in C 3 , where increasing N reduces the constant up to N = d .

To retrieve the convergence rate of Newton's method with cubic regularization, the approximation needs to satisfy three properties: 1) the points contained in Y t and Z t must be close to each other, and to x t to reduce δ and ∥ ε ∥ ; 2) the condition number of D should be close to 1 to reduce κ ; 3) D should span a maximum dimension in R d to improve the approximation of ∇ 2 f ( x ) by P ∇ 2 f ( x ) P .

For example, Z t = x t 1 T N , D t = h I N with h small, and Y t = Z t + D t achieve these conditions. This (naive) strategy estimates all directional second derivatives with a finite difference for all coordinates and is equivalent to performing a Newton's step in terms of complexity.

210

211

212

213

214

215

216

217

218

219

220

221

222

Algorithm 4 Type-I subroutine with backtracking for the accelerated method

Require: First-order oracle f , matrices G, D , vector ε , iterate x , smoothness M 0 , minimal norm ∆ Initialize M ← M 0 2 , γ ← 1 4 ∥ ε ∥ ∥ D ∥ ( 1 + κ 2 D ) , ExitFlag ← False while ExitFlag is False do

end while

return x + , α , M , γ , ExitFlag

Algorithm 5 Adaptive Accelerated Type-I Algorithm (Sketch, see appendix D for the full version)

4 Accelerated Algorithm for Convex Functions

This section introduces algorithm 5, an accelerated variant of algorithm 3 for convex functions, designed using the estimate sequence technique from [43]. It consists in iteratively building a function Φ t ( x ) , a regularized lower bound on f , that reads

where λ (1 , 2) t are non-decreasing. The key aspects of acceleration are as follows (see section 4 for more details): 1) The accelerated algorithm makes a step at a linear combination between v t , the optimum of Φ t , and the previous iterate x t . 2) It uses a modified version of algorithm 1, see algorithm 4. 3) Under some conditions, the step size can be considered as "large", i.e., similar to a cubic-Newton step. The ∆ > 0 ensures the step is sufficiently large to ensure theoretical convergence - but setting ∆ = 0 does not seem to impact the numerical convergence. The presence of both small and large steps is crucial to obtain the theoretical rate of convergence.

Theorem 7. Assume f satisfy Assumptions 1, 2 and 4. Let Requirements 1b to 3 hold. Then, algorithm 5 starting at x 0 with M 0 achieves, for all ∆ > 0 and for t ≥ 1 ,

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

Interpretation The interpretation is similar to the one from Section 3. Ignoring ˜ λ (1 , 2) , the rate of Theorem 7 combines the one of accelerated gradient and accelerated cubic Newton [44, 43]. The constant M 0 blends the Lipschitz constant of the Hessian L with its approximation errors (2 κ 2 + κ ) δ and ∥ ( I -P ) ∇ 2 f ( x ) ∥ . The better the Hessian is approximated, the smaller the constant.

5 Some update strategies for matrices Y, Z, D, G

The framework presented in this paper is characterized by its generality, requiring only minimal assumptions on the matrix D and vector ε . This section explores different strategies for updating the matrices from (3), which can be classified into two categories: online and batch techniques .

Recommended method. Among all the methods presented in this section, the most promising technique seems to be the Orthogonal Forward Estimates Only , as it ensures that the condition number κ D = 1 and the norm of the error vector ∥ ε ∥ is small.

5.1 Online Techniques

The online technique updates the matrix D while algorithms 3 and 5 are running. To achieve Requirement 1b, the method employs either a steepest or orthogonal forward estimate, defined as

Then, it include x t + 1 2 -x t in the matrix D t . The projector P t -1 is defined in (12), and parameter h can be a fixed small value (e.g., h = 10 -9 ). This section investigates three different strategies for storing past information: Iterates only , Forward Estimates Only , and Greedy , listed below.

Iterates only: In the case of quasi-Newton updates and Nonlinear/Anderson acceleration, the iterates are constructed using the equation x t +1 -x t ∈ ∇ f ( x t ) + span { x t -i +1 -x t -i } i =1 ...N . The update draws inspiration from this observation. However, it does not provide control over the condition number of D t or the norm ∥ ε ∥ . To address this, one can either accept a potentially high condition number or remove the oldest points in D and G until the condition number is bounded (e.g., κ = 10 9 ).

Forward Estimates Only: This method provides more control over the iterates added to Y and Z . When using the orthogonal technique to compute x i + 1 2 reduces the constants in Theorems 4, 5 and 7: the condition number of D is equal to 1 as D T D = h 2 I , and the norm of ε is small ( ∥ ε ∥ ≤ O ( h ) ).

Greedy: The greedy approach involves storing both the iterates and the forward approximations. It shares the same drawback as the Iterates only strategy but retains at least the most recent information about the Hessian-vector product approximation, thereby reducing the ∥ z i -x i ∥ term in ε (7).

5.2 Batch Techniques

Instead of making individual updates, an alternative approach is to compute them collectively, centered on x t . This technique generates a matrix D t consisting of N orthogonal directions d 1 , · · · , d N of norm h . The corresponding Y t , Z t , G t matrices are then defined as follows:

This section explores two batch techniques that generate orthogonal directions: Orthogonalization and Random Subspace . Both lead to δ = 3 h and κ = 1 in Requirements 2 and 3. However, they require N additional gradient computations at each iteration (instead of one for the online techniques). For clarity, in the experiments, only the Greedy version is considered.

Orthogonalization: This technique involves using any online technique discussed in the previous section and storing the directions in a matrix ˜ D t . Then, it constructs the matrices D t by performing an orthogonalization procedure on ˜ D t , such as the QR algorithm. This approach provides Hessian estimates in relevant directions, which can be more beneficial than random ones.

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

0

100

200

300

400

500

10 -10

10 -5

10 0

Iterate Only

Forward Estimate Only

Greedy

Random subspace

Orthogonalized greedy

Accelerated Forward Only

0

100

200

300

400

500

10 -10

10 -5

10 0

Iterate Only

Forward Estimate Only

Greedy

Random subspace

Orthogonalized greedy

Accelerated Forward Only

Random Subspace: Inspired by [34], this technique randomly generates D t at each iteration by either taking D t to be N random (rescaled) canonical vectors or by using the Q matrix from the QR decomposition of a random N × D matrix. This ensures that D t satisfies Requirement 1a. For clarity, in the experiments, only the QR version is considered.

6 Numerical Experiments

This section compares the methods generated by this paper's framework to the fine-tuned ℓ -BFGS algorithm from minFunc [52]. More experiments are conducted in appendix E. The tested methods are the Type-I iterative algorithms (algorithm 3 with the techniques from section 5). The step size of the forward estimation was set to h = 10 -9 , and the condition number κ D t is maintained below κ = 10 9 with the iterates only and Greedy techniques. The accelerated algorithm 6 is used only with the Forward Estimates Only technique. The compared methods are evaluated on a logistic regression problem with no regularization on the Madelon UCI dataset [33]. The results are shown in fig. 1.

Regarding the number of iterations, the greedy orthogonalized version outperforms the others due to the orthogonality of directions (resulting in a condition number of one) and the meaningfulness of previous gradients/iterates. However, in terms of gradient oracle calls, the recommended method, orthogonal forward iterates only , achieves the best performance by striking a balance between the cost per iteration (only two gradients per iteration) and efficiency (small and orthogonal directions, reducing theoretical constants). Surprisingly, the accelerated method's performance is suboptimal, possibly because it tightens the theoretical analysis, diminishing its inherent adaptivity.

7 Conclusion, Limitation, and Future work

This paper introduces a generic framework for developing novel quasi-Newton and Anderson/Nonlinear acceleration schemes, offering a global convergence rate in various scenarios, including accelerated convergence on convex functions, with minimal assumptions and design requirements.

One limitation of the current approach is requiring an additional gradient step for the forward estimate , as discussed in Section 5. However, this forward estimate is crucial in enabling the algorithm's adaptivity, eliminating the need to initialize a matrix H 0 (quasi-Newton) or employ a mixing parameter h 0 (Anderson acceleration).

In future research, although unsuitable for large-scale problems, the method presented in this paper can achieve super-linear convergence rates, as with infinite memory, they would be as fast as cubic Newton methods. Utilizing the average-case analysis framework from existing literature, such as [47, 55, 21, 16, 46], could also improve the constants in Theorems 4 and 5 to match those in Theorem 6. Furthermore, exploring convergence rates for type-2 methods, which are believed to be effective for variational inequalities, is a worthwhile direction.

Ultimately, the results presented in this paper open new avenues for researchs. It may also provide a 296 potential foundation for investigating additional properties of existing quasi-Newton methods and 297 may even lead to the discovery of convergence rates for an adaptive, cubic-regularized BFGS variant. 298

299

300

301

References

Donald G Anderson. 'Iterative procedures for nonlinear integral equations'. In: Journal of the ACM (JACM) 12.4 (1965), pp. 547-560.

Kimon Antonakopoulos, Ali Kavis, and Volkan Cevher. 'Extra-Newton: A First Approach to 302 Noise-Adaptive Accelerated Second-Order Methods'. In: arXiv preprint arXiv:2211.01832 303 (2022). 304

Claude Brezinski. 'Application de l' ε -algorithme à la résolution des systèmes non linéaires'. 305 In: Comptes Rendus de l'Académie des Sciences de Paris 271.A (1970), pp. 1174-1177. 306

Claude Brezinski. 'Sur un algorithme de résolution des systèmes non linéaires'. In: Comptes 307 Rendus de l'Académie des Sciences de Paris 272.A (1971), pp. 145-148. 308

Claude Brezinski and Michela Redivo-Zaglia. 'The genesis and early developments of Aitken's 309 process, Shanks' transformation, the ε -algorithm, and related fixed point methods'. In: Nu310 merical Algorithms 80.1 (2019), pp. 11-133. 311

Claude Brezinski, Michela Redivo-Zaglia, and Yousef Saad. 'Shanks sequence transformations 312 and Anderson acceleration'. In: SIAM Review 60.3 (2018), pp. 646-669. 313

Claude Brezinski and M Redivo Zaglia. Extrapolation methods: theory and practice . Elsevier, 314 1991. 315

Claude Brezinski et al. 'Shanks and Anderson-type acceleration techniques for systems of 316 nonlinear equations'. In: arXiv:2007.05716 (2020). 317

Charles G Broyden. 'The convergence of a class of double-rank minimization algorithms: 2. 318 The new algorithm'. In: IMA journal of applied mathematics 6.3 (1970), pp. 222-231. 319

Charles George Broyden. 'The convergence of a class of double-rank minimization algorithms 320 1. general considerations'. In: IMA Journal of Applied Mathematics 6.1 (1970), pp. 76-90. 321

Richard H Byrd and Jorge Nocedal. 'A tool for the analysis of quasi-Newton methods with 322 application to unconstrained minimization'. In: SIAM Journal on Numerical Analysis 26.3 323 (1989), pp. 727-739. 324

325

326

327

Richard H Byrd, Jorge Nocedal, and Ya-Xiang Yuan. 'Global convergence of a cass of quasiNewton methods on convex problems'. In: SIAM Journal on Numerical Analysis 24.5 (1987), pp. 1171-1190.

Marco Canini and Peter Richtárik. 'Direct nonlinear acceleration'. In: Operational Research 328 2192 (2022), p. 4406. 329

330

331

Yair Carmon et al. 'Recapp: Crafting a more efficient catalyst for convex optimization'. In: International Conference on Machine Learning . PMLR. 2022, pp. 2658-2685.

Andrew R Conn, Nicholas IM Gould, and Ph L Toint. 'Convergence of quasi-Newton matrices 332 generated by the symmetric rank one update'. In: Mathematical programming 50.1-3 (1991), 333 pp. 177-195. 334

Leonardo Cunha et al. 'Only tails matter: Average-Case Universality and Robustness in the 335 Convex Regime'. In: 2022. 336

Alexandre d'Aspremont, Damien Scieur, Adrien Taylor, et al. 'Acceleration methods'. In: 337 Foundations and Trends® in Optimization 5.1-2 (2021), pp. 1-245. 338

William C Davidon. 'Variable metric method for minimization'. In: SIAM Journal on Opti339 mization 1.1 (1991), pp. 1-17. 340

Nikita Doikov, El Mahdi Chayti, and Martin Jaggi. 'Second-order optimization with lazy 341 Hessians'. In: arXiv preprint arXiv:2212.00781 (2022). 342

Nikita Doikov, Peter Richtárik, et al. 'Randomized block cubic Newton method'. In: Interna343 tional Conference on Machine Learning . PMLR. 2018, pp. 1290-1298. 344

Carles Domingo-Enrich, Fabian Pedregosa, and Damien Scieur. 'Average-case acceleration 345 for bilinear games and normal matrices'. In: arXiv preprint arXiv:2010.02076 (2020). 346

John R Engels and Hector J Martinez. 'Local and superlinear convergence for partially known 347 quasi-Newton methods'. In: SIAM Journal on Optimization 1.1 (1991), pp. 42-56. 348

Haw-Ren Fang and Yousef Saad. 'Two classes of multisecant methods for nonlinear accelera349 tion'. In: Numerical Linear Algebra with Applications 16.3 (2009), pp. 197-221. 350

Roger Fletcher. 'A new approach to variable metric algorithms'. In: The computer journal 351 13.3 (1970), pp. 317-322. 352

Roger Fletcher and Michael JD Powell. 'A rapidly convergent descent method for minimiza353 tion'. In: The computer journal 6.2 (1963), pp. 163-168. 354

William F Ford and Avram Sidi. 'Recursive algorithms for vector extrapolation methods'. In: 355 Applied numerical mathematics 4.6 (1988), pp. 477-489. 356

Alexander Gasnikov et al. 'Near optimal methods for minimizing convex functions with 357 lipschitz p -th derivatives'. In: Conference on Learning Theory . PMLR. 2019, pp. 1392-1393. 358

Eckart Gekeler. 'On the solution of systems of equations by the epsilon algorithm of Wynn'. 359 In: Mathematics of Computation 26.118 (1972), pp. 427-436. 360

Saeed Ghadimi, Han Liu, and Tong Zhang. 'Second-order methods with cubic regularization 361 under inexact information'. In: arXiv preprint arXiv:1710.05782 (2017). 362

Donald Goldfarb. 'A family of variable-metric methods derived by variational means'. In: 363 Mathematics of computation 24.109 (1970), pp. 23-26. 364

Robert Gower et al. 'Rsn: Randomized subspace newton'. In: Advances in Neural Information 365 Processing Systems 32 (2019). 366

Andreas Griewank and Ph L Toint. 'Local convergence analysis for partitioned quasi-Newton 367 updates'. In: Numerische Mathematik 39.3 (1982), pp. 429-448. 368

Isabelle Guyon. 'Design of experiments of the NIPS 2003 variable selection benchmark'. In: 369 NIPS 2003 workshop on feature extraction and feature selection . Vol. 253. 2003. 370

Filip Hanzely et al. 'Stochastic subspace cubic Newton method'. In: International Conference 371 on Machine Learning . PMLR. 2020, pp. 4027-4038. 372

K Jbilou and H Sadok. 'Vector extrapolation methods. Applications and numerical com373 parison'. In: Journal of Computational and Applied Mathematics 122.1-2 (2000), pp. 149374 165. 375

376

377

Khalide Jbilou and Hassane Sadok. 'Analysis of some vector extrapolation methods for solving systems of linear equations'. In: Numerische Mathematik 70.1 (1995), pp. 73-89.

378

379

380

381

382

383

384

385

386

387

388

389

Khalide Jbilou and Hassane Sadok. 'Some results about vector extrapolation methods and related fixed-point iterations'. In: Journal of Computational and Applied Mathematics 36.3 (1991), pp. 385-398.

Dmitry Kamzolov et al. 'Accelerated Adaptive Cubic Regularized Quasi-Newton Methods'. In: arXiv preprint arXiv:2302.04987 (2023).

Dmitry Kovalev and Alexander Gasnikov. 'The first optimal acceleration of high-order methods in smooth convex optimization'. In: arXiv preprint arXiv:2205.09647 (2022).

Dachao Lin, Haishan Ye, and Zhihua Zhang. 'Explicit convergence rates of greedy and random quasi-Newton methods'. In: Journal of Machine Learning Research 23.162 (2022), pp. 1-40.

Dachao Lin, Haishan Ye, and Zhihua Zhang. 'Greedy and random quasi-newton methods with faster explicit superlinear convergence'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 6646-6657.

Renato DC Monteiro and Benar Fux Svaiter. 'An accelerated hybrid proximal extragradient 390 method for convex optimization and its implications to second-order methods'. In: SIAM 391 Journal on Optimization 23.2 (2013), pp. 1092-1125. 392

Yurii Nesterov. 'Accelerating the cubic regularization of Newton's method on convex prob393 lems'. In: Mathematical Programming 112.1 (2008), pp. 159-181. 394

Yurii Nesterov. Introductory lectures on convex optimization . Springer, 2004. 395

Yurii Nesterov and Boris T Polyak. 'Cubic regularization of Newton method and its global 396 performance'. In: Mathematical Programming 108.1 (2006), pp. 177-205. 397

Courtney Paquette et al. 'Halting Time is predictable for large models: A universality property 398 and average-case analysis'. In: Foundations of Computational Mathematics (2022). 399

Fabian Pedregosa and Damien Scieur. 'Acceleration through spectral density estimation'. In: 400 Proceedings of the 37th International Conference on Machine Learning (ICML) . 2020. 401

MJD Powell. 'How bad are the BFGS and DFP methods when the objective function is 402 quadratic?' In: Mathematical Programming 34 (1986), pp. 34-47. 403

Anton Rodomanov and Yurii Nesterov. 'Greedy quasi-Newton methods with explicit superlin404 ear convergence'. In: SIAM Journal on Optimization 31.1 (2021), pp. 785-811. 405

Anton Rodomanov and Yurii Nesterov. 'New results on superlinear convergence of classical 406 quasi-Newton methods'. In: Journal of optimization theory and applications 188 (2021), 407 pp. 744-769. 408

Anton Rodomanov and Yurii Nesterov. 'Rates of superlinear convergence for classical quasi409 Newton methods'. In: Mathematical Programming (2021), pp. 1-32. 410

411

412

Mark Schmidt. 'minFunc: unconstrained differentiable multivariate optimization in Matlab'. In: Software available at http://www. cs. ubc. ca/˜ schmidtm/Software/minFunc. htm (2005).

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

Damien Scieur, Alexandre d'Aspremont, and Francis Bach. 'Regularized nonlinear acceleration'. In: Advances in Neural Information Processing Systems (NIPS) . 2016.

Damien Scieur, Alexandre d'Aspremont, and Francis Bach. 'Regularized nonlinear acceleration'. In: Mathematical Programming (2020).

Damien Scieur and Fabian Pedregosa. 'Universal Asymptotic Optimality of Polyak Momentum'. In: Proceedings of the 37th International Conference on Machine Learning (ICML) . 2020.

Damien Scieur et al. 'Generalization of Quasi-Newton methods: application to robust symmetric multisecant updates'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2021, pp. 550-558.

Damien Scieur et al. 'Online Regularized Nonlinear Acceleration'. In: arXiv:1805.09639 (2018).

David F Shanno. 'Conditioning of quasi-Newton methods for function minimization'. In: Mathematics of computation 24.111 (1970), pp. 647-656.

Avram Sidi. 'Convergence and stability properties of minimal polynomial and reduced rank extrapolation algorithms'. In: SIAM Journal on Numerical Analysis 23.1 (1986), pp. 197-209.

Avram Sidi. 'Efficient implementation of minimal polynomial and reduced rank extrapolation methods'. In: Journal of Computational and Applied Mathematics 36.3 (1991), pp. 305-337.

Avram Sidi. 'Extrapolation vs. projection methods for linear systems of equations'. In: Journal of Computational and Applied Mathematics 22.1 (1988), pp. 71-88.

Avram Sidi. 'Minimal polynomial and reduced rank extrapolation methods are related'. In: Advances in Computational Mathematics 43.1 (2017), pp. 151-170.

Avram Sidi. Vector extrapolation methods with applications . SIAM, 2017.

Avram Sidi. 'Vector extrapolation methods with applications to solution of large systems of equations and to PageRank computations'. In: Computers & Mathematics with Applications 56.1 (2008), pp. 1-24.

Avram Sidi and Jacob Bridger. 'Convergence and stability analyses for some vector extrapolation methods in the presence of defective iteration matrices'. In: Journal of Computational and Applied Mathematics 22.1 (1988), pp. 35-61.

Avram Sidi and Yair Shapira. 'Upper bounds for convergence rates of acceleration methods 442 with initial iterations'. In: Numerical Algorithms 18.2 (1998), pp. 113-132. 443

Andrzej Stachurski. 'Superlinear convergence of Broyden's bounded θ -class of methods'. In: 444 Mathematical Programming 20.1 (1981), pp. 196-212. 445

Alex Toth and CT Kelley. 'Convergence analysis for Anderson acceleration'. In: SIAM Journal 446 on Numerical Analysis 53.2 (2015), pp. 805-819. 447

Homer F Walker and Peng Ni. 'Anderson acceleration for fixed-point iterations'. In: SIAM 448 Journal on Numerical Analysis 49.4 (2011), pp. 1715-1735. 449

Zengxin Wei et al. 'The superlinear convergence of a modified BFGS-type method for uncon450 strained optimization'. In: Computational optimization and applications 29 (2004), pp. 315451 332. 452

Hiroshi Yabe, Hideho Ogasawara, and Masayuki Yoshino. 'Local and superlinear convergence 453 of quasi-Newton methods based on modified secant conditions'. In: Journal of Computational 454 and Applied Mathematics 205.1 (2007), pp. 617-632. 455

Hiroshi Yabe and Naokazu Yamaki. 'Local and superlinear convergence of structured quasi456 Newton methods for nonlinear optimization'. In: Journal of the Operations Research Society 457 of Japan 39.4 (1996), pp. 541-557. 458

Junzi Zhang, Brendan O'Donoghue, and Stephen Boyd. 'Globally convergent type-I Anderson 459 acceleration for nonsmooth fixed-point iterations'. In: SIAM Journal on Optimization 30.4 460 (2020), pp. 3170-3197. 461

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents first-order iterative methods for solving unconstrained minimization problem. The connection between the proposed methods, quasi-Newton and Anderson accelaration methods are illustrated, which gives an insight of the motivation. Under certain assumptions, the methods exhibit explicit gobal non-asympotic convergence rates adaptively using backtracking strategy. The effiency of the propsed methods is compared to a fine-tuned BFGS algorithm with line search in the numerical experiments.\n\nSTRENGTHS:\n- The problem is well-motivated.\n\n- Under certain assumptions, the proposed adaptive first-order methods achieve explicit global convergence rates that blend those of gradient descent and cubic regularized Newton's method.\n\n- The inputs of the algorithms are carefully discussed. It should be clear for others to implement.\n\n- The algorithm complexity is analyzed and compared in the numerical experiments.\n\nWEAKNESSES:\n- Too many assumptions and requirements in the theoretical part. Can those requirements be verified? Seems the requirements are post to let the proof go through.  \n\n- The algorithm setting for the numerical experiments in section 6 is a bit confusing. For example, what online techniques are used for 'Iterate Only', 'Accelareted Forward Only', 'Forward Estimate Only' and 'Greedy'?\n\n- The performance of the accelarated algorithm is suboptimal and unstable.\n\nQUESTIONS:\n- Section 1.1 Contributiions: \n  - any reference to support the statement in bold: that has remained unanswered for over 50 years.\n  - for strongly convex cases, can the author prove the linear or super-linear convergence rate?\n- What is the motivation to use the update formula for $H$ in line 3 of Algorithm 1? \n- When taking orthogonal directions for $D$, the orthogonality is metric-dependent. For example, in conjugate gradient methods, it is $A$-orthogoal. \n- What is the definition of $r_i$ in the equation below line 104?\n- In Requirement 1a: where is the randomness of matrix $D_t$? Before that, all presentaion is for determinstic algorithms. \n- What is the step size $h$ in the methods below line 236? By the definition of steepest, it is chosen by the exact line search. But a small $h=10^{-9}$ is used. Then it is called gradient descent not steepest gradient method. And what is the reason to choose  such a tiny step size. \n- What is the 'Greedy' refers to in line 258? \n- Should the right hand side of inquality in Theorem 5 be written as $f(x_0) - f^\\star$ in the numerator of the first term? \n- Typos: line 7, 'cubic regularized'; line 197, 'Theorem 6'; line 237, 'includes'.\n\nLIMITATIONS:\n- The performance of the accelarated algorithm is suboptimal and unstable.\n- Analysis is provided only for Algorithm 1 not Algorithm 2 while the later seems better.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generic framework for developing new iterative schemes for smooth optimization in the deterministic setting. The derived new methods show some similarity to quasi-Newton methods and Anderson acceleration, while using a backtracking line search for estimating the Lipschitz constant and having step sizes adaptively determined by minimizing an upper bound of the objective function or the gradient norm. The explicit, global and non-asymptotic convergence rates are established for one type of the derived methods. Numerical results are presented for some specific implementations of the framework.\n\nSTRENGTHS:\nThe paper proposes a novel framework inspired by the recent development of cubic regularized Newton's method. This framework leads to several optimization methods that can be viewed as new variants of quasi-Newton methods and Anderson acceleration. The theory seems to be solid. The explicit, non-asymptotic convergence rates are established under different assumptions of the optimization problems, justifying the introduced techniques. The content is informative and the main idea is easy to follow. Numerical results are presented to support some claims of the paper.\n\nWEAKNESSES:\nThe weaknesses are listed as follows:\n\n1. The theory requires the Lipschitz continuity of the Hessian in $\\mathcal{R}^d$, which is a strong assumption in practice.\n\n2. The theorems only consider the iteration number towards convergence, which may overestimate the efficiency of the algorithms. Since the number of function evaluations can be large in each iteration, it needs to be clarified whether the claimed convergence rates are still valid when taking these hidden calculations into account.\n\n3. The biggest drawback of the proposed methods is their suitability for solving high-dimensional problems. To well approximate the Hessian which is critical for the convergence, the memory usage needs to be large, which can cause failure of the algorithms when the memory resource is limited. What's worse, the algorithms are very complicated. They use a backtracking line search to estimate the Lipschitz constant and solve a minimization problem to determine the stepsizes in each inner step of the line search. It is unknown how many times the minimization problem needs to be solved during each iteration. It is likely that the computational cost of each iteration is much higher than the classical quasi-Newton methods or Anderson acceleration. However, there is no discussion about this issue.\n\n4. The algorithms were only tested for solving small-scale logistic regression problems. Some numerical results do not support the theory. For instance, the accelerated method does not exhibit any acceleration in practice. In many cases, the simple BFGS with the default setting still achieves the best performance, while the proposed methods are less efficient due to higher costs.\n\n5. Since many additional calculations are hidden in the subroutines of the algorithms, it is more convincing to report comparisons of the considered algorithms with respect to running time. However, these results are missing.\n\nThere are also some other concerns about the paper:\n\n1. In Section 2, the paper claims that quasi-Newton methods and Anderson acceleration share the common property that the iterates are combinations of previous iterates and the current gradient. This claim may be true for quadratic optimization but seems to be wrong in general cases. There is no justification for this property throughout the paper. The presented framework is more like a generalized heavy-ball or Nesterov-like method, or can be viewed as an adaptive version of the subspace Newton method. Its connection with quasi-Newton methods and Anderson acceleration is not clear. It is misleading if such a connection is not valid.\n\n2. Although the presented framework does not use the traditional line-search or trust-region technique to guarantee global convergence, it uses a more costly backtracking line-search strategy to estimate the Lipschitz constant. Such complexity can impair the significance of this work.\n\n3. Section 1.2 says that Anderson acceleration does not generalize well outside quadratic minimization. It is not true since Anderson acceleration is well-known for its usefulness in accelerating fixed-point iterations in scientific computing.\n\n4. Some descriptions of the classical methods are not standard. The formulas of BFGS and Anderson acceleration in Section 2 are quite strange. The formula of BFGS below Line 97 seems to be wrong. The formula of Anderson acceleration below Line 104 is incorrect since $d_0$ and $g_0$ are undefined. The formula below Line 513 is not the Anderson acceleration.\n\n5. Many notations are not defined clearly, e.g., $P_i$ in Equation (2), $r_i$ below Line 104, and $R^\\dagger$ in Line 600. \n\n6. The paper discusses many possible methods derived from the framework, but the pros and cons of each method lack clear clarifications. For example, the orthogonalized greedy method seems to outperform BFGS in some cases, but it also doubles the memory usage. So the comparison in the experiments may be unfair.\n\n7. The proofs need to be reorganized to make them easy to follow. It is better to give more examples of the introduced notations and assumptions.\n\nQUESTIONS:\n1. How to derive (2), and what is the explicit form of $P_i$?\n\n2. In Section 5.1, how to guarantee the linear independence of the columns of $D$? When $P_{t-1}\\nabla f(x_t) = \\nabla f(x_t)$, the orthogonal option leads to $x_{t+1/2} = x_t$. Then how to define the iteration, and is the convergence theory still valid in this case?\n\n3. Section B.2.1 claims that the SR-1 method solves (21). It seems to be wrong, since (21) corresponds to the Powell-symmetric-Broyden method. The author(s) may provide some references or calculations to support this claim.\n\nLIMITATIONS:\nSome limitations have been mentioned in the main paper, but there is no discussion about the total computation cost and memory cost.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a generic framework for developing novel quasi-Newton and Anderson/Nonlinear acceleration schemes, offering a global convergence rate in various scenarios, including accelerated convergence on convex functions. They also provide empirical results in the numerical experiments.\n\nSTRENGTHS:\nThe structure is clear and the theoretical analysis and proof are correct. The results presented in the numerical experiments section are consistent with the theoretical results.\n\nWEAKNESSES:\nThere is the following weakness regarding this submission.\n\n1. First, the authors ignores a lot of existing quasi-Newton type methods or accelerated versions that achieve the non-asymptotic global convergence rates of $\\mathcal{O}(1/k)$ and $\\mathcal{O}(1/k^2)$ such as:\n\n“Practical inexact proximal quasi-Newton method with global complexity analysis”. Katya Scheinberg and Xiaocheng Tang. Mathematical Programming160(2016), pp.495–529.\n\n“Proximal quasi Newton methods for regularized convex optimization with linear and accelerated sublinear convergence rates”. Hiva Ghanbari and Katya Scheinberg. Computational Optimization and Applications69(2018),pp.597–627\n\n“Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization”. R. Jiang and A. Mokhtari. https://arxiv.org/abs/2306.02212\n\nRecently there also exists a paper proposing the quasi-Newton type method that could achieve a global explicit superlinear convergence rate:\n\n“Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence”. R. Jiang, Q. Jin, A. Mokhtari. Conference on Learning Theory (COLT), 2023.\n\nHowever, this global superlinear convergence rate assumes strong convexity of the objective function. The authors should compare the results in this submission to all these related literature in detail to check if the proposed algorithm could achieve an improvement in the aspect of convergence rate or computational cost per iteration.\n\n2. The authors didn't compare the results of the this paper to the global convergence rates of first-order gradient descent and accelerated gradient descent. The authors claimed that the proposed algorithm could match the results of accelerated gradient descent, but is the constant in the convergence rate of this proposed method better than the constant of accelerated gradient descent? What is the improvement of this algorithm compared with the accelerated gradient descent? Notice that the computational cost per iteration of this QN-type method is worse than the computational cost per iteration of accelerated gradient descent.\n\n3. It's better for the authors to use the notations of $s_t = x_{t + 1} - x_t$ and $y_t = \\nabla{f}(x_{t + 1}) - \\nabla{f}(x_{t})$. These notations are more commonly-used in the quasi-Newton methods.\n\n4. The notations used from equations (3) to (7) are quietly confusing. What's the exact definition of $y_i$ and $z_i$? Is it $y_i = x_{t - i + 1}$ and $z_i = x_{t - i}$. Also it seems that $D$, $G$ and $\\epsilon$ depend on the iterations index $t$. Then it should be $D_t$, $G_t$ and $\\epsilon_t$. These notations are messed and make the readers difficult to understand and follow the ideas of the authors.\n\n5. What is the definition or function of the parameters $M_0$, $M_t$ and $M_{min}$ presented in the algorithms? The authors said that the $M_0$ is the initial guess of the smoothness parameter, So $M$ is an estimation of the parameter $L$ in assumption 1? What's the value of $M_0$ in all the numerical experiments presented in this paper? What theoretical conditions should this $M_0$ satisfy?\n\n6. The authors presented some designed requirements in section 3.1. These expressions of requirements are a bit strange for the theoretical results. The authors should either make these requirements as the assumptions of the algorithms or make these requirements as some lemmas/theorems of the theoretical analysis. However, these requirements are too strong or restrictive as assumptions. On the other hand, the authors didn't give any strict mathematical proof to give any conditions to satisfy these requirements. The authors argue that these requirements are not restrictive in the text of section 3.1. But this is not enough for the theoretical analysis of the algorithm. We need strict and clear mathematical proof or empirical results from the numerical experiments. It is not clear how to make these requirements to be satisfied. There is no formal proof.\n\n7. The authors claimed that $N$ could be small around line 149. However, as expressed in the theorem 6, it seems that $N$ should be comparable to the dimension $d$ to reach a good convergence rate. But when $N = \\mathcal{O}(d)$, the computational cost of solving the sub-problems could be as expensive as $\\mathcal{O}(d^3)$ as presented in line 148. This is as costly as the Newton's method.\n\n8. From line 203 to line 210, the authors argue that some propertied needed to be satisfied to retrieve the convergence rate of Newton’s method with cubic regularization. However, it seems that when these properties are satisfied, the computational complexity is the same as the Newton's method. Then, what is the advantages or improvements of this method compared to Newton’s method with cubic regularization?\n\n9. What are the definitions of $b_i$ and $\\lambda_t^{(1, 2)}$ in the equation below line 213? The authors should explain these parameters for the accelerated algorithms.\n\n10. The most significant weakness or drawback of this submission is the lack of formal algorithm for the update of matrices $Y, Z, D, G$. Although the authors presented the online and batch techniques in section 5, these descriptions of the update scheme is just an outline and too introductory. We need a lot of details and formal description of the algorithm for the implementations or updates of matrices $Y, Z, D, G$. Without explicit algorithm like Algorithm 1 in the paper, there are a lot of unclear operations regarding the update of the corresponding matrices. The authors should present a formal algorithm regarding the implementations of these updates in the appendix. There also exists a lot of issues for these algorithms. For example, in the deviations of Iterates, Forward Estimates and Greedy updates after the line 239, it seems that column numbers of matrix $Y$ and $Z$ are $N + 1$ instead of $N$. The dimensions are not consistent. Also to implement the orthogonal iterate in the equation under line 236, it needs the matrix $P_{t - 1}$ defined in equation (12). However, this definition in equation (12) needs the operation of matrix multiplication and matrix inversion, which could be very expensive in high dimension condition. Also, the authors argue that Orthogonal Forward Estimates can ensure that the condition number $\\kappa_D = 1$ and the norm of the error vector is small. But this observation is not obvious and need detailed explanations. There is no formal or strict mathematical proof and theoretical analysis to ensure that these updates of matrix $D$ could satisfy the requirements presented in the section 3.1. At last for the batch technique, the authors apply the QR decomposition. However, the computational cost or complexity of QR decomposition could be very high in the high dimension condition and make the implementation very slow in practice.\n\nQUESTIONS:\nPlease check the weakness section for questions.\n\nLIMITATIONS:\nPlease check the weakness section for limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors propose a new framework that connects Quasi-Newton methods with Anderson acceleration. By exploiting the Cubic Regularization technique, the authors achieve new competitive convergence rates comparable to first-order methods in the worst case and second-order methods in the best case. The paper describes various problem setups with their respective convergence rates: non-accelerated methods for non-convex, star-convex, and convex functions, as well as an accelerated method for convex functions. The authors also propose different variants of approximation. Experiments are presented for both convex and non-convex functions.\n\nSTRENGTHS:\nI believe the results presented in the paper are both original and significant. The novel connection between Anderson averaging, Quasi-Newton methods, and subspace sketch-Newton-type methods is both new and prospective. The provided proofs seem mostly correct.\nFurthermore, introducing online approximation techniques adds value to the research. The thorough comparison of these techniques in practical applications showcases their effectiveness and underscores their relevance.\nIn conclusion, this paper contributes valuable insights to the field, and its innovative approach holds great potential for further advancements in optimization methods.\n\nWEAKNESSES:\nI may guess that the paper was created in a hurry before the conference deadline, which might explain why it contains a lot of small mistakes and misprints. While the mathematical results are sound, The mathematical results are sufficient but the overall presentation and clarity need improvement. A thorough review and revision by the authors are necessary to address these issues. Next, I will present some of the mistakes and misprints that I find.\n\n1) Page 6, line 191, Theorem 5. The brackets on the right side are unnecessary. On the right side, $f(x_t)-f^{\\ast}$ should be changed to $f(x_0)-f^{\\ast}$. The same is in the Appendix. \n2) Page 8, line 239. All $Z_t$ and $Y_t$ contain $N+1$ elements, but they should contain $N$. So, I think the indices should be corrected.\n3) Page 36, line 785. $r_i$ was not defined anywhere before, moreover, $r$ is used in other places for different things. It probably should be $G_i$.\n4) Page 36, line 787. The last formula in the line is incorrect; the left side is a vector, and the right side is a number.\n5) Page 36, line 791. The inequality is incorrect because $(r_i - \\nabla^2 f(x) d_i)^2$ is a vector and shouldn’t be squared in such a way.\n6) Page 36, line 795. The third transition is incorrect because $\\|w\\|$ is missing for the $L$ term.\n7) Page 37, line 800. I believe it is better to explicitly prove that the term $\\alpha^T D^TG\\alpha$ could be upper bounded by $\\alpha^T (D^T G+G^T D)\\alpha/2$; otherwise, it may be confusing. \n8) Page 5, line 174. The notation $\\varepsilon_t$ as a vector is confusing because a page before $\\varepsilon_i$ was a number, the coordinate of a vector $\\varepsilon$. \n9) Page 4, line 127. $x_{+}$ is defined in Theorem 1 but never used inside. So, I think it can be removed. On the other hand, $x$ is used inside but not really specified. \n\nNext are the moments that caused confusion for the first read. \n\n10) Page 3, Motivation. The transitions between line 97 to formula (2) and then to formula (4) are very confusing. It may help to specify what are the $\\alpha_i$ and $P_t$ for that case. For formula (4), the author may say that it is a special case of formula (2) when $H_0=0$. It may help to understand such a transition.\n11) Page 4, line 142, formula 10. The norm of $\\|D\\|$ is not defined, which may cause some confusion because $D$ is a rectangular matrix and could have different norms. \n12) Page 7, Algorithm 5. The notation $(M_0)_1$ is very confusing, especially when $M_0$ and $M_1$ also exist. Moving index $t$ of the step to the upper level, like $M^t_0$, may solve both this moment and $\\varepsilon_t$ moment, but it is up to the authors how to resolve these issues with the notation.\n13) Page 15, line 498, formula (19). As a small comment, I would suggest using the keyword “subspaced” or “sketched” instead of “stochastic” to avoid confusion with stochastic optimization methods such as SGD and others.\n\nQUESTIONS:\n1) It is quite challenging for me to understand how restrictive the Requirements for the methods are, especially Requirement 1b, as it appears to be quite limiting. Could you provide some comments on this and highlight specific examples where it is applicable and the reasoning woy?  \n\n2) In Sections 5 and 6, I am curious about the magnitude of $\\kappa$ both in practice and in theory. For instance, when $\\kappa = 10^9$ for the \"Iterates only\" approach, it seems to impose a very stringent condition for convergence because of the term $\\kappa^2$ in $C_2$ and $C_3$. What are the typical values of $\\kappa$ in practice for the \"Iterates only\" approach? Does such a large value of $\\kappa$ indicate that the only applicable regime is the \"Forward Estimates Only\"?  \n\n3) Additionally, I am interested in the practical and theoretical values of $h$ in Sections 5 and 6. My main concern revolves around computational errors and the stability of the gradient difference $\\nabla f(x_{t+1/2}) - \\nabla f(x_{t})$. If $h=10^{-9}$, then $\\nabla f(x_{t+1/2}) - \\nabla f(x_{t})$ should be almost equal to zero close to the solution, even in double precision. This fact may destroy practical convergence of the method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}