Title: Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
Venue: NeurIPS 2023
Accepted: no
Keywords: Cubic Newton

auxiliary information

variance reduction

gradient-dominated functions

non-convex optimization
Abstract: We study stochastic Cubic Newton methods for solving general possibly non-convex minimization problems. We propose a new framework, which we call the {\em helper framework}, that provides a unified view of the stochastic and variance-reduced second-order algorithms equipped with global complexity guarantees. It can also be applied to learning with auxiliary information. Our helper framework offers the algorithm designer high flexibility for constructing and analysis of the stochastic Cubic Newton methods, allowing arbitrary size batches, and the use of noisy and possibly biased estimates of the gradients and Hessians, incorporating both the variance reduction and the lazy Hessian updates. We recover the best-known complexities for the stochastic and variance-reduced Cubic Newton, under weak assumptions on the noise and avoiding artificial logarithms. A direct consequence of our theory is the new lazy stochastic second-order method, which significantly improves the arithmetic complexity for large dimension problems. We also establish complexity bounds for the classes of gradient-dominated objectives, that include convex and strongly convex problems. For Auxiliary Learning, we show that using a helper (auxiliary function) can outperform training alone if a given similarity measure is small.

Abstract

We study stochastic Cubic Newton methods for solving general possibly nonconvex minimization problems. We propose a new framework, which we call the helper framework , that provides a unified view of the stochastic and variancereduced second-order algorithms equipped with global complexity guarantees. It can also be applied to learning with auxiliary information. Our helper framework offers the algorithm designer high flexibility for constructing and analysis of the stochastic Cubic Newton methods, allowing arbitrary size batches, and the use of noisy and possibly biased estimates of the gradients and Hessians, incorporating both the variance reduction and the lazy Hessian updates. We recover the best-known complexities for the stochastic and variance-reduced Cubic Newton, under weak assumptions on the noise and avoiding artificial logarithms. A direct consequence of our theory is the new lazy stochastic second-order method, which significantly improves the arithmetic complexity for large dimension problems. We also establish complexity bounds for the classes of gradient-dominated objectives, that include convex and strongly convex problems. For Auxiliary Learning, we show that using a helper (auxiliary function) can outperform training alone if a given similarity measure is small.

1 Introduction

In many fields of machine learning, it is common to optimize a function f ( x ) that can be expressed as a finite sum:

or, more generally, as an expectation over some given probability distribution: f ( x ) = E ζ [ f ( x , ζ ) ] . When f is non-convex, this problem is especially difficult, since finding a global minimum is NP-hard in general [14]. Hence, the reasonable goal is to look for approximate solutions. The most prominent family of algorithms for solving large-scale problems of the form (1) are the first-order methods , such as the Stochastic Gradient Descent (SGD) [25, 16]. They employ only stochastic gradient information about the objective f ( x ) and guarantee the convergence to a stationary point, which is a point with a small gradient norm.

Nevertheless, when the objective function is non-convex, a stationary point may be a saddle point or even a local maximum, which is not desirable. Another common issue is that first-order methods typically have a slow convergence rate, particularly when the problem is ill-conditioned . Therefore, they may not be suitable when high precision for the solution is required.

To address these challenges, we can take into account second-order information (the Hessian matrix) 32 and apply Newton's method (see, e.g. [19]). Among the many versions of this algorithm, the Cubic 33 Newton method [20] is one of the most theoretically established. With the Cubic Newton method, we 34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

can guarantee global convergence to an approximate second-order stationary point (in contrast, the pure Newton method without regularization can even diverge when it starts far from a neighborhood of the solution). For a comprehensive historical overview of the different variants of Newton's method, see [24]. Additionally, the rate of convergence of the Cubic Newton is provably better than those for the first-order methods.

Therefore, theoretical guarantees of the Cubic Newton method seem to be very appealing for practical applications. However, the basic version of the Cubic Newton requires the exact gradient and Hessian information in each step, which can be very expensive to compute in the large scale setting. To overcome this issue, several techniques have been proposed:

One popular approach is to use inexact stochastic gradient and Hessian estimates with subsampling [33, 17, 32, 21, 12, 6, 1]. This technique avoids using the full oracle information, but typically it has a slower convergence rate compared to the exact Cubic Newton.

Variance reduction techniques [35, 29] combine the advantages of stochastic and exact methods, achievieng an improved rates by recomputing the full gradient and Hessian information at some iterations.

Lazy Hessian updates [26, 9] utilize a simple idea of reusing an old Hessian for several iterations of a second-order scheme. Indeed, since the cost of computing one Hessian is usually much more expensive than one gradient, it can improve the arithmetic complexity of our methods.

In addition, exploiting the special structure of the function f (if known) can also be helpful. For instance, some studies [20, 18] consider gradient-dominated objectives , a subclass of non-convex functions that have improved convergence rates and can even be shown to converge to the global minimum. Examples of such objectives include convex and starconvex functions, uniformly convex functions, and functions satisfying the PL condition [23] as a special case.

In this work, we revise the current state-of-the-art convergence theory for the stochastic Cubic Newton method and propose a unified and improved complexity guarantees for different versions of the method, which combine all the advanced techniques listed above.

Our developments are based on the new helper framework for the second-order optimization, that we present in Section 3. For the first-order optimization, a similar in-spirit techniques called learning with auxiliary information was developed recently in [7, 30]. Thus, our results can also be seen as a generalization of the Auxiliary Learning paradigm to the second-order optimization. However, note that in our second-order case, we have more freedom for choosing the "helper functions" (namely, we use one for the gradients and one for the Hessians). That brings more flexibility into our methods and it allows, for example, to use the lazy Hessian updates.

Our new helper framework provides us with a unified view of the stochastic and variance-reduced methods and can be used by an algorithm designed to construct new methods. Thus, we show how to recover already known versions of the stochastic Cubic Newton with the best convergence rates, as well as present the new Lazy Stochastic Second-Order Method , which significantly improves the total arithmetic complexity for large-dimension problems.

Contributions.

We introduce the helper framework which we argue encompasses multiple methods in a unified way. Such methods include stochastic methods, variance reduction, Lazy methods, core sets, and semi-supervised learning.

This framework covers previous versions of the variance-reduced stochastic Cubic Newton methods with known rates. Moreover, it provides us with new algorithms that employ Lazy Hessian updates and significantly improves the arithmetic complexity (for high dimensions), by using the same Hessian snapshot for several steps of the method.

In the case of Auxiliary learning we provably show a benefit from using auxiliary tasks as helpers in our framework. In particular, we can replace the smoothness constant by a similarity constant which might be smaller.

Moreover, our analysis works both for the general class of non-convex functions, as well as for the class of gradient-dominated problems, that includes convex and uniformly convex functions. Hence, in particular, we are the first to establish the convergence rates of the

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

stochastic Cubic Newton algorithms with variance reduction for the gradient-dominated case.

2 Notation and Assumptions

For simplicity, we consider the finite-sum optimization problem (1), while it can be also possible to generalize our results to arbitrary expectations. We assume that our objective f is bounded from below and denote f ⋆ := inf x f ( x ) , and use the following notation: F 0 := f ( x 0 ) -f ⋆ , for some initial x 0 ∈ R d . We denote by ∥ x ∥ := ⟨ x , x ⟩ 1 / 2 , x ∈ R d , the standard Euclidean norm for vectors, and the spectral norm for symmetric matrices, ∥ H ∥ := max { λ max ( H ) , -λ min ( H ) } , where H = H ⊤ ∈ R d × d . We will also use x ∧ y to denote min( x, y ) .

Throughout this work, we make the following smothness assumption on the objective f :

Our goal is to explore the potential of using the Cubically regularized Newton methods to solve problem (1). At each iteration, being at a point x ∈ R d , we compute the next point x + by solving the subproblem of the form

Here, g and H are estimates of the gradient ∇ f ( x ) and the Hessian ∇ 2 f ( x ) , respectively. Note that solving (2) can be done efficiently even for non-convex problems (see [8, 20, 5]). Generally, the cost of computing x + is O ( d 3 ) arithmetic operations, which are needed for evaluating an appropriate factorization of H . Hence, it is of a similar order as the cost of the classical Newton's step.

We will be interested to find a second-order stationary point to (1). We call ( ε, c ) -approximate second-order local minimum a point x that satisfies:

where ε, c > 0 are given tolerance parameters. Let us define the following accuracy measure (see [20]):

Note that this definition implies that if µ c ( x ) ≤ ε 3 / 2 then x is an ( ε, c ) -approximate local minimum.

Computing gradients and Hessians. It is clear that computing the Hessian matrix can be much more expensive than computing the gradient vector. We denote the corresponding arithmetic complexities by HessCost and GradCost . We will make and follow the convention that HessCost = d × GradCost , where d is the dimension of the problem. For example, this is known to hold for neural networks using the backpropagation algorithm [15]. However, if the Hessian has a sparse structure, the cost of computing the Hessian can be cheaper [22]. Then, we can replace d with the effective dimension d eff := HessCost GradCost ≤ d .

3 Second-Order Optimization with Helper Functions

In this section, we extend the helper framework previously introduced in [7] for first-order optimization methods to second-order optimization.

General principle. The general idea is the following: imagine that, besides the objective function f we have access to a help function h that we think is similar in some sense (that will be defined later) to f and thus it should help to minimize it.

Note that many optimization algorithms can be framed in the following sequential way. For a current state x , we compute the next state x + as:

where ˆ f x ( · ) is an approximation of f around current point x , and r x ( y ) is a regularizer that encodes 122 how accurate the approximation is, and M > 0 is a regularization parameter. In this work, we 123 are interested in cubically regularized second-order models of the form (2) and we use r x ( y ) := 124 1 6 ∥ y -x ∥ 3 . 125

126

127

Now let us look at how we can use a helper h to construct the approximation ˆ f . We notice that we can write

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

We discuss the actual practical choices of the helper function h below. We assume now that we can afford the second-order approximation for the cheap part h around the current point x . However, approximating the part f -h can be expensive (as for example when the number of elements n in finite sum (1) is huge), or even impossible (due to lack of data). Thus, we would prefer to approximate the expensive part less frequently. For this reason, let us introduce an extra snapshot point ˆ x that is updated less often than x . Then, we use it to approximate f -h . Another question that we still need to ask is what order should we use for the approximation of f -h ? We will see that order 0 (approximating by a constant) leads as to the basic stochastic methods, while for orders 1 and 2 we equip our methods with the variance reduction.

Combining the two approximations for h and f -h we get the following model of our objective f :

where C ( x , ˜ x ) is a constant, G ( h, x , ˜ x ) is a linear term, and H ( h, x , ˜ x ) is a matrix. Note that if ˜ x ≡ x , then the best second-order model of the form (3) is the Taylor polynomial of degree two for f around x , and that would give us the exact Newton-type method. However, when the points x and ˜ x are different, we obtain much more freedom in constructing our models.

For using this model in our cubically regularized method (2), we only need to define the gradient g = G ( h, x , ˜ x ) and the Hessian estimates H = H ( h, x , ˜ x ) , and we can also treat them differently (using two different helpers h 1 and h 2 , correspondingly). Thus we come to the following general second-order (meta)algorithm. We perform S rounds, the length of each round is m ≥ 1 , which is our key parameter:

Algorithm 1 Cubic Newton with helper functions

In Algorithm 1 we update the snapshot ˜ x regularly every m iterations. The two possible options are

or

Clearly, option (5) is available only in case we can efficiently estimate the function values. However, we will see that it serves us with better global convergence guarantees, for the gradient-dominated functions.

It remains only to specify how we choose the helpers h 1 and h 2 . We need to assume that they are 152 somehow similar to f . Let us present several efficient choices that lead to implementable second-order 153 schemes. 154

3.1 Basic Stochastic Methods 155

If the objective function f is very "expensive" (for example of the form (1) with n →∞ ), one option is to ignore the part f -h i.e. to approximate it by a zeroth-order approximation: f ( y ) -h ( y ) ≈ f (˜ x ) -h (˜ x ) . Since it is just a constant, we do not need to update ˜ x . In this case, we have:

To treat this choice of the helpers and motivated by the form of the errors in Lemma 5, we assume the 156 following similarity assumptions: 157

Assumption 2 (Bounded similarity) Let for some δ 1 , δ 2 ≥ 0 , it holds

Under this assumption, we prove the following theorem: 158

Theorem 1 Under Assumptions 1 and 2, and M ≥ L , for an output of Algorithm 1 x out chosen uniformly at random from ( x i ) 0 ≤ i ≤ Sm , we have:

We see that according to this result, we can get E [ µ M ( x out )] ≤ ε 3 / 2 only for ε > δ 1 . In other words, we can converge only to a certain neighbourhood around a stationary point , that is determined by the error δ 1 of the stochastic gradients.

However, as we will show next, this seemingly pessimistic dependence leads to the same rate of classical subsampled Cubic Newton methods discovered in [17, 32, 33].

Let us discuss now the specific case of stochastic optimization, where f has the specific form (1), with n potentially being very large. In this case, it is customary to sample batches at random and assume the noise to be bounded in expectation. Precisely speaking, if we assume the standard assumption that for one index sampled uniformly at random, we have E i ∥∇ f ( x ) -∇ f i ( x ) ∥ 2 ≤ σ 2 g and E i ∥∇ 2 f ( x ) -∇ 2 f i ( x ) ∥ 3 ≤ σ 3 h , then it is possible to show that for

for batches B g , B h ⊆ [ n ] sampled uniformly at random and of sizes b g and b h respectively, Assumption 2 is satisfied with [27]: δ 1 = σ g √ b g and δ 2 = ˜ O ( σ h √ b h ) . Note that we can use the same random subsets of indices B g , B h for all iterations.

Corollary 1 In Algorithm 1, let us choose M = L and m = 1 , with basic helpers (6) . Then, according to Theorem 1, for any ε > 0 , to reach an ( ε, L ) -approximate second-order local minimum, we need at most S = √ LF 0 ε 3 / 2 iterations with b g = ( σ g ε ) 2 and b h = σ 2 h ε . Therefore, the total arithmetic complexity of the method becomes

It improves upon the complexity O ( 1 ε 4 ) × GradCost of the first-order SGD for non-convex optimization [11], unless d eff > 1 ε 3 / 2 (high cost of computing the Hessians).

3.2 Let the Objective Guide Us

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

If the objective f is such that we can afford to access its gradients and Hessians from time to time 175 (functions of the form (1) with n < ∞ and 'reasonable"), then we can do better than the previous 176 chapter. In this case, we can afford to use a better approximation of the term f ( y ) -h ( y ) . From a 177 theoretical point of view, we can treat the case when f is only differentiable once, and thus we can 178 only use a first-order approximation of f -h , in this case, we will only be using the hessian of the 179 helper h but only gradients of f . However, in our case, if we assume we have access to gradients then 180

we can also have access to the Hessians of f as well (from time to time). For this reason, we consider 181 a second-order approximation of the term f -h , if we follow the procedure that we described above 182 we find: 183

We see that there is an explicit dependence on the snapshot ˜ x and thus we need to address the question 184 of how this snapshot point should be updated in Algorithm 1. In general, we can update it with a 185 certain probability p ∼ 1 m , and we can use more advanced combinations of past iterates (like the 186 average). However, for our purposes, we simply choose option 4 (i.e. the last iterate), thus it is only 187 updated once every m iterations. 188

189

190

191

192

193

194

195

196

197

We also need to address the question of the measure of similarity in this case. Since we are using a second-order approximation of f -h , it is very logical to compare them using the difference between their third derivatives or equivalently, the Hessian Lipschitz constant of their difference. Precisely we make the following similarity assumption :

Assumption 3 (Lipschitz similarity) Let for some δ 1 , δ 2 ≥ 0 , it holds, ∀ x , ˜ x ∈ R d :

In particular, if f -h 1 and f -h 2 have δ 1 and δ 2 Lipschitz Hessians respectively then h 1 and h 2 satisfy Assumption 3.

Under this assumption, we show that the errors resulting from the use of the snapshot can be successfully balanced by choosing M satisfying:

And we have the following theorem.

Theorem 2 For f, h 1 , h 2 verifying Assumptions 1,3. For a regularization parameter M chosen such that M ≥ L and (9) is satisfied. For an output of Algorithm 1 x out chosen uniformly at random from ( x i ) 0 ≤ i ≤ Sm := T , we have:

In particular, we can choose M = max( L, 32 δ 1 m 2 , 16 δ 2 m ) which gives 198

Based on the choices of the helpers h 1 and h 2 we can have many algorithms. We discuss these in the following sections. We start by discussing variance reduction and Lazy Hessians which rely on sampling batches randomly, then move to core-sets which try to find, more intelligently, representative weighted batches of data, after this, we discuss semi-supervised learning and how unlabeled data can be used to engineer the helpers. More generally, auxiliary learning tries to leverage auxiliary tasks in training a given main task, the auxiliary tasks can be treated as helpers.

3.3 Variance Reduction and Lazy Hessians

The following lemma demonstrates that we can create helper functions h with lower similarity to the main function f of the form (1) by employing sampling and averaging.

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

Lemma 1 Let f = 1 n ∑ n i =1 f i such that all f i are twice differentiable and have L -Lipschitz Hessians. Let B ⊂ { 1 , · · · , n } be of size b and sampled with replacement uniformly at random, and define h B = 1 b ∑ i ∈B f i , then h B satisfies Assumption 3 with δ 1 = L √ b and δ 2 = O ( √ log( d ) L √ b ) .

Choice of the parameter m in Algorithm 1. Minimizing the total arithmetic cost, we choose m = arg min m # Grad ( m,ε ) + d # Hess ( m,ε ) , where # Grad ( m,ε ) and # Hess ( m,ε ) denote the number of gradients and Hessians required to find an ε stationary point.

Now we are ready to discuss several special cases that are direct consequences from Theorem 2.

First, note that choosing h 1 = h 2 = f gives the classical Cubic Newton method [20], whereas choosing h 1 = f and h 2 = 0 , gives the Lazy Cubic Newton [9]. In both cases, we recuperate the known rates of convergence.

General variance reduction. If we sample batches B g and B h of sizes b g and b h consecutively at random and choose

and use these helpers along with the estimates (7), (8), we obtain the Variance Reduced Cubic Newton algorithm [35, 29]. According to Lemma 1, this choice corresponds to δ 1 = L √ b g and δ 2 = ˜ O ( L √ b h ) . For b g ∼ m 4 ∧ n, b h ∼ m 2 ∧ n and M = L , we have the non-convex convergence rate O ( √ LF 0 Sm ) , which is the same as that of the cubic Newton algorithm but with a smaller cost per iteration. Minimizing the total arithmetic cost, we can choose m = arg min m dn + d ( m 3 ∧ nm )+( m 5 ∧ nm ) m . Let us denote by g V R ( n, d ) the corresponding optimal value. Then we reach an ( ε, L ) -approximate second-order local minimum in at most O ( g V R ( n,d ) 3 / 2 ) × GradCost arithmetic operations.

ε

Variance reduction with Lazy Hessians. We can also use lazy updates for Hessians combined with variance-reduced gradients. This corresponds to choosing

which implies (according to Lemma 1) that δ 1 = L √ b g and δ 2 = L . In this case, we need b g ∼ m 2 to obtain a convergence rate of O ( √ LF 0 S √ m ) , which matches the convergence rate of the Lazy Cubic Newton method while using stochastic gradients. We choose this time m = arg min m nd +( m 3 ∧ mn ) √ m , as before. Let us denote g Lazy ( n, d ) the corresponding minimum. Then we guarantee to reach an ( ε, mL ) -approximate second-order local minimum in at most O ( g Lazy ( n,d ) ε 3 / 2 ) × GradCost operations.

To be lazy or not to be? We show that g Lazy ( n, d ) ∼ ( nd ) 5 / 6 ∧ n √ d and g V R ( n, d ) ∼ ( nd ) 4 / 5 ∧ ( n 2 / 3 d + n ) . In particular, for d ≥ n 2 / 3 we have g Lazy ( n, d ) ≤ g V R ( n, d ) and thus for d ≥ n 2 / 3 it is better to use Lazy Hessians than variance-reduced Hessians from a gradient equivalent cost perspective. We note also that for the Lazy approach, we can keep a factorization of the Hessian (this factorization induces most of the cost of solving the cubic subproblem) and thus it is as if we only need to solve the subproblem once every m iterations, so the Lazy approach has a big advantage compared to the general approach, and the advantage becomes even bigger for the case of large dimensions.

Note that according to the theory, we could use the same random batches B g , B h ⊆ [ n ] generated once for all iterations. However, using the resampled batches can lead to a more stable convergence.

3.4 Other Applications

The result in (10) is general enough that it can include many other applications that are only limited by our imagination. To cite a few such applications there are:

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

Core sets. [3] The idea of core sets is simple: can we summarize a potentially big data set using only a few (weighted) important examples? Many reasons such as redundancy make the answer yes. Devising approaches to find such core sets is outside of the scope of this work, but in general, we can see from (10) that if we have batches B g , B h such that they are ( δ 1 , 1) and ( δ 2 , 2) similar to f respectively, then we can keep reusing the same batch B g for at least √ L δ 1 times, and B h for L δ 2 all the while guaranteeing an improved rate. So then if we can design such small batches with small δ 1 and δ 2 then we can keep reusing them, and joy the improved rate without needing large batches.

Auxiliary learning. [4, 2, 31] study how a given task f can be trained in the presence of auxiliary (related) tasks. Our approach can be indeed used for auxiliary learning by treating the auxiliaries as helpers. If we compare (10) to the rate that we obtained without the use of the helpers: O ( √ LF 0 S ) , we see that we have a better rate using the helpers/auxiliary tasks when 1 m + √ δ 2 √ mL + √ δ 1 √ L ≤ 1 .

Semi-supervised learning. [34] Semi-supervised learning is a machine learning approach that combines the use of both labeled data and unlabeled data during training. In general, we can use the unlabeled data to construct the helpers, we can start for example by using random labels for the helpers and improving the labels with training. There are at least two special cases where our theory implies improvement by only assigning random labels to the unlabeled data. In fact, for both regularized least squares and logistic regression, we notice that the Hessian is independent of the labels (only depends on inputs) and thus if the unlabeled data comes from the same distribution as the labeled data, then we can use it to construct helpers which, at least theoretically, have δ 1 = δ 2 = 0 . Because the Hessian is independent of the labels, we can technically endow the unlabeled data with random labels. Theorem 2 would imply in this case E [ µ L ( x out )] = O ( √ LF 0 Sm ) , where S is the number of times we use labeled data and S ( m -1) is the number of unlabeled data.

4 Gradient-Dominated Functions

We consider now the class of gradient-dominated functions defined below.

Assumption 4 ( τ, α ) -gradient dominated. A function f is called gradient dominated on set if it holds, for some α ≥ 1 and τ > 0 :

Examples of functions satisfying this assumption are convex functions ( α = 1 ) and strongly convex functions ( α = 2 ), see Appendix D.1. For such functions, we can guarantee convergence (in expectation) to a global minimum , i.e. we can find a point x such that f ( x ) -f ⋆ ≤ ε .

The Gradient-dominance property is interesting because many non-convex functions have been shown to satisfy it [28, 13, 18]. Furthermore, besides convergence to a global minimum, we get accelerated rates.

We note that for α > 3 / 2 (and only for this case), we needed to assume the following (stronger) inequality:

where the expectation is taken with respect to the iterates ( x t ) of our algorithms. This is a stronger assumption than (11). To avoid using this stronger assumption, we can assume that the iterates belong to some compact set Q ⊂ R d and that the gradient norm is uniformly bounded: ∀ x ∈ Q : ∥∇ f ( x ) ∥ ≤

G . Then, a ( τ, α ) -gradient dominated on set Q function is also a ( τG α -3 / 2 , 3 / 2) -gradient dominated on this set for any α > 3 / 2 .

In Theorem 3 we extend the results of Theorem 1 to gradient-dominated functions.

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

Theorem 3 shows (up to the noise level) for 1 ≤ α < 3 / 2 a sublinear rate, for α = 3 / 2 a linear rate (obtained by taking the limit α → 3 / 2 ) and a superlinear rate for α > 3 / 2 .

We do the same thing for Theorem 2 which we extend in Theorem 4. In this case, we need to set the snapshot line 3 in Algorithm 1) as in 5 i.e. the snapshot corresponds to the state with the smallest value of f during the last m iterations.

Again, the same behavior is observed as for Theorem 3 but this time without noise (variance reduction is working). To the best of our knowledge, this is the first time such analysis is made. As a direct consequence of our results, we obtain new global complexities for the variance-reduced and lazy variance-reduced Cubic Newton methods on the class of gradient-dominated functions.

To compare the statements of Theorems 3 and 4, for convex functions (i.e. α = 1 ), Theorem 3 guarantees convergence to a ε -global minimum in at most O ( 1 ε 5 / 2 + d ε 3 / 2 ) GradCost , whereas Theorem 4 only needs O ( g ( n,d ) √ ε ) GradCost , where g ( n, d ) is either g Lazy ( n, d ) = ( nd ) 5 / 6 ∧ n √ d or g V R ( n, d ) = ( nd ) 4 / 5 ∧ ( n 2 / 3 d + n ) . See the Appendix D.3 for more details.

5 Limitations and possible extensions

Estimating similarity between the helpers and the main function. While we show in this work that we can have an improvement over training alone, this supposes that we know the similarity constants δ 1 , δ 2 , hence it will be interesting to have approaches that can adapt to such constants.

Engineering helper functions. Building helper task with small similarities is also an interesting idea. Besides the examples in supervised learning and core-sets that we provide, it is not evident how to do it in a generalized way.

Using the helper to regularize the cubic subproblem. We note that while we proposed to approximate the 'cheap" part as well in Section 3, one other theoretically viable approach is to keep it intact and approximately solve a 'proximal type" problem involving h , this will lead to replacing L by δ , but the subproblem is even more difficult to solve. However our theory suggests that we don't need to solve this subproblem exactly, we only need m ≥ L δ . We do not treat this case here.

6 Conclusion

In this work, we proposed a general theory for using auxiliary information in the context of the cubically regularized Newton's method. Our theory encapsulates the classical stochastic methods as well as variance reduction and Lazy methods. For auxiliary learning, we showed a provable benefit compared to training alone. Besides studying the convergence for general non-convex functions for which we show convergence to approximate local minima, we also study gradient-dominated functions, for which convergence is accelerated and is to approximate global minima.

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

References

A. Agafonov, D. Kamzolov, P. Dvurechensky, A. Gasnikov, and M. Takáˇ c. Inexact tensor methods and their application to stochastic convex optimization. arXiv preprint arXiv:2012.15636 , 2020.

N. Aviv, A. Idan, M. Haggai, C. Gal, and F. Ethan. Auxiliary learning by implicit differentiation. ICLR 2021.

O. Bachem, M. Lucic, and K. Andreas. Practical coreset constructions for machine learning. arXiv:1703.06476 [stat.ML] https://arxiv.org/abs/1703.06476 , 2017.

S. Baifeng, H. Judy, S. Kate, D. Trevor, and X. Huijuan. Auxiliary task reweighting for minimum-data learning. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

C. Cartis, N. I. Gould, and P. L. Toint. Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Mathematical Programming , 127(2):245-295, 2011.

C. Cartis and K. Scheinberg. Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Mathematical Programming , 169:337-375, 2018.

E. M. Chayti and S. P. Karimireddy. Optimization with access to auxiliary information. 327 arXiv:2206.00395 [cs.LG] , 2022. 328

A. R. Conn, N. I. Gould, and P. L. Toint. Trust region methods . SIAM, 2000. 329

N. Doikov, E. M. Chayti, and M. Jaggi. Second-order optimization with lazy hessians. 330 arXiv:2212.00781 [math.OC] , 2022. 331

N. Doikov and Y. Nesterov. Minimizing uniformly convex functions by cubic regularization of 332 newton method. Journal of Optimization Theory and Applications 189:317-339 , 2021. 333

334

335

S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

S. Ghadimi, H. Liu, and T. Zhang. Second-order methods with cubic regularization under 336 inexact information. arXiv preprint arXiv:1710.05782 , 2017. 337

338

M. Hardt and T. Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.04231 , 2016.

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

C. J. Hillar and L.-H. Lim. Most tensor problems are np-hard. Journal of the ACM (JACM) 60 45. , 2013.

H. J. Kelley. Gradient theory of optimal flight paths. Ars Journal , 30(10):947-954, 1960.

J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. Volume 23, Number 3, 462-466 , 1952.

J. M. Kohler and A. Lucchi. Sub-sampled cubic regularization for non-convex optimization. arXiv preprint arXiv:1705.05933 , 2017.

S. Masiha, S. Salehkaleybar, N. He, N. Kiyavash, and P. Thiran. Stochastic second-order methods improve best-known sample complexity of sgd for gradient-dominated functions. In NeurIPS 2022 - Advances in Neural Information Processing Systems , volume 35, pages 10862-10875, 2022.

Y. Nesterov. Lectures on convex optimization , volume 137. Springer, 2018.

Y. Nesterov and B. Polyak. Cubic regularization of Newton's method and its global performance. Mathematical Programming , 108(1):177-205, 2006.

T. Nilesh, S. Mitchell, J. Chi, R. Jeffrey, and J. Michael I. Stochastic cubic regularization for fast nonconvex optimization. Part of Advances in Neural Information Processing Systems 31 , 2018.

J. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006. 356

B. T. Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel'noi Matematiki 357 i Matematicheskoi Fiziki, 3(4):643-653, , 1963. 358

B. T. Polyak. Newton's method and its use in optimization. European Journal of Operational 359 Research , 181(3):1086-1096, 2007. 360

H. Robbins and S. Monro. A stochastic approximation method the annals of mathematical 361 statistics. Vol. 22, No. 3. pp. 400-407 , 1951. 362

V. Shamanskii. A modification of Newton's method. Ukrainian Mathematical Journal , 363 19(1):118-122, 1967. 364

J. A. Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® 365 in Machine Learning , 8(1-2):1-230, 2015. 366

L. uanzhi and Y. Yang. Convergence analysis of two-layer neural networks with relu activation. 367 Advances in neural information processing systems, 30 , 2017. 368

Z. Wang, Z. Yi, L. Yingbin, and L. Guanghui. Stochastic variance-reduced cubic regularization 369 for nonconvex optimization. AISTATS , 2019. 370

B. Woodworth, K. Mishchenko, and F. Bach. Two losses are better than one: Faster optimization 371 using a cheaper proxy. arXiv preprint arXiv:2302.03542 , 2023. 372

L. Xingyu, S. B. Harjatin, K. George, and H. David. Adaptive auxiliary task weighting for 373 reinforcement learning. 33rd Conference on Neural Information Processing Systems (NeurIPS 374 2019), Vancouver, Canada. 375

P. Xu, F. Roosta-Khorasani, and M. W. Mahoney. Newton-type methods for non-convex 376 optimization under inexact Hessian information. arXiv preprint arXiv:1708.07164 . , 2017. 377

P. Xu, J. Yang, F. Roosta-Khorasani, and M. W. Mahoney. Sub-sampled newton methods with 378 non-uniform sampling. 2016. 379

X. Yang, Z. Song, I. King, and Z. Xu. A survey on deep semi-supervised learning. Technical 380 report, 2021. 381

D. Zhou, P. Xu, and Q. Gu. Stochastic variance-reduced cubic regularization methods. Journal 382 of Machine Learning Research 20 1-47 , 2019. 383

Peer reviews:
comments: SUMMARY:
The authors propose a new way of solving minimization problems that are not necessarily convex.
The main element of their approach is that they use some helper functions which give auxiliary information about the optimization problem at hand.
That enables them to give improved optimization algorithms.

STRENGTHS:
Originality:
The main approach is original.
That is, combining stochastic and variance-induced second-order algorithms is a novel direction.

Quality:
Writing and results are of good quality; see questions.

Clarity:
Clear writing; see questions.

Significance:
Work is significant, as optimization is a central task in computer science.

WEAKNESSES:
Not significant weaknesses found;
see questions.

QUESTIONS:
Line 28:
Why is a saddle point not desirable?

Line 30:
Please explain what you mean by ill-conditioned.

Line 58:
What is the PL condition?

Line 60:
Can you please elaborate on the discussion regarding previous work?

Line 70:
Unclear.

Line 83:
Bullet is unclear.
Why is it important in the parameter is smaller?

Line 86:
Bullet:
Why are stochastic Cubic Newton algorithms with variance reduction for gradient-dominated case important?
Because of their applicability?

Line 98:
Can you please discuss a bit Assumption 1?
It is not trivial, right?

Line 105:
Can you please elaborate on the accuracy measure $\mu_c$?

Line 120:
Can you please cite some relevant work here?

Line 157:
Can you please elaborate on Assumption 2?

Line 271:
I do not understand the discussion before Equation (12).

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a meta-algorithm for second-order methods which they refer to as the ``helper framework''. Their helper framework unifies stochastic and variance-reduced versions of Newton's method, and the framework could also be viewed as auxiliary tasks when applied to first-order methods. The paper uses the helper framework to establish various convergence rates for objectives with Lipschitz Hessian and helper functions with two different similarity conditions (bounded similarity and Lipschitz similarity). Their algorithm (Algorithm 1) applies this framework to analyze cubic Newton. Different settings of Algorithm 1 (e.g. using different helper functions) replicate stochastic Newton, variance-reduced Newton, stochastic and variance-reduced Newton, lazy Hessian Newton, etc. They apply the helper framework to gradient-dominated objectives. They derive convergence rates for stochastic cubic Newton algorithms with variance reduction in the gradient-dominated case, which they claim to be the first to establish.\n\nSTRENGTHS:\n1. Introduces a new framework to unify analysis of different variants of Newton's method. \n2. Instead of assuming convexity or strong-convexity, extends framework to gradient-dominated functions in general.\n3. Through this framework, the authors claim to achieve improved complexity guarantees.\n4. Well-written paper with clear notation and good recap of related work.\n\nWEAKNESSES:\n1. Unclear why we need a new framework to analyze the different variants of Newton's method.\n2. Unclear if the meta-algorithm presented is implementable.\n3. The improved complexity guarantees claimed in the paper are unclear.\n4. The additional variables in the helper framework may make it harder to understand basic second-order methods.\n\nThe reason why I only rated a 2 for Contribution of this paper are mainly due to the four points above. I may have misread or overlooked some details. Please see the questions below that address the perceived weaknesses in more detail.\n\nQUESTIONS:\nQuestions:\n1. Is it useful to have a framework that unifies the different variants of Newton's method? What additional insights are revealed by using the helper framework?\n2. Could you expand on the statement ``we recuperate the known rates of convergence'' on line 213? For readers unfamiliar with the convergence rates of various second-order methods, it could be useful to have a table that shows (method, previous known rate, rate from helper framework).\n3. Is their main algorithm (Algorithm 1) implementable/ practical? For example, in one case (line 208), the input parameter $m$ in Algorithm 1 is value that minimizes the number of gradient and Hessian evaluations to find an $\\epsilon$ stationary point. This seems to be a difficult optimization problem itself. Are there more practical ways of setting $m$? Another example is in Corollary 1 where $M$ is chosen to equal $L$.\n4. The framework introduces some variables such as $\\delta_1$ and $\\delta_2$ in Assumptions 2 and 3. This is expected when a more general framework is introduced and we have additional concepts such as similarity. On the other hand, there is no closed-form for the values of these new variables in general and they also appear hard to estimate. Could the authors provide more insight/ theory/ experimental results for the new variables introduced so that the new complexity guarantees are more understandable?\n\nMinor questions:\n1. The parameter $M$ is used in equation 2 but not defined until later at line 123?\n2. What is $F_0$ in Theorem 1?\n3. Line 169. Could you be more specific about which inequality is used so that Assumption 2 is satisfied?\n4. Line 172. Is the result of Corollary 1 a complexity guarantee for a second-order method? If so, why is the comparison against first-order SGD?\n5. Is there any difference between the definitions of $h_1$ and $h_2$ on line 215 and Equation 6?\n6. There seems to be experimental results for the discussion of ``To be lazy or not to be?'' in the appendix. Why not add a link to this?\n7. Line 244. The similarity measures in Assumption 2 and 3 only take in $\\delta_1$ and $\\delta_2$. What is the second variable in $(\\delta_1,1)$ and $(\\delta_2,2)$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a unified analysis for the Cubic regularized Newton Methods with variance-reduced estimators to solve stochastic nonconvex optimization problems. The results rely on the helper framework which generalizes the use of stochastic estimators for first-order and second-order oracles. The analysis recovers best-known rates and improves in terms of Hessian computation via Lazy Hessian update. Convergence is considered in general nonconvex and gradient-dominated (PL condition) problems.\n\nSTRENGTHS:\nThe strengths of the paper lie within the unified convergence analysis that can recover best-known rates. This can be done via using the helper function which is a clever way to generalize the estimation of stochastic first-order amd second-order (possibly higher...) oracles.\n\nAlthough Algorithm 1 is similar to the original Cubic Newton method but slightly difference in the gradient/Hessian estimators and the update of snapshot points.\n\nAll claims are backed by proof. I have gone through the proofs and I find no problems.\n\nWEAKNESSES:\nThe paper lists 2 options to update the snapshot but there is lack of discussion on why these options are chosen (not others).\n\nOther than that, I agree with the weaknesses pointed out by the authors, especially on the choice of the helper function. As helper function is a new concept, the paper does not provide much details on how to find a concrete choice of helper function given an application. Also note that details on the choice of helper function are not needed in the results due to the similarity assumption.\n\nQUESTIONS:\nAs we have freedom to choose helper function, is there a preferred choice of helper function and what properties that we are looking for in general?\n\nFor the two options (4) and (5), the theoretical results only consider the first and I find nowhere else in the paper that uses the second option. Do experiments all use option (4)? If we use option (5) do we expect improvement in performance?\n\nLIMITATIONS:\nI believe the authors have clearly pointed out the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

comments: SUMMARY:
The paper studies stochastic Cubic Newton methods for solving possibly nonconvex minimization problems. The authors propose a flexible “helper framework” that unifies stochastic and variance-reduced second-order algorithms, offering global complexity guarantees.
This framework allows arbitrary batch sizes, noisy gradient and Hessian estimates, and includes variance reduction and lazy Hessian updates. The work achieves state-of-the-art complexities for stochastic and variance-reduced Cubic Newton methods without artificial logarithms. It also introduces a new efficient lazy stochastic second-order method for large dimension problems.

STRENGTHS:
This paper proposes a new lazy stochastic second-order method.

WEAKNESSES:
- The paper lacks a discussion of variance-reduced methods in the literature, neglecting 
numerous related papers on the subject. 
- The experiments are only in Appendix and weak with a “toy” dataset. 
- The loss function is just similar to “convex” although using the non-convex regularizer. 
- Many assumptions have been used in the paper. That could restrict the applications of the 
problems. 
- It is not clear about the results that could use for machine learning applications.

QUESTIONS:
- What value of lambda for non-convex regularizer in line 411 did you choose?
- Is it more suitable to submit this paper to an optimization journal?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22