Title: Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: Optimization

First-order optimization

Non-convex optimization

Distributed optimization
Abstract: This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. 
The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. 
Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. 
As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. 
Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence.


Abstract

This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence.

1 Introduction

The minimization of smooth and non-convex functions is a fundamental problem in various domains of applied mathematics. Most machine learning algorithms rely on solving optimization problems for training and inference, often with structural constraints or non-convex objectives to accurately capture the learning and prediction problems in high-dimensional or non-linear spaces. However, non-convex problems are typically NP-hard to solve, leading to the popular approach of relaxing them to convex problems and using traditional methods. Direct approaches to non-convex optimization have shown success but their convergence and properties are not well understood, making them challenging for large scale optimization. While its convex alternative has been extensively studied and is generally an easier problem, the non-convex setting is of greater practical interest often being the computational bottleneck in many applications.

In this paper, we consider the general minimization problem:

where f : R d → R is a differentiable function. In order for this problem to have a finite solution we will assume throughout the paper that f is bounded from below.

Assumption 1. There exists f inf ∈ R such that f ( x ) ≥ f inf for all x ∈ R d .

The stochastic gradient descent (SGD) algorithm [MB11, B + 15, GLQ + 19] is one of the most common algorithms to solve this problem. In its most general form, it can be written as

where g ( x k ) is a stochastic estimator of ∇ f ( x k ) and γ > 0 is a positive scalar stepsize. A particular case of interest is the compressed gradient descent (CGD) algorithm [KFJ18], where the estimator g

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

is taken as a compressed alternative of the initial gradient:

and the compressor C is chosen to be a "sparser" estimator that aims to reduce the communication overhead in distributed or federated settings. This is crucial, as highlighted in the seminal paper by [KMY + 16], which showed that the bottleneck of distributed optimization algorithms is the communication complexity. In order to deal with the limited resources of current devices, there are various compression objectives that are practical to achieve. These include also compressing the model broadcasted from server to clients for local training, and reducing the computational burden of local training. These objectives are mostly complementary, but compressing gradients has the potential for the greatest practical impact due to slower upload speeds of client connections and the benefits of averaging [KMA + 21]. In this paper we will focus on this latter problem.

An important subclass of compressors are the sketches. Sketches are linear operators defined on R d , i.e., C ( y ) = S y for every y ∈ R d , where S is a random matrix. A standard example of such a compressor is the Randk compressor, which randomly chooses k entries of its argument and scales them with a scalar multiplier to make the estimator unbiased. Instead of communicating all d coordinates of the gradient, one communicates only a subset of size k , thus reducing the number of communicated bits by a factor of d/k . Formally, Randk is defined as follows: S = ∑ k j =1 d k e glyph[latticetop] i j e glyph[latticetop] i j , where i j are the selected coordinates of the input vector. We refer the reader to [SSR22] for an overview on compressions.

Besides the assumption that function f is bounded from below, we also assume that it is L matrix smooth, as we are trying to take advantage of the entire information contained in the smoothness matrix L and the stepsize matrix D .

Assumption 2 (Matrix smoothness) . There exists L ∈ S d + such that

holds for all x, y ∈ R d .

The assumption of matrix smoothness, which is a generalization of scalar smoothness, has been shown to be a more powerful tool for improving supervised model training. In [SHR21], the authors proposed using smoothness matrices and suggested a novel communication sparsification strategy to reduce communication complexity in distributed optimization. The technique was adapted to three distributed optimization algorithms in the convex setting, resulting in significant communication complexity savings and consistently outperforming the baselines. The results of this study demonstrate the efficacy of the matrix smoothness assumption in improving distributed optimization algorithms.

The case of block-diagonal smoothness matrices is particularly relevant in various applications, such as neural networks (NN). In this setting, each block corresponds to a layer of the network, and we characterize the smoothness with respect to nodes in the i -th layer by a corresponding matrix L i . Unlike in the scalar setting, we favor the similarity of certain entries of the argument over the others. This is because the information carried by the layers becomes more complex, while the nodes in the same layers are similar. This phenomenon has been observed visually in various studies, such as those by [YCN + 15] and [ZCAW17].

Another motivation for using a layer-dependent stepsize has its roots in physics. In nature, the propagation speed of light in media of different densities varies due to frequency variations. Similarly, different layers in neural networks carry different information, metric systems, and scaling. Thus, the stepsizes need to be picked accordingly to achieve optimal convergence.

We study two matrix stepsized CGD-type algorithms and analyze their convergence properties for non-convex matrix-smooth functions. As mentioned earlier, we put special emphasis on the blockdiagonal case. We design our sketches and stepsizes in a way that leverages this structure, and we show that in certain cases, we can achieve compression without losing in the overall communication complexity.

1.1 Related work

Many successful convex optimization techniques have been adapted for use in the non-convex 78 setting. Here is a non-exhaustive list: adaptivity [DOG + 19, ZKV + 20], variance reduction [JRSPS16, 79

LBZR21], and acceleration [GNDG19]. A paper of particular importance for our work is that of 80 [KR20], which proposes a unified scheme for analyzing stochastic gradient descent in the non-convex 81 regime. A comprehensive overview of non-convex optimization can be found in [JK + 17, DDG + 22]. 82

89

90

91

[GR15] analyzed sketched gradient descent by making the compressors unbiased with a sketch-andproject trick. They provided an analysis of the resulting algorithm for the linear feasibility problem. Later, [HMR18] proposed a variance-reduced version of this method.

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

Leveraging the layer-wise structure of neural networks has been widely studied for optimizing the training loss function. For example, [ZTJY19] propose SGD with different scalar stepsizes for each layer, [YHL + 17, GCH + 19] propose layer-wise normalization for Stochastic Normalized Gradient Descent, and [DBA + 20, WSR22] propose layer-wise compression in the distributed setting.

DCGD, proposed by [KFJ18], has since been improved in various ways, such as in [HHH + 19, LKQR20]. There is also a large body of literature on other federated learning algorithms with unbiased compressors [AGL + 17, MGTR19, GBLR21, MMSR22, MSR22, HKM + 23].

1.2 Contributions

Our paper contributes in the following ways:

We propose two novel matrix stepsize sketch CGD algorithms in Section 2, which, to the best of our knowledge, are the first attempts to analyze a fixed matrix stepsize for nonconvex optimization. We present a unified theorem in Section 3 that guarantees stationarity for minimizing matrix-smooth non-convex functions. The results shows that taking our algorithms improve on their scalar alterantives. The complexities are summarized in Table 1 for some particular cases.

We design our algorithms' sketches and stepsize to take advantage of the layer-wise structure of neural networks, assuming that the smoothness matrix is block-diagonal. In Section 4, we prove that our algorithms achieve better convergence than classical methods.

Assuming the that the server-to-client communication is less expensive [KMY + 16, KMA + 21], we propose distributed versions of our algorithms in Section 5, following the standard FL scheme, and prove weighted stationarity guarantees. Our theorem recovers the result for DCGD in the scalar case and improves it in general.

We validate our theoretical results with experiments. The plots and framework are provided in the Appendix.

1.3 Preliminaries

2 The algorithms 127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

Below we define our two main algorithms:

Here, D ∈ S d ++ is the fixed stepsize matrix. The sequences of random matrices S k and T k satisfy the next assumption.

Assumption 3. We will assume that the random sketches that appear in our algorithms are i.i.d., unbiased, symmetric and positive semi-definite for each algorithm. That is

A simple instance of det-CGD1 and det-CGD2 is the vanilla GD. Indeed, if S k = T k = I d and D = γ I d , then x k +1 = x k -γ ∇ f ( x k ) . In general, one may view these algorithms as Newton-type methods. In particular, our setting includes the Newton Star (NS) algorithm by [IQR21]:

The authors prove that in the convex case it converges to the unique solution x inf locally quadratically, provided certain assumptions are met. However, it is not a practical method as it requires knowledge of the Hessian at the optimal point. This method, nevertheless, hints that constant matrix stepsize can yield fast convergence guarantees. Our results allow us to choose the D depending on the smoothness matrix L . The latter can be seen as a uniform upper bound on the Hessian.

The difference between det-CGD1 and det-CGD2 is the update rule. In particular, the order of the sketch and the stepsize is interchanged. When the sketch S and the stepsize D are commutative w.r.t. matrix product, the algorithms become equivalent. In general, a simple calculation shows that if we take

then det-CGD1 and det-CGD2 are the same. Defining T k according to (5), we recover the unbiasedness condition:

However, in general D E [ S k ] D -1 is not necessarily symmetric, which contradicts to Assumption 3. Thus, det-CGD1 and det-CGD2 are not equivalent for our purposes.

3 Main results

Before we state the main result, we present a stepsize condition for det-CGD1 and det-CGD2, respectively:

In the case of vanilla GD (7) and (8) become γ < L -1 , which is the standard condition for convergence.

Below is the main convergence theorem for both algorithms in the single-node regime.

Theorem 1. Suppose that Assumptions 1-3 are satisfied. Then, for each k ≥ 0

if one of the below conditions is true: 158

and and

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

182

183

184

185

186

187

188

The vectors x k are the iterates of det-CGD1 and D satisfies (7) ;

ii) The vectors x k are the iterates of det-CGD2 and D satisfies (8) .

It is important to note that Theorem 1 yields the same convergence rate for any D ∈ S d ++ , despite the fact that the matrix norms on the left-hand side cannot be compared for different weight matrices. To ensure comparability of the right-hand side of (9), it is necessary to normalize the weight matrix D that is used to measure the gradient norm. We propose using determinant normalization, which involves dividing both sides of (9) by det( D ) 1 /d , yielding the following:

This normalization is meaningful because adjusting the weight matrix to D det( D ) 1 /d allows its determinant to be 1 , making the norm on the left-hand side comparable to the standard Euclidean norm. It is important to note that the volume of the normalized ellipsoid { x ∈ R d : ‖ x ‖ 2 D / det( D ) 1 /d ≤ 1 } does not depend on the choice of D ∈ S d ++ . Therefore, the results of (9) are comparable across different D in the sense that the right-hand side of (9) measures the volume of the ellipsoid containing the gradient.

3.1 Optimal matrix stepsize

In this section, we describe how to choose the optimal stepsize that minimizes the iteration complexity. The problem is easier for det-CGD2. We notice that (8) can be explicitly solved. Specifically, it is equivalent to

We want to emphasize that the RHS matrix is invertible despite the sketches not being so. Indeed. The map h : T → TLT is convex on S d + . Therefore, Jensen's inequality implies

This explicit condition on D can assist in determining the optimal stepsize. Since both D and 178 ( T k LT k ) -1 are positive definite, then the right-hand side of (10) is minimized exactly when 179

The situation is different for det-CGD1. According to (10), the optimal D is defined as the solution 180 of the following constrained optimization problem: 181

Proposition 1. The optimization problem (13) with respect to stepsize matrix D ∈ S d ++ , is a convex optimization problem with convex constraint.

The proof of this proposition can be found in the Appendix. It is based on the reformulation of the constraint to its equivalent quadratic form inequality. Using the trace trick, we can prove that for every vector chosen in the quadratic form, it is convex. Since the intersection of convex sets is convex, we conclude the proof.

One could consider using the CVXPY [DB16] package to solve (13), provided that it is first transformed 189 into a Disciplined Convex Programming (DCP) form [GBY06]. Nevertheless, (7) is not recognized 190 as a DCP constraint in the general case. To make CVXPY applicable, additional steps tailored to the 191 problem at hand must be taken. 192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

4 Leveraging the layer-wise structure

In this section we focus on the block-diagonal case of L for both det-CGD1 and det-CGD2. In particular, we propose hyper-parameters of det-CGD1 designed specifically for training NNs. Let us assume that L = Diag( L 1 , . . . , L glyph[lscript] ) , where L i ∈ S d i ++ . This setting is a generalization of the classical smoothness condition, as in the latter case L i = L I d i for all i = 1 , . . . , glyph[lscript] . Respectively, we choose both the sketches and the stepsize to be block diagonal: D = Diag( D 1 , . . . , D glyph[lscript] ) and S k = Diag( S k 1 , . . . , S k glyph[lscript] ) , where D i , S k i ∈ S d i ++ .

Let us notice that the left hand side of the inequality constraint in (13) has quadratic dependence on D , while the right hand side is linear. Thus, for every matrix W ∈ S d ++ , there exists γ > 0 such that

Therefore, for γ W we deduce

The following theorem is based on this simple fact applied to the corresponding blocks of the matrices D , L , S k for det-CGD1.

Theorem 2. Let f : R d → R satisfy Assumptions 1 and 2, with L admitting the layer-separable structure L = Diag( L 1 , . . . , L glyph[lscript] ) , where L 1 , . . . , L glyph[lscript] ∈ S d i ++ . Choose random matrices S k 1 , . . . , S k glyph[lscript] ∈ S d + to satisfy Assumption 3 for all i ∈ [ glyph[lscript] ] , and let S k := Diag( S k 1 , . . . , S k glyph[lscript] ) . Furthermore, choose matrices W 1 , . . . , W glyph[lscript] ∈ S d ++ and scalars γ 1 , . . . , γ glyph[lscript] > 0 such that

Letting W := Diag( W 1 , . . . , W glyph[lscript] ) , Γ := Diag( γ 1 I d 1 , . . . , γ glyph[lscript] I d glyph[lscript] ) and D := Γ W , we get 209

In particular, if the scalars { γ i } are chosen to be equal to their maximum allowed values from (15), 210 then the convergence factor of (16) is equal to 211

Table 1 contains the (expected) communication complexities of det-CGD1, det-CGD2 and GD for 212 several choices of W , D and S k . Here are a few comments about the table. We deduce that taking a 213 matrix stepsize without compression (row 1) we improve GD (row 13). A careful analysis reveals 214 that the result in row 5 is always worse than row 7 in terms of both communication and iteration 215 complexity. However, the results in row 6 and row 7 are not comparable in general, meaning that 216 neither of them is universally better. More discussion on this table can be found in the Appendix. 217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

Compression for free. Now, let us focus on row 12, which corresponds to a sampling scheme where the i -th layer is independently selected with probability q i . Mathematically, it goes as follows:

Jensen's inequality implies that

The equality is attained when q i = q for all i ∈ [ glyph[lscript] ] . The expected bits transferred per iteration of this algorithm is then equal to k exp = qd and the communication complexity equals d det( L ) 1 /d . Comparing with the results for det-CGD2 with randk exp on row 11 and using the fact that det( L ) ≤ det (diag( L )) , we deduce that the Bernoulli scheme is better than the uniform sampling scheme. Notice also, the communication complexity matches the one for the uncompressed det-CGD2 displayed on row 9. This, in particular means that using the Bernq sketches we can compress the gradients for free. The latter means that we reduce the number of bits broadcasted at each iteration without losing in the total communication complexity. In particular, when all the layers have the same width d i , the number of broadcasted bits for each iteration is reduced by a factor of q .

5 Distributed setting

In this section we describe the distributed versions of our algorithms and present convergence guarantees for them. Let us consider an objective function that is sum decomposable:

where each f i : R d → R is a differentiable function. We assume that f satisfies Assumption 1 and the component functions satisfy the below condition.

Assumption 4. Each component function f i is L i -smooth and is bounded from below: f i ( x ) ≥ f inf i for all x ∈ R d .

This assumption also implies that f is of matrix smoothness with ¯ L ∈ S d ++ , where ¯ L = 1 n ∑ n i =1 L i . Following the standard FL framework [KMY + 16, MMR + 17, KFJ18], we assume that the i -th component function f i is stored on the i -th client. At each iteration, the clients in parallel compute and compress the local gradient ∇ f i and communicate it to the central server. The server, then aggregates the compressed gradients, computes the next iterate, and in parallel broadcasts it to the clients. See the algorithms below for the pseudo-codes.

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

Theorem 3. Let f i : R d → R satisfy Assumption 4 and let f satisfy Assumption 1 and Assumption 2 with smoothness matrix L . If the stepsize satisfies

then the following convergence bound is true for the iteration of Algorithm 1:

where ∆ inf := f inf -1 ∑ n i =1 f inf i and

The same result is true for Algorithm 2 with a different constant λ D . The proof of Theorem 3 and its analogue for Algorithm 2 are presented in the Appendix. The analysis is largely inspired by [KR20, Theorem 1]. Now, let us examine the right-hand side of (20). We start by observing that the first term has exponential dependence in K . However, the term inside the brackets, 1 + λ D /n , depends on the stepsize D . Furthermore, it has a second-order dependence on D , implying that λ α D = α 2 λ D , as opposed to det( α D ) 1 /d , which is linear in α . Therefore, we can choose a small enough coefficient α to ensure that λ D is of order n/K . This means that for a fixed number of iterations K , we choose the matrix stepsize to be "small enough" to guarantee that the denominator of the first term is bounded. The following corollary summarizes these arguments, and its proof can be found in the Appendix.

Corollary 1. We reach an error level of ε 2 in (20) if the following conditions are satisfied:

Proposition 2 in the Appendix proves that these conditions with respect to D are convex. In order to minimize the iteration complexity for getting ε 2 error, one needs to solve the following optimization problem

Choosing the optimal stepsize for Algorithm 1 is analogous to solving (13). One can formulate the distributed counterpart of Theorem 2 and attempt to solve it for different sketches. Furthermore, this leads to a convex matrix minimization problem involving D . We provide a formal proof of this property in the Appendix. Similar to the single-node case, computational methods can be employed using the CVXPY package. However, some additional effort is required to transform (21) into the disciplined convex programming (DCP) format.

The second term in (20) corresponds to the convergence neighborhood of the algorithm. It does not depend on the number of iteration, thus it remains unchanged, after we choose the stepsize.

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

0

2000

4000

6000

8000

10000

Iterations

10

-

3

10

-

2

10

-

1

GK,

D

λ

= 0

.

1,

n

= 100

Standard DCGD

DCGD-mat

D det-CGD1 diagonal

D

D det-CGD2 diagonal

D

0

2000

4000

6000

8000

10000

Iterations

10

-

4

10

-

3

10

-

2

GK,

D

λ

= 0

.

1,

n

= 500

Standard DCGD

DCGD-mat

D det-CGD1 diagonal

D

D det-CGD2 diagonal

D

0

2000

4000

6000

8000

10000

Iterations

10

-

4

10

-

3

10

-

2

10

-

1

GK,

D

λ

= 0

.

1,

n

= 500

Standard DCGD

DCGD-mat

D det-CGD1 diagonal

D

D det-CGD2 diagonal

D

Nevertheless, it depends on the number of clients n . In general, the term ∆ inf /n can be unbounded, when n → + ∞ . However, per Corollary 1, we require λ D to be upper-bounded by n/K . Thus, the neighborhood term will indeed converge to zero when K → + ∞ , if we choose the stepsize accordingly.

We compare our results with the existing results for DCGD. In particular we use the technique from [KR20] for the scalar smooth DCGD with scalar stepsizes. This means that the parameters of algorithms are L i = L i I d , L = L I d , D = γ I d , ω = λ max ( E [ ( S k i ) glyph[latticetop] S k i ]) -1 . One may check that (21) reduces to

As expected, this coincides with the results from [KR20, Corollary 1]. See the Appendix for the details on the analysis of [KR20]. Finally, we back up our theoretical findings with experiments. See Figure 1 for a simple experiment confirming that Algorithms 1 and 2 have better iteration and communication complexity compared to scalar stepsized DCGD. For more details on the experiments we refer the reader to the corresponding section in the Appendix.

6 Conclusion

6.1 Limitations

glyph[negationslash]

It is worth noting that every point in R d can be enclosed within some volume 1 ellipsoid. To see this, let 0 = v ∈ R d and define Q := α ‖ v ‖ 2 vv glyph[latticetop] + β ∑ d i =1 v i v glyph[latticetop] i , where v 1 = v ‖ v ‖ , v 2 , . . . , v d form an orthonormal basis. The eigenvalues of Q are β (with multiplicity d -1 ) and α (with multiplicity 1 ), so we have det( Q ) = β d -1 α ≤ 1 . Furthermore, we have ‖ v ‖ 2 Q = v glyph[latticetop] Q v = α ‖ v ‖ 2 . By choosing α = 1 ‖ v ‖ 2 and β = ‖ v ‖ 2 / ( d -1) , we can obtain det( Q ) = 1 while ‖ v ‖ 2 Q ≤ 1 . Therefore, having the average D -norm of the gradient bounded by a small number does not guarantee that the average Euclidean norm is small. This implies that the theory does not guarantee stationarity in the Euclidean sense.

6.2 Future work

Matrix stepsize gradient methods are still not well studied and require further analysis. Although 293 many important algorithms have been proposed using scalar stepsizes and are known to have good 294 performance, their matrix analogs have yet to be thoroughly examined. The distributed algorithms 295 proposed in Section 5 follow the structure of DCGD by [KFJ18]. However, other federated learning 296 mechanisms such as MARINA, which has variance reduction [GBLR21], or EF21 by [RSF21], which 297 has powerful practical performance, should also be explored. 298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

References

[ABK07] Mehiddin Al-Baali and H Khalfan. An overview of some practical quasi-newton methods for unconstrained optimization. Sultan Qaboos University Journal for Science [SQUJS] , 12(2):199-209, 2007.

[ABSM14] Mehiddin Al-Baali, Emilio Spedicato, and Francesca Maggioni. Broyden's quasiNewton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems. Optimization Methods and Software , 29(5):937-954, 2014.

[AGL + 17] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. Advances in neural information processing systems , 30, 2017.

[B + 15] Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

[Bro65] Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation , 19(92):577-593, 1965.

[CL11] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

[DB16] Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. The Journal of Machine Learning Research , 17(1):2909-2913, 2016.

[DBA + 20] Aritra Dutta, El Houcine Bergou, Ahmed M Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis. On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3817-3824, 2020.

[DDG + 22] Marina Danilova, Pavel Dvurechensky, Alexander Gasnikov, Eduard Gorbunov, Sergey Guminov, Dmitry Kamzolov, and Innokentiy Shibaev. Recent theoretical advances in non-convex optimization. In High-Dimensional Optimization and Probability: With a View Towards Data Science , pages 79-163. Springer, 2022.

[DM77] John E Dennis, Jr and Jorge J Moré. Quasi-Newton methods, motivation and theory. SIAM review , 19(1):46-89, 1977.

[DOG + 19] Darina Dvinskikh, Aleksandr Ogaltsov, Alexander Gasnikov, Pavel Dvurechensky, Alexander Tyurin, and Vladimir Spokoiny. Adaptive gradient descent for convex and non-convex stochastic optimization. arXiv preprint arXiv:1911.08380 , 2019.

[GBLR21] Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richtárik. MARINA: Faster non-convex distributed learning with compression. In International Conference on Machine Learning , pages 3788-3798. PMLR, 2021.

[GBY06] Michael Grant, Stephen Boyd, and Yinyu Ye. Disciplined convex programming. Global optimization: From theory to implementation , pages 155-210, 2006.

[GCH + 19] Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, Yang Zhang, and Jonathan M Cohen. Stochastic gradient methods with layer-wise adaptive moments for training of deep networks. arXiv preprint arXiv:1905.11286 , 2019.

[GLQ + 19] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In International Conference on Machine Learning , pages 5200-5209. PMLR, 2019.

[GNDG19] SV Guminov, Yu E Nesterov, PE Dvurechensky, and AV Gasnikov. Accelerated primal-dual gradient descent with linesearch for convex, nonconvex, and nonsmooth optimization problems. In Doklady Mathematics , volume 99, pages 125-128. Springer, 2019.

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies optimization problem under non-convex and smooth setting, where the smoothness is parametrized by a PSD matrix instead of a scalar. It considers using matrix instead of scalar as stepsize, which can trace back to the Newton's method. In addition, it considers the distributed/federated optimization setting, where the communication efficiency is a major concern. To this end, this paper considers two ways of performing updates: $DS^k \\nabla f(x_k)$ and $T^k D \\nabla f(x_k)$. The matrix $D$ for the second one can be easily computed, and it requires to solve a convex program for the first one. They show how to extend this formulation to layerwise structure, which is widely applied in neural network training. They further show a federated algorithm using matrix stepsize. Experiments are conducted and results are in favor of the proposed algorithm, compared to DCGD.\n\nSTRENGTHS:\nDistributed optimization with matrix smoothness and stepsize is indeed an intriguing topic. This paper further analyzes its compatibility with compressor/sketches, which are widely used in federated learning. It provides explicit construction of the stepsize matrix to obtain the optimal convergence behavior. This is the first work that provides theoretical understanding in the federated setting, which has become one of the most applicable fields in optimization. I believe this framework can be further studied and yield more uses in the context of federated learning.\n\nWEAKNESSES:\nAs noted in the paper, one of the first matrix stepsize algorithms is the Newton's method. It is also highlighted the reason why this paper seeks a better matrix stepsize --- as computing and inverting the Hessian is always expensive. However, only the $T^k D \\nabla f(x_k)$ case is easy to compute, and the $DS^k \\nabla f(x_k)$ requires solving a convex program. It is important to quantify the computation complexity of the $DS^k \\nabla f(x_k)$ case, otherwise it might just be better to use the Newton's method. \n\nThe sketching matrices $S^k$ and $T^k$ seem to be only useful when the problem has layerwise or blockwise structure. For a more general purpose optimization framework, how does one design such distributions? Perhaps this correlates to a recent work for first-order federated learning paradigm, where one first sketches the gradient, sends it to the server and uses the transpose of the sketches to recover the averaged gradients for model update [SWYZ22]? This point should be further elaborated, otherwise the proposed method has only limited uses.\n\nAs the paper focuses on factoring the sketching matrices into analysis, more literature regarding such approach should be examined. For example, using sketching for second-order method in the case of linear program and empirical risk minimization has been heavily studied before [LSZ19, SY21, QSZZ23]. Can we devise matrix stepsize method for these interior point methods?\n\n[LSZ19]: Y. Lee, Z. Song and Q. Zhang. Solving empirical risk minimization in the current matrix multiplication time. COLT'19.\n\n[SY21]: Z. Song and Z. Yu. Oblivious sketching-based central path method for linear programming. ICML'21.\n\n[QSZZ23]. L. Qin, Z. Song, L. Zhang and D. Zhuo. An online and unified algorithm for projection matrix vector multiplication with application to empirical risk minimization. AISTATS'23.\n\n[SWYZ22]: Z. Song, Y. Wang, Z. Yu and L. Zhang. Sketching for First Order Method: Efficient Algorithm for Low-Bandwidth Channel and Vulnerability. Arxiv'22.\n\nQUESTIONS:\nCan you provide some implementation details for computing $D$ in the $DS^k \\nabla f(x_k)$? What will be a practical algorithm and what is its runtime complexity?\n\nAuthors should also discuss how to realize the operators $S^k$ and $T^k$ to realize communication cost reduction for general optimization tasks.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper analyzed two different non-convex optimization algorithms with matrix step sizes with sketching.

STRENGTHS:
1. The paper coupled sketching with matrix stepsize, and detailed a number of conditions that need to be satisfied in order for the algorithm to work.

WEAKNESSES:
1. (9) and (10) do not clearly communicate why using a matrix stepsize D is better than a constant stepsize. Right now the machine learning community is concerned more with the quality of the local minima (generalization abilities, saddle-escaping, etc), not the speed at which we arrive at any local minima. Therefore this paper does not convince me to use matrix step-size over constant stepsize in its current form.

2. We don't know why matrix stepsize and sketching matrices need to be coupled. What if you only use D, not S and T? Would the results be better? Or simply because someone else already performed that analysis.

QUESTIONS:
See weakness number 2

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies matrix step-sized and compressed gradient descent method for non-convex optimization. Two types of update are given (det-CGD1 and det-CGD2) . They are extensions of the vanilla gradient descent from scalar step-sizes to matrix cases, and can be considered as Newton-Type methods with a constant Hessian approximation. Convergence results in the single-node regime are given under generalized step-size conditions. Optimal matrix step-sizes are discussed for two types of update. Particularly, the block diagonal case of matrix smoothness is considered to leverage the layer-wise structure of NN training. In this case, gradient compression with Bernoulli sketches comes for free, compared to the uncompressed version. Two types of update are discussed under distributed setting as well and shown to have different theoretical results which guarantee the convergence when the chosen matrix step-size is bounded by n/K. Experiments are provided to show the advantages of the proposed algorithms.\n\nSTRENGTHS:\n1. proposed two types of CGD with matrix step-sizes for non-convex optimization and analyzed matrix step-size conditions and optimal matrix step-size for convergence.\n2. specialized to the block-diagonal case of matrix smoothness to leverage layer-wise structure for NN training, and found gradient compression for free.\n3. extended to distributed setting.\n4. demonstrated good performance of the proposed algorithm\n\nWEAKNESSES:\nAlthough this matrix step-size extension may be useful, it seems to be straightforward to get theoretical results on convergence. Also, matrix step-size tuning may be a difficult job in practice, and step-size conditions seem hard to check in the general case. Optimal step-size conditions for det-CGD2 seem computationally hard to meet by CVX and are required for each step.\n\nIt needs more efforts to determine the matrix smoothness than in the scalar case. Also, it would be space-inefficient in the general case.\n\nSince matrix step-size was proposed previously, it would be better to make a clear comparison with SHR21.\n\nExperiment section is incomplete in the main text.\n\nThere are some typos.\n\nLine 46: first occurrence of $e_ij^T$\n\nLine 133: No definition of $\\mathcal{S}$, $\\mathcal{T}$\n\nLine 255: \"denominator\"\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nLimitations of the work were discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two novel single-node and distributed algorithms for non-convex optimization via SGD with matrix stepsizes and compressed gradients via random sketches. The authors also provide theoretical guarantees for convergence of their methods to a stationary point under a matrix smoothness assumption, and also show improved communication complexities for the distributed algorithms. Further, they specialize their results for the block-diagonal smoothness assumption with block-diagonal stepsizes, which corresponds to multilayer neural networks. Theoretical results are corroborated by experiments.\n\nSTRENGTHS:\n1. The paper proposes an interesting and novel SGD type algorithm for non-convex optimization that combines random sketching for compressed gradients with matrix stepsizes. \n2. The algorithm extends to distributed computation settings, and leads to gains in communication complexity in various settings including multilayer neural networks by leveraging different gradient compression schemes that fall under the model studied by the paper.\n3. The presentation is fairly clear and easy to follow.\n4. I did not check all the proofs but they seem sound.\n\nWEAKNESSES:\n1. The main convergence result is in the $D$-norm (or the determinant normalized version), and not in the euclidean norm. The authors note this weakness in the limitations section.\n2. Some definitions are not clearly stated (like the error $\\epsilon^2$)\n3. There were no experiments on the neural network setting (specifically with the block-diagonal smoothness assumption). Additionally, more experiments highlighting the improved communication complexity implied by the theory would be nice as well.\n\nQUESTIONS:\n1. Additional experiments that compare convergence vs total communication overhead instead of just iteration count would highlight the advantages of the proposed method.\n2. Perhaps a simple question -- but why is the error level treated as $\\epsilon^2$ instead of just $\\epsilon$? This term was not clearly defined, but I believe the error convergence is still a $1/K$ rate after $K$ iterations, is this correct?\n3. How would the results and algorithm extend to the case of acceleration? And further, how would this compare to other adaptive matrix step-size style algorithms with acceleration like Adam?\n\nLIMITATIONS:\nAs pointed out by the authors, the convergence results do not extend to the Euclidean norm.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}

comments: SUMMARY:
1. This work analyzes two compressed gradient descent methods with matrix step sizes for non-convex optimization. 
2. This work designed the sketches and step sizes to take advantage of the layer-wise structure. 
3. The distributed version of the proposed method is also discussed under the Federated Learning framework.

STRENGTHS:
Please refer to the first section.

WEAKNESSES:
This paper can be improved in written style and flow.

QUESTIONS:
1. I am not an expert in this topic. I am curious if the Set-CGD can be analyzed under the mirror-descent type algorithms. Can you elaborate on the intuition behind the algorithmic design? (line 128 and line 129)
2. In figure 1, why are the green and the red lines exactly overlapping?

LIMITATIONS:
As discussed in Section 6.1 and 6.2.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-04