Title: Analyzing and Improving Greedy 2-Coordinate Updates For Equality-Constrained Optimization via Steepest Descent in the 1-Norm
Venue: NeurIPS 2023
Accepted: no
Keywords: Coordinate descent

SVM

LIBSVM

Steepest descent

convex optimization
Abstract: We first consider minimizing a smooth function subject to a summation constraint over its variables. 
By exploiting a connection between the greedy 2-coordinate update for this problem and equality-constrained steepest descent in the 1-norm, we give a convergence rate for greedy selection that is faster than random selection and independent of the problem dimension $n$ (under a proximal Polyak-Lojasiewicz assumption). We then consider minimizing with both a summation constraint and bound constraints, as they arise in the support vector machine dual problem. Existing greedy rules for this setting either only guarantee trivial progress or require $O(n^2)$ time to compute. We show that bound- and summation-constrained steepest descent in the L1-norm guarantees more progress per iteration than previous rules and can be computed in only $O(n \log n)$.

Abstract

We consider minimizing a smooth function subject to a summation constraint over its variables. By exploiting a connection between the greedy 2-coordinate update for this problem and equality-constrained steepest descent in the 1-norm, we give a convergence rate for greedy selection under a proximal PolyakŁ ojasiewicz assumption that is faster than random selection and independent of the problem dimension n . We then consider minimizing with both a summation constraint and bound constraints, as arises in the support vector machine dual problem. Existing greedy rules for this setting either guarantee trivial progress only or require O ( n 2 ) time to compute. We show that bound- and summation-constrained steepest descent in the L1-norm guarantees more progress per iteration than previous rules and can be computed in only O ( n log n ) time.

1 Introduction

Coordinate descent (CD) is an iterative optimization algorithm that performs a gradient descent step on a single variable at each iteration. CD methods are appealing because they have a convergence rate similar to gradient descent, but for some common objective functions the iterations have a much lower cost. Thus, there is substantial interest in using CD for training machine learning models.

Unconstrained coordinate descent : Nesterov [2012] considered CD with random choices of the coordinate to update, and proved non-asymptotic linear convergence rates for strongly-convex functions with Lipschitz-continuous gradients. It was later shown that these linear convergence rates are achieved under a generalization of strong convexity called the PolyakŁ ojsiewicz condition [Karimi et al., 2016]. Moreover, greedy selection of the coordinate to update also leads to faster rates than random selection [Nutini et al., 2015]. These faster rates do not depend directly on the dimensionality of the problem due to an equivalence between the greedy coordinate update and steepest descent on all coordinates in the 1-norm. For a discussion of many other problems where we can implement greedy selection rules at similar cost to random rules, see Nutini et al. [2022, Sections 2.4-2.5].

Bound-constrained coordinate descent : CD is commonly used for optimization with lower and/or upper bounds on each variable. Nesterov [2012] showed that the unconstrained rates of randomized CD can be achieved under these separable constraints using a projected-gradient update of the coordinate. Richtárik and Takáˇ c [2014] generalize this result to include a non-smooth but separable term in the objective function via a proximal-gradient update; this justifies using CD in various constrained and non-smooth settings, including least squares regularized by the 1-norm and support vector machines with regularized bias. Similar to the unconstrained case, Karimireddy et al. [2019] show that several forms of greedy coordinate selection lead to faster convergence rates than random selection for problems with bound constraints or separable non-smooth terms.

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

62

Equality-constrained coordinate descent : many problems in machine learning require us to satisfy an equality constraint. The most common example is that discrete probabilities must sum to one. Another common example is SVMs with an unregularized bias term. The (non-separable) equality constraint cannot be maintained by single-coordinate updates, but it can be maintained if we update two variables at each iteration. Necoara et al. [2011] analyze random selection of the two coordinates to update, while Fang et al. [2018] discuss randomized selection with tighter rates. The LIBSVM package [Chang and Lin, 2011] uses a greedy 2-coordinate update for fitting SVMs which has the same cost as random selection. But despite LIBSVM being perhaps the most widely-used CD method of all time, current analyses of greedy 2-coordinate updates either result in sublinear convergence rates or do not lead to faster rates than random selection [Tseng and Yun, 2009, Beck, 2014].

Our contributions : we first give a new analysis for the greedy 2-coordinate update for optimizing a smooth function with an equality constraint. The analysis is based on an equivalence between the greedy update and equality-constrained steepest descent in the 1-norm. This leads to a simple dimension-independent analysis of greedy selection showing that it can converge substantially faster than random selection. Next, we consider greedy rules when we have an equality constraint and bound constraints. We argue that the rules used by LIBSVM cannot guarantee non-trivial progress on each step. We analyze a classic greedy rule based on maximizing progress, but this analysis is dimension-dependent and the cost of implementing this rule is O ( n 2 ) if we have both lower and upper bounds. Finally, we show that steepest descent in the 1-norm with equalities and bounds guarantees a fast dimension-independent rate and can be implemented in O ( n log n ) . This rule may require updating more than 2 variables, in which case the additional variables can only be moved to their bounds, but this can only happen for a finite number of early iterations.

2 Equality-Constrained Greedy 2-Coordinate Updates

We first consider the problem of minimizing a twice-differentiable function f subject to a simple linear equality constraint,

where n is the number of variables and γ is a constant. On iteration k the 2-coordinate optimization 60 method chooses a coordinate i k and a coordinate j k and updates these two coordinates using 61

for a scalar δ k (the other coordinates are unchanged). We can write this update for all coordinates as

If the iterate x k satisfies the constraint then this update maintains the constraint. In the coordinate 63 gradient descent variant of this update we choose δ k = -α k 2 ( ∇ i k f ( x k ) -∇ j k f ( x k )) for a step size 64 α k . This results in an update to i k and j k of the form 65

If f is continuous, this update is guaranteed to decrease f for sufficiently small α k . The greedy rule 66 chooses the coordinates to update by maximizing the difference in their partial derivatives, 67

At the solution of the problem we must have partial derivatives being equal, so intuitively this greedy 68 choice updates the coordinates that are furthest above/below the average partial derivative. This choice 69 also minimizes the set of 2-coordinate quadratic approximations to the function (see Appendix A.1) 70

which is a special case of the Gauss-Southwell-q (GS-q) rule of Tseng and Yun [2009]. 71

We assume that the gradient of f is Lipschitz continuous, and our analysis will depend on a quantity 72 we call L 2 . The quantity L 2 bounds the change in the 2 -norm of the gradient with respect to any two 73 coordinates i and j under a two-coordinate update of any x of the form (3). 74

Note that L 2 is less than or equal to the Lipschitz constant of the gradient of f . 75

76

2.1 Connections between Greedy 2-Coordinate Updates and the 1-Norm

Our analysis relies on several connections between the greedy update and steepest descent in the 77 1-norm, which we outline in this section. First, we note that vectors d k of the form (3) satisfy 78 ‖ d k ‖ 2 1 = 2 ‖ d k ‖ 2 2 , since 79

Second, if a twice-differentiable function's gradient satisfies the 2-coordinate Lipschitz continuity 80 assumption (7) with constant L 2 , then the full gradient is Lipschitz continuous in the 1 -norm with 81 constant L 1 = L 2 / 2 (see Appendix B). Finally, we note that applying the 2-coordinate update (4) 82 is an instance of applying steepest descent over all coordinates in the 1 -norm. In particular, in 83 Appendix A.2 we show that steepest descent in the 1 -norm always admits a greedy 2-coordinate 84 update as a solution. 85

Lemma 2.1. Let α > 0 . Then at least one steepest descent direction with respect to the 1 -norm has 86 exactly two non-zero coordinates. That is, 87

88

89

90

91

92

93

94

95

96

97

98

99

This lemma allows us to equate the progress of greedy 2-coordinate updates to the progress made by a full-coordinate steepest descent step descent step in the 1-norm.

2.2 Proximal-PL Inequality in the 1-Norm

For lower bounding sub-optimality in terms of the 1-norm, we introduce the proximal-PL inequality in the 1-norm. The proximal-PL condition was introduced to allow simpler proofs for various constrained and non-smooth optimization problems [Karimi et al., 2016]. The proximal-PL condition is normally defined based on the 2 -norm, but we define a variant for the summation-constrained problem where distances are measured in the 1 -norm.

Definition 2.2. A function f , that is L 1 -Lipschitz with respect to the 1 -norm and has a summmation constraint on its parameters, satisfies the proximal-PL condition in the 1 -norm if for a positive constants µ 1 we have

for all x satisfying the equality constraint. Here, f ∗ is the constrained optimal function value and

It follows from the equivalence between norms that summation-constrained functions satisfying the 100 proximal-PL condition in the 2-norm will also satisfy the above proximal-PL condition in the 1-norm. 101 In particular, if µ 2 is the proximal-PL constant in the 2 -norm, then we have µ 2 n ≤ µ 1 ≤ µ 2 (see 102 Appendix C). Functions satisfying these conditions include any strongly-convex function f as well as 103 relaxations of strong convexity, such as functions of the form f = g ( Ax ) for a strongly-convex g and 104 a matrix A [Karimi et al., 2016]. In the g ( Ax ) case f is not strongly-convex if A is singular, and we 105 note that the SVM dual problem can be written in the form g ( Ax ) . 106

107

108

109

110

111

112

113

114

2.3 Convergence Rate of Greedy 2-Coordinate Updates under Proximal-PL

Weanalyze the greedy 2-coordinate method under the proximal-PL condition based on the connections to steepest descent in the 1-norm.

Theorem 2.3. Let f be a twice-differentiable function whose gradient is 2-coordinate-wise Lipschitz (7) and restricted to the set where x T 1 = γ . If this function satisfies the proximal-PL inequality in the 1-norm (9) for some positive µ 1 , then the iterations of the 2-coordinate update (4) with α k = 1 /L 2 and the greedy rule (5) satisfy:

Proof. Starting from the descent lemma restricted to the coordinates i k and j k we have 115

Now subtracting f ∗ from both sides and using the definition of D from the proximal-PL assumption, 116

Applying the inequality recursively completes the proof. 117

118

119

Note that the above rate also holds if we choose α k to maximally decrease f , and the same rate holds up to a constant if we use a backtracking line search to set α k .

120

121

122

2.4 Comparison to Randomized Selection

If we sample the two coordinates i k and j k from a uniform distribution, then it is known that the 2-coordinate descent method satisfies [She and Schmidt, 2017]

A similar result for a more-general problem class was shown by Necoara and Patrascu [2014]. This is 123 substantially slower than the rate we show for the greedy 2-coordinate descent method. This rate 124 is slower even in the extreme case where µ 1 is similar to µ 2 /n , due to the presence of the n 2 term. 125

There also exist analyses for cyclic selection in the equality-constrained case but existing rates for 126 cyclic rules are slower than the random rates Wang and Lin [2014]. 127

128

129

130

131

132

133

134

In the case where f is a dense quadratic function of n variables, which includes SVMs under the most popular kernels, both random selection and greedy selection cost O ( n ) per iteration to implement. If we consider the time required to reach an accuracy of /epsilon1 under random selection using the rate (12) we obtain O ( n 3 κ log(1 / /epsilon1 )) where κ = L 2 /µ 2 . While for greedy selection under (11) it is between O ( n 2 κ log(1 / /epsilon1 )) if µ 1 is close to µ 2 /n and O ( n κ log(1 / /epsilon1 )) if µ 1 is close to µ 2 . Thus, the reduction in total time complexity from using the greedy method is between a factor of O ( n ) and O ( n 2 ) . This is a large difference which has not been reflected in previous analyses.

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

There exist faster rates than (12) in the literature, but these require additional assumptions such as f being separable or that we know the coordinate-wise Lipschitz constants [Necoara et al., 2011, Necoara and Patrascu, 2014, Necoara et al., 2017, Fang et al., 2018]. However, these assumptions restrict the applicability of the results. Further, unlike convergence rates for random coordinate selection, we note that the new linear convergence rate (11) for greedy 2-coordinate method avoids requiring a direct dependence on the problem dimension. The only previous dimension-independent convergence rate for the greedy 2-coordinate method that we are aware of is due to Beck [2014, Theorem 5.2b]. Their work considers functions that are bounded below, which is a weaker assumption than the proximal-PL assumption. However, this only leads to sublinear convergence rates and only on a measure of the violation in the Karush-Kuhn-Tucker conditions. Beck [2014, Theorem 6.2] also gives convergence rates in terms of function values for the special case of convex functions, but these rates are sublinear and dimension dependent.

3 Equality- and Bound-Constrained Greedy Coordinate Updates

Equality constraints often appear alongside lower and/or upper bounds on the values of the individual variables. This results in problems of the form

This framework includes our motivating problems of optimizing over the probability simplex ( l i = 0 150 for all i since probabilites are non-negative), and optimizing SVMs with an unregularized bias (where 151 we have lower and upper bounds). With bound constraints we use a d k of form (3) but where δ k is 152 defined so that the step respects the constraints, 153

154

155

156

157

Unfortunately, analyzing the bound-constrained case is more complicated. There are several possible generalizations of the greedy rule for choosing the coordinates i k and j k to update, depending on what properties of (5) we want to preserve [see Nutini, 2018, Section 2.7]. In this section we discuss several possibilities, and how the choice of greedy rule affects the convergence rate and iteration cost.

158

3.1 GS-s: Minimizing Directional Derivative

Up until version 2.7, the greedy rule used in LIBSVM was the Gauss-Southwell-s (GS-s) rule. The 159 GS-s rule chooses the coordinates resulting in the d k with the most-negative directional derivative. 160 This is a natural generalization of the idea of steepest descent, and the first uses of the method that 161 we aware of are by Keerthi et al. [2001] for SVMs and by Shevade and Keerthi [2003] for 1-norm 162 regularized optimization. For problem (13) the GS-s rule chooses 163

This is similar to the unbounded greedy rule (5) but excludes variables where the update would 164 immediately violate a bound constraint. 165

Unfortunately, the per-iteration decrease in f obtained by the GS-s rule can be arbitrarily small. In 166 particular, consider the case where the variable i maximizing ∇ i f ( x k ) has a value of x k i = l i + /epsilon1 167 for an arbitrarily small /epsilon1 . In this case, we would choose i k and take an arbitrarily small step of 168

δ k = /epsilon1 . Steps like this that truncate δ k are called 'bad' steps, and the GS-s rule does not guarantee a 169 non-trivial decrease in f on bad steps. If we only have bound constraints and do not have an equality 170 constraint (so we can update on variable at a time), Karimireddy et al. [2019] show that at most half 171 of the steps are bad steps. Their argument is that after we have taken a bad step on coordinate i , 172 then the next time i is chosen we will not take a bad step. However, with an equality constraint it is 173 possible for a coordinate to be involved in consecutive bad steps. It is possible that a combinatorial 174 argument similar to Lacoste-Julien and Jaggi [2015, Theorem 8] could bound the number of bad 175 steps, but it is not obvious that we do not require an exponential total number of bad steps. 176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

3.2 GS-q: Minimum 2-Coordinate Approximation

A variant of the Gauss-Southwell-q (GS-q) rule of Tseng and Yun [2009]for problem (13) is

This minimizes a quadratic approximation to the function, restricted to the feasible set. For problem (13), the GS-q rule is equivalent to choosing i k and j k to maximize (14), the distance that we move. We show the following result for the GS-q rule in Appendix D.

Theorem 3.1. Let f be a differentiable function whose gradient is 2-coordinate-wise Lipschitz (7) and restricted to the set where x T 1 = γ and l i ≤ x i ≤ u i . If this function satisfies the proximalPL inequality in the 2-norm [Karimi et al., 2016] for some positive µ 2 , then the iterations of the 2-coordinate update (3) with δ k given by (14) , α k = 1 /L 2 , and the greedy GS-q rule (16) satisfy:

The proof of this result is more complicated than our previous results, relying on the concept of conformal realizations used by Tseng and Yun [2009]. We prove the result for general block sizes and then specialize to the two-coordinate case. Unlike the GS-s rule, this result shows that the GS-q guarantees non-trivial progress on each iteration. Note that while this result does have a dependence on the dimension n , it does not depend on n 2 as the random rate (12) does. Moreover, the dependence on n can be improved by increasing the block size.

Unfortunately, the GS-q rule is not always efficient to use. As discussed by Beck [2014], there is no known algorithm faster than O ( n 2 ) for computing the GS-q rule (16). One special case where this can be solved in O ( n ) given the gradient is if we only have lower bounds (or only have upper bounds) [Beck, 2014]. An example with only lower bounds is our motivating problem of optimizing over the probability simplex, which only requires variables to be non-negative and sum to 1. On the other hand, our other motivating problem of SVMs requires lower and upper bounds so computing the GS-q rule would require O ( n 2 ) . Beginning with version 2.8, LIBSVM began using an approximation to the GS-q rule that can be computed in O ( n ) . In particular, LIBSVM first chooses one coordinate using the GS-s rule, and then optimizes the other coordinate according to a variant of the GS-q rule [Fan et al., 2005]. 1 While other rules have been proposed, the LIBSVM rule remains among the best-performing methods in practice [Horn et al., 2018]. However, similar to the GS-s rule we cannot guarantee non-trivial progress for the practical variant of the GS-q rule used by LIBSVM.

3.3 GS-1: Steepest Descent in the 1-Norm

Rather than using the classic GS-s or GS-q selection rules, the Gauss-Southwell-1 (GS-1) rule performs steepest descent in the 1-norm. For problem (13) this gives the update

The GS-1 rule was proposed by Song et al. [2017] for (unconstrained) 1-norm regularized problems. To analyze this method, we modify the definition of D ( x, L ) in the proximal-PL assumption to be

We then have the following dimension-independent convergence rate for the GS-1 rule. 209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

Theorem 3.2. Let f be a differentiable function whose gradient is 2-coordinate-wise Lipschitz (7) and restricted to the set where x T 1 = γ and l i ≤ x i ≤ u i . If this function satisfies the proximal-PL inequality in the 1-norm (9) for some positive µ 1 with the definition (19) , then the iterations of the update x k +1 = x k + d k with the greedy rule (18) and α k = 1 /L 1 = 2 /L 2 satisfy:

Proof. The proof follows the same reasoning as Theorem 2.3, but beginning after the application of Lemma 2.1 since we are directly computing the steepest descent direction.

This GS-1 convergence rate is at least as fast as the convergence rate for GS-q, and thus by exploiting a connection to the 1-norm we once again obtain a faster dimension-independent rate. In Algorithm 1 we give a method to construct a solution to the GS-1 rule (18) in O ( n log n ) time (due to sorting the ∇ i f ( x k ) values). Thus, our new GS-1 update guarantees non-trivial progress at each step (unlike the GS-s rule) and is efficient to compute (unlike the GS-q rule). The precise logic of Algorithm 1 is somewhat complicated, but it can intuitively be viewed as a version of GS-s that fixes the bad steps where δ k is truncated. Roughly, if the GS-s rule gives a bad step then the GS-1 moves the violating variable to its boundary and then may also update the variable with the next largest/smallest ∇ i f ( x k ) .

The drawback of the GS-1 update is that it is not strictly a 2-coordinate method. While the GS-1 224 update moves at most 2 variables within the interior of the bound constraints, it may move additional 225 variables to their boundary. The iteration cost of the method will be higher on iterations where more 226 than 2 variables are updated. However, by using an argument similar to Sun et al. [2019], we can show 227 that the GS-1 rule will only update more than 2 variables on a finite number of early iterations. This 228 is because, after some finite number of iterations, the variables actively constrained by their bounds 229 will remain at their bounds. At this point, each GS-1 update will only update 2 variables within the 230

interior of the bounds. In the case of SVMs, moving a variable to its lower bound corresponds to 231 removing it as a potential support vector. Thus, this 'bug' of GS-1 that it may update more than 2 232 variables can allow it to quickly remove many support vectors. In our experiments, we found that 233 GS-1 identified the support vectors more quickly than other rules and that most GS-1 updates only 234 updated 2 or 3 coordinates. 235

236

4 Greedy Updates using Coordinate-Wise Lipschitz Constants

237

238

239

240

241

242

243

244

245

246

247

248

249

250

Up until this point, we have measured smoothness based on the maximum blockwise Lipschitzconstant L 2 . An alternative measure of smoothness is Lipschitz continuity of individual coordinates. In particular, coordinate-wise Lipschitzness of coordinate i requires that for all x and α

where e i is a vector with a one in position i and zeros in all other positions. For twice-differentiable convex functions, the Lipschitz constant with respect to the block ( i, j ) is upper bounded by the sum of the coordinate-wise constants L i and L j [Nesterov, 2012, Lemma 1]. For equality-constrained optimization, Necoara et al. [2011] uses the coordinate-wise Lipschitz constants to design sampling distributions for i k and j k . Their analysis gives rates that can be faster than uniform sampling (12).

In Appendix E, we consider greedy rules that depend on the L i values for the equality-constrained case. In particular, we show that the equality-constrained GS-q rule chooses i k and j k by solving

which yields the standard greedy rule (5) if all L i values are equal. We show that the coordinate descent update with this selection rule and

can be written as steepest descent in the norm defined by || d || L /defines ∑ i √ L i | d i | . This yields a convergence rate that can be faster than the greedy rate (11).

Unfortunately, it is not obvious how to solve (21) faster than O ( n 2 ) . Nevertheless a reasonable 251 approximation is to use 252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

which we call the ratio approximation. This approximation is (21) after re-parameterizing in terms of variables x i / √ L i so that all coordinate-wise Lipschitz constants are 1 in the transformed problem. We can also use this re-parameterization to implement variations of the GS-s/GS-q/GS-1 rules if we also have bound constraints. While the ratio approximation (23) performed nearly as well as the more expensive (21) in our experiments, we found that the gap could be improved slightly if we choose one coordinate according to the ratio approximation and then the second coordinate to optimize (21). 2

5 Experiments

Our first experiment evaluates the performance of various rules on a synthetic equality-constrained least squares problem. Specifically, the objective is f ( x ) = 1 2 || Ax -b || 2 subject to x T 1 = 0 . We generate the elements of A ∈ R 1000 × 1000 from a standard normal and set b = Ax + z where x and z are generated from standard normal distributions. We also consider a variant where each column of A is scaled by a sample from a standard normal to induce very-different L i values. In Figure 1 we compare several selection rules: random i k and j k , the greedy rule (5), sampling i k and j k proportional to L i , the exact greedy L i rule (21), the ratio greedy L i rule (23), and a variant where we set one coordinate using (23) and other using (21) (switching between the two). All algorithms use the update (22). In these experiments we see that greedy rules lead to faster convergence than random rules in all cases. We see that knowing the L i values does not significantly change the performance of the random method, nor does it change the performance of the greedy methods in the case when

the L i were similar. However, with different L i the (expensive) exact greedy method exploiting L i 271 works much better. We found that the ratio method worked similar to or better than the basic greedy 272 method (depending on the random seed), while the switching method often performed closer to the 273 exact method. 274

.

Our second experiment considers the same problem but with the additional constraints x i ∈ [ -1 , 1] 275 Figure 2 compares the GS-s, GS-q, and GS-1 rules in this setting. We see that the GS-s rule results in 276 the slowest convergence rate, while the GS-q rule rule takes the longest to identify the active set. The 277 GS-1 rule typically updates 2 or 3 variables, but on early iterations it updates up to 5 variables. 278

6 Discussion

279

280

281

282

283

284

285

Despite the popularity of LIBSVM, up until this work we did not have a strong justification for using greedy 2-coordinate methods over simpler random 2-coordinate methods for equality-constrained optimization methods. This work shows that greedy methods may be faster by a factor ranging from O ( n ) up to O ( n 2 ) . This work is the first to identify the equivalence between the greedy 2-coordinate update and steepest descent in the 1-norm. The connection to the 1-norm is key to our simple analyses and also allows us to analyze greedy rules depending on coordinate-wise Lipschitz constants.

For problems with bound constraints and equality constraints, we analyzed the classic GS-q rule but 286 also proposed the new GS-1 rule. Unlike the GS-s rule the GS-1 rule guarantees non-trivial progress 287 on each iteration, and unlike the GS-q rule the GS-1 rule can be implemented in O ( n log n ) . We 288 further expect that the GS-1 rule could be implemented in O ( n ) by using randomized algorithms, 289 similar to the techniques used to implement O ( n ) -time projection onto the 1-norm ball Duchi et al. 290 [2008], van den Berg et al. [2008]. The disadvantage of the GS-1 rule is that on some iterations it may 291 update more than 2 coordinates on each step. However, when this happens the additional coordinates 292 are simply moved to their bound. This can allow us to identify the active set of constraints more 293 quickly. For SVMs this means identifying the support vectors faster, giving cheaper iterations. 294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

References

Amir Beck. The 2-coordinate descent method for solving double-sided simplex constrained minimization problems. Journal of Optimization Theory and Applications , 162(3):892-919, 2014.

Dimitri P Bertsekas. Network optimization: continuous and discrete models . Athena Scientific Belmont, 1998.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the l 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning , pages 272-279, 2008.

Rong-En Fan, Pai-Hsuen Chen, Chih-Jen Lin, and Thorsten Joachims. Working set selection using second order information for training support vector machines. Journal of machine learning research , 6(12), 2005.

Qin Fang, Min Xu, and Yiming Ying. Faster convergence of a randomized coordinate descent method for linearly constrained optimization problems. Analysis and Applications , 16(05):741-755, 2018.

Daniel Horn, Aydın Demircio˘ glu, Bernd Bischl, Tobias Glasmachers, and Claus Weihs. A comparative study on large scale kernelized support vector machines. Advances in Data Analysis and Classification , 12(4):867-883, 2018.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the polyakł ojasiewicz condition. In Joint European conference on machine learning and knowledge discovery in databases , pages 795-811. Springer, 2016.

Sai Praneeth Karimireddy, Anastasia Koloskova, Sebastian U Stich, and Martin Jaggi. Efficient greedy coordinate descent for composite problems. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2887-2896. PMLR, 2019.

Sai Praneeth Reddy Karimireddy, Sebastian Stich, and Martin Jaggi. Adaptive balancing of gradient and update computation times using global geometry and approximate subproblems. In International Conference on Artificial Intelligence and Statistics , pages 1204-1213. PMLR, 2018.

Sathiya Keerthi, Shirish Krishnaj Shevade, Chiranjib Bhattacharyya, and Karuturi Radha Krishna Murthy. Improvements to platt's smo algorithm for svm classifier design. Neural computation , 13 (3):637-649, 2001.

Simon Lacoste-Julien and Martin Jaggi. On the global linear convergence of frank-wolfe optimization variants. Advances in neural information processing systems , 28, 2015.

I Necoara, Y Nesterov, and F Glineur. A random coordinate descent method on large optimization problems with linear constraints. Technical Report, University Politehnica Bucharest , 2011.

Ion Necoara and Andrei Patrascu. A random coordinate descent algorithm for optimization problems with composite objective function and linear coupled constraints. Computational Optimization and Applications , 57(2):307-337, 2014.

Ion Necoara, Yurii Nesterov, and François Glineur. Random block coordinate descent methods for linearly constrained optimization over networks. Journal of Optimization Theory and Applications , 173(1):227-254, 2017.

Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

Julie Nutini. Greed is good: greedy optimization methods for large-scale structured problems . PhD thesis, University of British Columbia, 2018.

Julie Nutini, Mark Schmidt, Issam Laradji, Michael Friedlander, and Hoyt Koepke. Coordinate descent converges faster with the gauss-southwell rule than random selection. In International Conference on Machine Learning , pages 1632-1641. PMLR, 2015.

Julie Nutini, Issam Laradji, and Mark Schmidt. Let's make block coordinate descent converge faster: Faster greedy rules, message-passing, active-set complexity, and superlinear convergence. Journal of Machine Learning Research , 23(131):1-74, 2022.

Peter Richtárik and Martin Takáˇ c. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematical Programming , 144(1):1-38, 2014.

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

Jennifer She and Mark Schmidt. Linear convergence and support vector identification of sequential minimal optimization. In 10th NIPS Workshop on Optimization for Machine Learning , volume 5, page 50, 2017.

Shirish Krishnaj Shevade and S Sathiya Keerthi. A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics , 19(17):2246-2253, 2003.

Chaobing Song, Shaobo Cui, Yong Jiang, and Shu-Tao Xia. Accelerated stochastic greedy coordinate descent by soft thresholding projection onto simplex. Advances in Neural Information Processing Systems , 30, 2017.

Yifan Sun, Halyun Jeong, Julie Nutini, and Mark Schmidt. Are we there yet? manifold identification of gradient-related proximal methods. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1110-1119. PMLR, 2019.

Paul Tseng and Sangwoon Yun. Block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization. Journal of optimization theory and applications , 140(3): 513-535, 2009.

van den Berg, M. Schmidt, M. Friedlander, and K. Murphy. Group sparsity via linear-time projection. 2008.

Po-Wei Wang and Chih-Jen Lin. Iteration complexity of feasible descent methods for convex optimization. The Journal of Machine Learning Research , 15(1):1523-1548, 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents new update rules for block-coordinate descent (BCD) methods to minimize a smooth function subject to one linear equality constraint (precisely, all variables must sum to 1) and possibly a box constraint. A popular method to solve large-scale problems of this type is BCD with blocks of size 2 (i.e. we always update 2 coordinates at a time). A prominent example is SVM training in LibSVM.\n\nFor problems without the box constraint, the key observation is lemma 2.1, which shows that greedy version of such updates is equivalent to 1-norm steepest descent. This allows the authors to employ known results from non-smooth optimization and propose better update rules and derive their convergence rate. This rate is better than known rates for non-greedy updates.\n\nFor problems with a box constraint, the situation is more complicated because Lemma 2.1 does not apply. The authors nevertheless propose a new update (GS-1) based on 1-norm steepest descent, which is not guaranteed to change at most 2 variables in every update, but more than 2 variables are changed only in a finite number of updates. This can be seen as a greedy version of BCD with small blocks. The update has a better convergence rate than known updates that can be computed in reasonable (less than $O(n^2)$) time.\n\nThe theory is supported by a synthetic toy experiment on the linear LS problem with one linear equality constraint and a box constraint.\n\nSTRENGTHS:\nCrisp and elegant theoretical result, non-trivial.\nHowever, let me admit that convergence analysis of optimization algorithms is not precisely my expertise, so I am not reliably judge the novelty (these or similar results may be already known or obvious, ...).\n\nThe text is precise and clear.\n\nWEAKNESSES:\nA major weakness is experiments: they are limited only to a simple problem (linear least squares) on synthetic data, end even then the data variability tested is small (only square system). To my understanding, the GS-1 update has never been applied to SVM training - so such experiments would be very interesting.\n\nMoreover, in the experiments the difference $f(x)-f^*$ (vertical axis in Figure 1) is never shown to converge to zero, it is shown only in the range $10^{4.8} - 10^{5.2}$. In my experience, it sometimes happens that greedy updates improve the objective better initially but later they slow down and cyclic/random updates catch up. Or, perhaps, the vertical axis in Figure 1 shows $f(x)$ rather than $f(x)-f^*$ (as in the Supplement)?\n\nMinor issues in the text:\n\n- The five-line derivation on line 79-80 is trivial - it can be shortened to save space.\n\n- Lemma 2.1 is very much related to the well-known result that greedy coordinate descent is equivalent to 1-norm steepest descent (see, e.g., Section 9.4 in Boyd's book on convex optimization). This may deserve a citation.\n\n- Lemma 2.1 would be clearer if $\\nabla f(x)$ were replaced by a general vector, say c.\n\n- Line 128: mentioning SVM is not OK here because SVM needs also a box constraint.\n\n- Line 139: the notion of \"dimension independent convergence rate\" is not well-defined because the problem dimension can be hidden in constants (such as $L_1, \\mu_1$). But I understand that this notion may be widely used.\n\nQUESTIONS:\nHow would Figure 1 look like if the iterations continued to a very small (10^-6) error f(x)-f^* ?\n\nHow does the GS-1 update behave in SVM training?\n\nAre there any other problem types on which you tested the new updates? Were there any negative results (i.e., the convergence was slow in practice despite theoretical guarantees)?\n\nLIMITATIONS:\nThe experiments are limited to a very simple problem with synthetic data.\n\nThough the theoretical results are elegant, the optimization problem considered has a rather limited applicability.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe first goal of the paper is to minimize a smooth function subject to a summation constraint. The authors demonstrate that the greedy 2-coordinate descent (CD) method, when applied to the problem with equality constraints, achieves a linear rate of convergence under the proximal PL inequality under the L1-norm formulation. Notably, this convergence rate remains unaffected by the problem dimension, which sets it apart from random selection methods. Furthermore, they establish that there exists at least one steepest descent direction with respect to L1-norm, which can be utilized as a 2-coordinate descent update. They leverage this relationship to derive the convergence rates of the CD algorithm.\nAdditionally, they explore the minimization involving a summation constraint and prescribed lower and upper bounds on the coordinates. They demonstrate that employing bound- and summation-constrained steepest descent in the L1-norm guarantees significant progress at each step, unlike the GS-s rule. Moreover, this method (called GS-1) is computationally more efficient than GS-q, requiring only $O(n \\log(n))$ iterations instead of $O(n^2)$\n\nSTRENGTHS:\n1. The paper gives linear convergence rates for greedy 2-coordinates CD, and the steepest descent in l1-norm under proximal PL for l1-norm. For problems with equality constraints, this work shows that greedy methods may be faster by a factor ranging from $O(n)$ up to $O(n^2)$ than methods picking coordinates at random,\n\n2. For problems with bound constraints and equality constraints, the paper shows that GS-1 rule has the benefits of both GS-q and GS-s. Contrary to GS-q, GS-1 rule can be implemented in $O(n \\log n)$ and contrary to GS-s, GS-1 guarantees non-trivial progress\nat each iteration.\n\n3. The authors also proved a linear convergence rate for GS-q rule under proximal PL for L2-norm. It is not straightforward to use prox-PL to obtain the linear convergence rate for this update rule. They consider the conformal realization used by Tseng and Yun [2009] to upper bound the extra terms in descent inequality by a notion of gradient mapping used in prox-PL’s definition.\n\n4. The paper is clear and well written.\n\nWEAKNESSES:\nThe experimental part is limited, which is fine for a theoretical optimization paper, but for a ML venue and given the claim made in the paper that many problems in ML require to satisfy an equality constraint, such as discrete probabilities or SVMs with an unregularized bias term, one could expect the experimental section to show the power of GS-1 for other such examples than only a synthetic equality-constrained least square problem.\n\nQUESTIONS:\n1. L42-44: More comments are needed when the authors write that ``despite LIBSVM being perhaps the most widely-used CD method of all time, current analyses of greedy 2-coordinate updates either result in sublinear convergence rates or do not lead to faster rates than random selection [Tseng and Yun, 2009, Beck, 2014].\" Did they use proximal PL inequality in L1-norm to analyze the convergence rate of greedy 2-coordinate methods? The linear convergence rate of this method may be due to the PL property.\n\n2. L47-49 \"The analysis is based on an equivalence between the greedy update and equality-constrained steepest descent in the 1-norm. This leads to a simple dimension-independent analysis of greedy selection showing that it can converge substantially faster than random selection.\" Under which assumptions is the faster convergence proven?\n\n3. L. 66 ``If f is continuous, this update is guaranteed to decrease $f$ for sufficiently small $\\alpha_k$'' Please quote a relevant paper, or a short proof for this sentence.\n\n4. Question 4 removed as workshop papers are not listed in reviews.\n\n5. L91 “For lower bounding sub-optimality in terms of the 1-norm, we introduce the proximal-PL inequality in the 1-norm.” What do you mean by lower bounding sub-optimality? As far as I can see, you are trying to give an upper bound on sub-optimality in Theorem 2.3.\n\n6. Definition 2.2. “A function, that is L1-Lipschitz”:  is it L1-Lipschitz gradient? Did this definition not appear in [Karimi et al. 2016]?\n\n7. Does $f=g(Ax)$ satisfy proximal PL condition in the L1-norm for strongly convex function $f$ and a singular matrix $A$? Can we verify this property for functions in practice such as the SVM dual problem?\n\n8. Did [She and Schmidt 2017] assume the same PL condition in the L1-norm (or L2-norm)? \n\n9. Connection between the prox-PL function used in Theorem 3.2 and prox-PL as defined only by (9) in Def. 2.2: could you give an example of a function that satisfies this new assumption?\n\n10. Some minor questions/typos in the appendix:\n\nL372 Should $d_1 + d_2$ not be $d_i + d_j$\n\nL378 second line before end: one $)$ too much\n\nL390 Why do you have $\\lambda 1$ in (29) and not simply $\\lambda$ \n\nL428 $d_1 + d_2 + d_3$ should be $d_1 + d_2 - d_3$\n\nL437 Why do we have the parameter $x^\\prime$Conv$(\\{x,x+d\\})$~?\n\n----\nThe authors have addressed adequately my questions; they intend to conduct extensive experiments. This is indeed missing in the current version of the paper, it is difficult to evaluate that part with the paper as is. But I find the theoretical contribution solid enough to justify publication.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies new coordinate descent-type methods for equality-constrained problems, where 2 coordinates are updated on each iteration and proves new convergence guarantees under suitable proximal-PL conditions that allow to obtain linear convergence rates for the proposed methods. \nThe first main result considers the case of a single constraint on the sum of coordinates with a simple greedy rule for selecting the two coordinates to update. A linear dimension-independent convergence rate is established using a fixed step-size and assuming PL inequality w.r.t. 1 norm.\n\nNext the authors consider the more challenging setting in which on top to the constraints on the sum of coordinates, there are also box constraints on the individual coordinates. Here the authors propose to use a greedy-proixmal update named Gauss-Southwell-q which leads to a linear rate using a fixed step-size and assuming PL inequality w.r.t. 2 norm, however with linear dependence in exponent of the dimension.\n\nFinally, in the latter setting (equality constraint on sum + box constraints), considering a PL inequality w.r.t. 1 norm and a diffrerent greedy rule (which corresponds to solving a proximal-style problem w.r.t to 1 norm, which given the gradient direction could be solved in O(nlogn) time), the authors obtain a linear dimension-independent rate.\n\nThe proofs hinge on relating the 2-coordinate descent updates to the steepest descent method w.r.t. the 1 norm, which is interesting.\n\nThe authors present experiments that support their findings on a random least squares problem with equality (and then also box) constraints.\n\nSTRENGTHS:\nThe paper is very well written and is easy to follow. The results and ideas are clearly presented. I enjoyed reading it. \n\nThe problems considered are sound and make sense, and the type of methods discussed is of clear practical interest to the community.\n\nThe technical argument of relating 2-coordinate descent to steepest descent w.r.t. 1 norm is interesting and might be of further use.\n\nThe convergence rates obtained are novel and interesting to the best of my knowledge.\n\nWEAKNESSES:\n1. I think the authors could do a better job in rigorously comparing the complexity of their methods to previous ones. It is not clear at all times if the obtained complexities are state-of-the-art (when not considering only 2-CD greedy methods). In particular, from first paragraph of 2nd page, it is not clear how this compete with known random selection methods.\n\n2. To continue the above point, it seems all selection rules suggested require the full gradient direction. However, in such cases why these methods are better than full-gradient methods? For instance, for unconstrained least squares indeed doing an update to a single coordinate is much faster than computing entire gradient, and makes perfect sense when using random selection (e.g., w.r.t. to coordinate-wise Lipchitz parameters). But in the greedy rules we already need the full gradient. \n\n3. The linear rates that are dimension-independent rely on PL w.r.t 1 norm. This might seem as ``hiding the dimension under the choice of norm''. Could the authors provide examples of interest that will satisfy this inequality with constant independent of the dimension? Or more precisely, that the ratio of the 1 norm constant and 2 norm constant is dimension independent?\n\n4. At some point the authors claim that computing the coordinate-wise Lipchitz parameters can be difficult, however, in the most relevent setting I guess which allows also for the PL inequality: f(x) = g(Ax), with g strongly convex, in case g is well conditioned (e.g., squared Euclid norm in many cases), it is easy to estimate them I believe from the columns of A.\n\n5. Throughout the paper the authors mention SVM as a major application, so why not conduct experiments with this application?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies minimizing a smooth function with a summation equality constraint over its variables. The authors show a connection between the greedy 2-coordinate update and steepest descent w.r.t. 1-norm, and introduce a new proximal PL assumption w.r.t. 1-norm. They show improved convergence rates under such assumption over random selection. The authors also consider coordinate-wise Lipschitz smoothness and introduce an approximation for greedy 2-coordinate methods. They complement their theoretical results through numerical experiments on randomly generated problems.\n\nSTRENGTHS:\n1. The connection from 2-coordinate greedy update to steepest descent w.r.t. 1 norm seems non-trival and potentially interesting for equality-constrained problems.\n2. The authors offer detailed comparisons on various Gauss-Southwell update rules (GS-1, GS-q and GS-s).\n\nWEAKNESSES:\nI believe this work has potential but lacks a few key ingredients.\n1) This work lacks justification for their proximal-PL condition w.r.t. 1-norm. For example, Karimi et al (2016) provides five important classes of functions that statisfy the proximal-PL condition w.r.t. 2-norm. It is unclear to me what function classes can statisfy the proximal-PL condition w.r.t. 1-norm where the worst case dependence on $n$ for $\\mu_1$ can be avoided. It would also be useful if there can be empirical comparisions for $\\mu_1$ against $\\mu_2$ as $n$ scales. At its current state, I do not think it is reasonable to claim that the convergence rate is independent of the problem dimension $n$.\n2) I would like to see experiments performed on practical datasets (e.g., LIBSVM datasets) rather than randomly generated Gaussian data, since much of the theoretical results aim to improve upon the rules in LIBSVM. I would also like to see experiments on SVM problems to support their claimed improvements for SVM.\n3) I suggest adding markers to the plots in Figure 1 and 2, as the overlapping nature of the results renders some plots unparsable.\n\nQUESTIONS:\n1. What are the main differences of GS-1 rule (Algorithm 1) for proximal-PL assumption w.r.t. 1-norm, compared to Song et al. (2017)?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-30"}