Title: ReHLine: Regularized Composite ReLU-ReHU Loss Minimization with Linear Computation and Linear Convergence
Venue: NeurIPS 2023
Accepted: yes
Keywords: coordinate descent

linear convergence

primal-dual methods

empirical risk minimization

linear constraints

quantile regression
Abstract: Empirical risk minimization (ERM) is a crucial framework that offers a general approach to handling a broad range of machine learning tasks. In this paper, we propose a novel algorithm, called ReHLine, for minimizing a set of regularized ERMs with convex piecewise linear-quadratic loss functions and optional linear constraints. The proposed algorithm can effectively handle diverse combinations of loss functions, regularization, and constraints, making it particularly well-suited for complex domain-specific problems. Examples of such problems include FairSVM, elastic net regularized quantile regression, Huber minimization, etc. In addition, ReHLine enjoys a provable linear convergence rate and exhibits a per-iteration computational complexity that scales linearly with the sample size. The algorithm is implemented with both Python and R interfaces, and its performance is benchmarked on various tasks and datasets. Our experimental results demonstrate that ReHLine significantly surpasses generic optimization solvers in terms of computational efficiency on large-scale datasets. Moreover, it also outperforms specialized solvers such as Liblinear in SVMs, hqreg in Huber minimization, and Lightning (SAGA, SAG, SDCA, SVRG) in smoothed SVMs, exhibiting exceptional flexibility and efficiency. The source code, project page, accompanying software, and the Python/R interface can be accessed through the link: https://github.com/softmin/ReHLine.

ReHLine: Regularized Composite ReLU-ReHU Loss Minimization with Linear Computation and Linear Convergence

Abstract

Empirical risk minimization (ERM) is a crucial framework that offers a general approach to handling a broad range of machine learning tasks. In this paper, we propose a novel algorithm, called ReHLine, for minimizing a set of regularized ERMs with convex piecewise linear-quadratic loss functions and optional linear constraints. The proposed algorithm can effectively handle diverse combinations of loss functions, regularization, and constraints, making it particularly well-suited for complex domain-specific problems. Examples of such problems include FairSVM, elastic net regularized quantile regression, Huber minimization, etc. In addition, ReHLine enjoys a provable linear convergence rate and exhibits a per-iteration computational complexity that scales linearly with the sample size. The algorithm is implemented with both Python and R interfaces, and its performance is benchmarked on various tasks and datasets. Our experimental results demonstrate that ReHLine significantly surpasses generic optimization solvers in terms of computational efficiency on large-scale datasets. Moreover, it also outperforms specialized solvers such as LIBLINEAR in SVMs, HQREG in Huber minimization and LIGHTNING (SAGA, SAG, SDCA, SVRG) in smoothed SVMs, exhibiting exceptional flexibility and efficiency. The source code, project page, accompanying software, and the Python/R interface can be accessed through the link: ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴/a115♦❢/a116♠✐♥✴❘❡❍▲✐♥❡ .

1 Introduction

Empirical risk minimization (ERM) with different losses is a fundamental framework in statistical machine learning [43], which has demonstrated a significant impact on many application areas, such as artificial intelligence, computational biology, computer vision, natural language processing, electronic engineering, and finance. In fact, many common practical ERMs are associated with a piecewise linear-quadratic (PLQ; [34]) loss function, including support vector machines (SVMs; [7]), the least absolute deviation regression, quantile regression [26], Huber regression [22], borrowing cost function in portfolio selection [28], and pricing/planning problems in electricity and operational research [16, 1], among many others. In addition, ERMs are frequently blended with linear constraints, for instance, the fairness constraints in fair classification [48], monotonicity constraints in classification or regression, etc. More linear constraints according to the prior domain knowledge, such as the sum-to-one constraints in the portfolio variance minimization problem [24], negative or positive constraints in non-performing loan [36], are considered in various real applications. Therefore, it has become of utmost importance to develop fast and scalable numerical algorithms to solve ERMs with PLQ losses based on large-scale datasets and flexible linear constraints.

In this paper, we consider a general regularized ERM based on a convex PLQ loss with linear constraints:

where x i ∈ R d is the feature vector for the i -th observation, β ∈ R d is an unknown coefficient vector, A ∈ R K × d and b ∈ R K are defined as linear inequality constraints for β , and L i ( · ) ≥ 0 is a convex PLQ loss function. The PLQ function class greatly generalizes existing popular loss functions, and the convexity of L i ( · ) guarantees the global convergence of optimization. Here, we focus on working with a large-scale dataset, where the dimension of the coefficient vector and the total number of constraints are comparatively much smaller than the sample size, that is, d ≪ n and K ≪ n .

Although (1) is a strongly convex problem with affine constraints, thus admitting a unique global optimum, directly solving (1) by a generic solver may encounter various challenges. For example, L i ( · ) is in general non-smooth, so gradient-based methods may fail, and hence slow-convergent subgradient methods need to be used. In addition, the constraint set is a polyhedron, whose projection operator is non-trivial to compute, making various projection-based methods difficult to apply.

Of course, for some specific forms of L i ( · ) , highly efficient solvers for (1) indeed exist. One remarkable example is the LIBLINEAR solver [12], which has gained great success in solving largescale SVMs. However, LIBLINEAR highly relies on the hinge loss function, so its success does not directly transfer to more general loss functions.

Outline. To this end, in this article we develop a novel algorithm to solve (1) with the greatest generality. The proposed algorithm, named ReHLine, has three key ingredients. First, we show that any convex PLQ loss function can be decomposed as the sum of a finite number of rectified linear units (ReLU, [15]) and rectified Huber units (ReHU, defined in Section 2). Second, based on this decomposition, the dual problem of (1) is shown to be a box-constrained quadratic programming (box-QP) problem. Finally, a special paired primal-dual coordinate descent algorithm is developed to simultaneously solve the primal and dual problems, with a computational cost linear in n .

We emphasize that for the second point, the dual box-QP problem is highly structured. Therefore, a general-purpose solver may not fully exploit the structure, thus leading to slower convergence rates or higher computational costs. As a direct comparison between the proposed ReHLine solver and existing algorithms, including the projected (sub-)gradient descent (P-GD; [6]), the coordinate descent (CD; [45]), the interior-point methods (IPMs; [13, 14, 18]), the stochastic dual coordinate ascent (SDCA; [39, 40]) related methods, and the alternating direction method of multipliers (ADMM; [5]), Table 1 summarizes the required number of iterations and per-iteration cost of each method. More details and discussions for this comparison are given in Section 4.

Contribution. Compared with other specialized ERM solvers or general-purpose box-QP solvers, the proposed ReHLine solver has four appealing 'linear properties':

It applies to any convex piecewise linear-quadratic loss function (potential for non-smoothness included), including the hinge loss, the check loss, the Huber loss, etc.

In addition, it supports linear equality and inequality constraints on the parameter vector.

The optimization algorithm has a provable linear convergence rate.

The per-iteration computational complexity is linear in the sample size.

Moreover, ReHLine is designed to be a computationally efficient and practically useful software package for large-scale ERM problems. As an example, our experiments in Section 5 have shown that ReHLine significantly surpasses generic solvers in terms of computational efficiency on large-scale datasets, and provides reasonable improvements over specialized solvers and problems, such as LIBLINEAR, HQREG and LIGHTNING, given that LIBLINEAR, HQREG and LIGHTNING are one of the fastest and most widely-used solvers for SVM, Huber minimization, and smoothed SVM, respectively. Finally, ReHLine is available as user-friendly software in both Python and R, which facilitates ease of use for researchers, practitioners, and a wider audience.

2 The ReLU-ReHU decomposition

As is introduced in Section 1, we attempt to solve ERM with a general loss function, and one sensible choice is the class of convex PLQ functions. A univariate loss function f : R → R ≥ 0 is PLQ if there exist a finite number of knots t 1 < t 2 < · · · < t K such that f is equal to a linear or quadratic function on each of the intervals ( -∞ , t 1 ] , [ t 1 , t 2 ] , . . . , [ t K -1 , t K ] , [ t K , ∞ ) . In what follows we merely consider nonnegative convex PLQ functions, which also imply the continuity of the loss function. It is well known that PLQ functions are universal approximators of continuous functions [38], but the canonical definition of PLQ is not convenient for optimization algorithms. Instead, in this section we prove that convex PLQ functions can actually be decomposed into a series of simple and basic units.

1

0

1

2

3

4

5

0

2

4

6

8

10

12

ReLU

ReHU-1

ReHU-2

ReHU-3

ReQU/ReHU-

Before proceeding, we introduce a generalized quadratic version of ReLU, the so-called rectified Huber unit, ReHU:

where τ ∈ [0 , ∞ ] is a hyperparameter of ReHU switching from quadratic to linear form, see examples in Figure 1. ReHU also incorporates the rectified quadratic unit (ReQU) as a special case, that is, ReHU ∞ ( z ) = ReQU( z ) / 2 = ( z + ) 2 / 2 when τ = ∞ , where z + = max { z, 0 } . Then we give the characterization of a composite ReLUReHU function.

Definition 1 (Composite ReLU-ReHU function) . A function L ( z ) is composite ReLU-ReHU, if there exist u , v ∈ R L and τ , s , t ∈ R H such that

where ReLU( z ) = z + , and ReHU τ h ( z ) is defined in (2) .

Clearly, the composite ReLU-ReHU function class is closed under affine transformations, as is indicated by Proposition 1.

Proposition 1 (Closure under affine transformation) . If L ( z ) is a composite ReLU-ReHU function as in (3) , then for any c > 0 , p ∈ R , and q ∈ R , cL ( pz + q ) is also composite ReLU-ReHU, that is,

where u ′ l = cpu l , v ′ l = cu l q + cv l , τ ′ h = √ cτ h , s ′ h = √ cps h , and t ′ h = √ c ( s h q + t h ) .

More importantly, we show that the composite ReLU-ReHU function class is equivalent to the class of convex PLQ functions.

Theorem 1. A loss function L : R → R ≥ 0 is convex PLQ if and only if it is composite ReLU-ReHU.

Based on the fact that the loss functions of various SVMs (the hinge loss, the squared hinge loss, and the smoothed hinge loss), quantile regression (the check loss), and least absolute deviation (the absolute loss) are all convex PLQs, Theorem 1 suggests that Definition 1 incorporates numerous widely-used losses in statistics and machine learning applications. Table 2 provides a convenient translation from some commonly-used loss functions to their ReLU-ReHU representation.

Taken together, our algorithm is designed to address the empirical regularized ReLU-ReHU minimization problem, named ReHLine optimization , of the following form:

where U = ( u li ) , V = ( v li ) ∈ R L × n and S = ( s hi ) , T = ( t hi ) , τ = ( τ hi ) ∈ R H × n are the ReLU-ReHU loss parameters, as illustrated in Table 2, and ( A , b ) are the constraint parameters, as illustrated in Table 3. This formulation has a wide range of applications, including statistics, machine learning, computational biology, and social studies. Some popular examples include SVMs with fairness constraints (FairSVM), elastic net regularized quantile regression (ElasticQR), ridge Huber minimization (RidgeHuber), and smoothed SVMs (sSVM). See Appendix A for details.

Despite appearing to impose limitations on the l 2 -regularization in (4), our ReHLine formulation can actually be extended to more general regularization forms, including the elastic net penalty [49].

Proposition 2. If L i is a composite ReLU-ReHU loss function with parameters ( u li , v li ) L l =1 and ( s hi , t hi , τ hi ) H h =1 , then its elastic net regularized minimization with λ 1 ≥ 0 and λ 2 > 0 ,

can be rewritten as the form of ReHLine optimization (4) with

where ← indicates assigning a value to a parameter.

3 Optimization algorithm

In this section, we present the ReHLine algorithm to tackle the optimization in (4). The proposed algorithm is inspired by LIBLINEAR [12], which has gained great success in solving standard SVMs. The key idea of ReHLine is to leverage the linear property of Karush-Kuhn-Tucker (KKT) conditions, thus simplifying the iterative complexity of CD and achieving rapid convergence.

3.1 Primal and dual formulations

To proceed, we first demonstrate equivalent formulations for ReLU and ReHU, respectively, where the equivalence means that the value on the left is equal to the minimum value of the right-hand side:

It is worth noting that when τ = ∞ , the optimal solution for σ is 0, and hence the last constraint ( σ ≥ 0 ) can be removed. Then according to the definition of the composite ReLU-ReHU loss function and the equivalent forms in (5), the ReHLine optimization problem (4) can be rewritten as:

where Π = ( π li ) ∈ R L × n , Θ = ( θ hi ) ∈ R H × n and Σ = ( σ hi ) ∈ R H × n are slack variables. Theorem 2 shows the dual form of the primal formulation (6) by using the Lagrangian duality theory, and the relation between primal and dual variables is also provided.

Theorem 2. The Lagrangian dual problem of (6) is:

where ξ ∈ R K , Λ = ( λ li ) ∈ R L × n , and Γ = ( γ hi ) ∈ R H × n are dual variables, ¯ U (3) ∈ R d × nL and ¯ S (3) ∈ R d × nH are the mode-3 unfolding of the tensors ¯ U = ( u lij ) ∈ R L × n × d and ¯ S = ( s hij ) ∈ R H × n × d , respectively, u lij = u li x ij , s hij = s hi x ij , I is the identity matrix, and all inequalities are elementwise.

Moreover, the optimal point ̂ β of (6) can be recovered as:

3.2 ReHLine update rules

Indeed, (9) is a KKT condition of (6), and the main idea of the proposed ReHLine solver is to update the dual variables ( ξ , Λ , Γ ) by CD on (7) with an analytic solution , and simultaneously update the primal variable β by the KKT condition (9). Most importantly, the computational complexity of the CD updates on (7) can be significantly reduced by using the information of β .

Canonical CD updates. As a first step, we consider the canonical CD update rule that directly optimizes the dual problem (7) with respect to a single variable. For brevity, in this section we only illustrate the result for λ li variables, and the full details are given in Appendix B.

By excluding the terms unrelated to λ li , we have λ new li = argmin 0 ≤ λ ≤ 1 L li ( λ ) , where

̸

Therefore, by simple calculations we obtain

̸

where P [ a,b ] ( x ) = max( a, min( b, x )) means projecting a real number x to the interval [ a, b ] .

Clearly, assuming the values x ⊺ i a k and ∥ x i ∥ 2 2 are cached, updating one λ li value requires O ( K + nd + nL + nH ) of computation, and updating the whole Λ matrix requires O ( nL ( K + nd + nL + nH ) ) . Adding all variables together, the canonical CD update rule for one full cycle has a computational complexity of O ( ( K + nd + nL + nH )( K + nL + nH ) ) .

ReHLine updates. The proposed ReHLine algorithm, on the other hand, significantly reduces the computational complexity of canonical CD by updating β according to the KKT condition (9) after each update of a dual variable. To see this, let µ := ( ξ , Λ , Γ ) denote all the dual variables, and define

Then it can be proved that ( ∇ λ L li )( λ li ) = -( u li x ⊺ i β ( µ ) + v li ) . Therefore, when µ is fixed at µ old = ( ξ old , Λ old , Γ old ) and let β old = β ( µ old ) , (10) can be rewritten as

Accordingly, the primal variable β is updated as

which can then be used for the next dual variable update. Simple calculations show that this scheme only costs O ( d ) of computation for one λ li variable.

The update rules for other dual variables are similar, with the complete algorithm shown in Algorithm 1 and the full derivation details given in Appendix B. Overall, the ReHLine update rule has a computational complexity of O ( ( K + nL + nH ) d ) for one full cycle. Given that ( K,L,H ) are all small numbers, the complexity is linear with respect to both the sample size n and the dimension d .

We emphasize that the linear relationship (9) between the primal and dual variables, as well as its consequence of reducing the computational cost, is highly non-trivial. This technique has been proposed for specialized problems such as SVMs via the LIBLINEAR solver [21], and here we show that its success can be greatly generalized to any convex PLQ loss functions.

3.3 Global convergence rate

Finally, we show that the ReHLine algorithm has a fast global linear convergence rate.

Theorem 3. Let µ ( q ) := ( ξ ( q ) , Λ ( q ) , Γ ( q ) ) be a sequence of iterates generated by Algorithm 1, and µ ∗ be a minimizer of the dual objective (8) . Then µ ( q ) is feasible, and the dual objective value converges at least linearly to that of µ ∗ , that is, there exist 0 < η < 1 and q 0 such that for all q ≥ q 0 ,

4 Related work

Table 1 summarizes the existing methods in solving the primal or dual problem of (1). Note that Column #ITERATIONS is the best-fit theoretical results of convergence analysis for each algorithm based on our literature search, which may be improved with further developments of convergence analysis. These algorithms can be broadly divided into the following categories.

Generic QP solvers. For instance, a projected sub-gradient descent is readily applicable to the primal form (1) by computing the sub-gradient of each loss and the projection under affine constraints, although the projection operator itself is already difficult. As for the box-QP dual problem (7), CD converges linearly to the optimal solution [27, 31, 35, 42]. Box-QP can also be efficiently solved using IPM, which leverages second-order cone programming techniques [13, 14]. Moreover, ADMM was employed to iteratively minimize the augmented Lagrangian of (1).

ERMsolvers. There are multiple existing works developed to tackle ERM problems, such as SAG [37], SVRG [25], SAGA [8], proximal SDCA [40] and SDCA [39]. However, SAG, SVRG, SAGA and proximal SDCA can only handle smooth loss functions, with an optional non-smooth term that has an easy proximal operator. SDCA applies to general loss functions, but it requires the convex conjugate of loss functions, which is not necessarily simple to compute, also see discussion in Section 3.1 in [40]. Moreover, it only guarantees a sublinear convergence rate for non-smooth loss functions. In contrast, ReHLine supports all convex PLQ loss functions with optional general linear constraints, which could be non-smooth by construction, and they all enjoy a linear computational complexity and provable linear convergence.

For non-smooth loss functions, another existing method to solve ERM is the smoothing technique [40, 3, 30], which approximates the non-smooth terms by smooth functions, and then uses smoothbased algorithms to solve the smooth problem. However, the choice of the smoothing function and smoothing parameter typically requires additional knowledge, and the linear convergence rate is not always guaranteed.

Despite the successful use of the existing algorithms in solving (1), these methods either suffer from slow convergence (P-GD), or high computational costs (CD, IPM, and ADMM). This is partly because these general methods do not fully exploit the inherent structure of the problem. In contrast, ReHLine leverages the KKT condition in (9) to substantially improve numerical performance, thus achieving a linear computational complexity per iteration with respect to the sample size, as well as a linear convergence rate.

In addition to generic algorithms, specialized algorithms have been developed for specific types of (1). For example, the dual coordinate descent of LIBLINEAR in solving SVMs [21], the semismooth Newton coordinate descent algorithm of HQREG in solving the ridge regularized Huber minimization [46], and SAGA, SAG, proximal SDCA, and SVRG of LIGHTNING in solving smooth ERMs [4], such as smoothed SVMs. Our algorithm is also compared to these specialized methods in Section 5.

Software. Below are some commonly-used software for solving (1) based on the algorithms available.

CVX/CVXPY [17, 10]: a modeling language for convex optimization problems and supports a wide range of solvers such as ECOS, SCS, OSQP, and MOSEK.

MOSEK [2]: a commercial convex optimization solver based on IPM.

ECOS [11]: an open-source solver of second-order cone programming based on IPM.

SCS [32]: an open-source convex quadratic cone solver based on ADMM.

DCCP [41]: a disciplined convex-concave programming solver built on top of CVXPY, which has been used to solve classification problems with fairness constraints [47].

LIBLINEAR [21]: a library for large linear classification.

HQREG [46]: an R package solving Lasso or elastic net penalized Huber loss regression and quantile regression.

LIGHTNING [4]: a Python library solving various ERMs based on primal SDCA, SGD, AdaGrad, SAG, SAGA, SVRG.

5 Experiments and benchmarks

In this section we demonstrate the performance of ReHLine compared with the existing state-ofthe-art (SOTA) solvers on various machine learning tasks. The source code of ReHLine, along with Python/R interface, is readily accessible on our GitHub repository ( ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴ /a115♦❢/a116♠✐♥✴❘❡❍▲✐♥❡ ) and our project page ( ❤/a116/a116♣/a115✿✴✴/a114❡❤❧✐♥❡✳❣✐/a116❤✉❜✳✐♦✴ ). Our experiments involve five prominent machine learning tasks that draw from diverse sets of data. Specifically, we focus on four classification datasets and five regression datasets sourced from OpenML ( ❤/a116/a116♣/a115✿ ✴✴✇✇✇✳♦♣❡♥♠❧✳♦/a114❣✴ ), with various scales and dimensions. The experiment settings are summarized in Table 4. To achieve a fair comparison, we use a well-organized toolset and framework, the BENCHOPT library [29], to implement optimization benchmarks for all the SOTA solvers.

Due to representation limits, we only present the results using a variant of the ReHLine algorithm that supports variable shrinkage (refer to Algorithm 2). Results utilizing ReHLine with or without shrinkage are reported on our benchmark repository ( ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴/a115♦❢/a116♠✐♥✴ ❘❡❍▲✐♥❡✲❜❡♥❝❤♠❛/a114❦ ). Note that there is no substantial difference between these variations, and they do not influence the conclusions drawn from our experiments. Simultaneously, in the benchmark repository, we provide a more detailed 'time versus objective' plot for each task/dataset and each solver produced by BENCHOPT.

For the FairSVM task, ReHLine is compared with the original solver [48] based on DCCP and other generic solvers in CVXPY. In this case, both feasibility and optimality are examined, where the

optimality is measured by the objective function, and the feasibility is measured by the violation of constraints: max ( n -1 ∣ ∣ ∑ n i =1 z i β ⊺ x i ∣ ∣ -ρ, 10 -6 ) . Moreover, we examine the performance of ElasticQR by considering the model defined in (A.2) with λ 1 = λ 2 = 1 , RidgeHuber of (A.4) with λ 1 = 0 , λ 2 = 1 , and SVM of (A.1) and sSVM of (A.5) with C = 1 . Table 5 presents the running times in seconds of all solvers that converge based on a relative/absolute tolerance of 10 -5 .

The results in Table 5 indicate that the proposed ReHLine algorithm/software achieves visible improvements over existing solvers. The major empirical findings are as follows. First, the amount of improvement achieved by ReHLine in terms of running time is substantial over generic solvers, including ECOS, SCS, and MOSEK based on CVX, with the largest improvement 100x-1000x speed-up on the largest-scale dataset. The improvement continues to expand even further as the scale of the problem grows. Second, ReHLine also yields considerable improvements for specialized algorithms, including LIBLINEAR in SVM, HQREG in Huber minimization, and LIGHTNING in sSVM. Third, given that the generic solvers fail or time out in multiple tasks and datasets, ReHLine has shown remarkable flexibility when it comes to solving various domain-specific problems.

6 Conclusion

In this paper, we present a new algorithm ReHLine designed to efficiently solve general regularized ERM with the convex PLQ loss function and optional linear constraints. Through the transformation of a PLQ loss into a composite ReLU-ReHU function, we have developed a novel CD algorithm that updates primal and dual variables simultaneously. This approach has proven to be highly effective in reducing the computational complexity while at the same time achieving a fast linear convergence rate. Based on our experiments, ReHLine demonstrates remarkable flexibility and computational efficiency, outperforming existing generic and specialized methods in a range of problems.

Limitation. Although our approach has exhibited substantial versatility in addressing a diverse range of problems, it is important to acknowledge that it may not be universally applicable to all convex optimization problems. The main limit of our approach stems from the restriction imposed by the formulation (1). For instance, (1) requires a convex PLQ loss function and a strict l 2 -regularization. We recognize this limitation and intend to address it in future research pursuits. Moreover, we are interested in exploiting additional loss function properties, such as symmetry.

References

Alekseeva, E., Brotcorne, L., Lepaul, S., and Montmeat, A. (2018). A bilevel approach to optimize electricity prices. Yugoslav Journal of Operations Research , 29(1):9-30.

ApS, M. (2019). The MOSEK Optimization Toolbox for MATLAB Manual. Version 9.0.

Beck, A. and Teboulle, M. (2012). Smoothing and first order methods: A unified framework. SIAM Journal on Optimization , 22(2):557-580.

Blondel, M. and Pedregosa, F. (2016). Lightning: large-scale linear classification, regression and ranking in Python.

Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning , 3(1):1-122.

Bubeck, S. et al. (2015). Convex optimization: algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357.

Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine Learning , 20(3):273-297.

Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems , 27.

Deng, W., Lai, M.-J., Peng, Z., and Yin, W. (2017). Parallel multi-block ADMM with o(1/k) convergence. Journal of Scientific Computing , 71:712-736.

Diamond, S. and Boyd, S. (2016). CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(1):2909-2913.

Domahidi, A., Chu, E., and Boyd, S. (2013). Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC) , pages 3071-3076. IEEE.

Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., and Lin, C.-J. (2008). LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research , 9:1871-1874.

Ferris, M. C. and Munson, T. S. (2002). Interior-point methods for massive support vector machines. SIAM Journal on Optimization , 13(3):783-804.

Fine, S. and Scheinberg, K. (2001). Efficient SVM training using low-rank kernel representations. Journal of Machine Learning Research , 2(Dec):243-264.

Fukushima, K. (1969). Visual feature extraction by a multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics , 5(4):322-333.

Gérard, M., Clautiaux, F., and Sadykov, R. (2016). Column generation based approaches for a tour scheduling problem with a multi-skill heterogeneous workforce. European Journal of Operational Research , 252(3):1019-1030.

Grant, M. and Boyd, S. (2014). CVX: Matlab software for disciplined convex programming, version 2.1.

Griva, I., Shanno, D. F., and Vanderbei, R. J. (2004). Convergence analysis of a primal-dual interior-point method for nonlinear programming. Optimization Online .

Gu, Y., Fan, J., Kong, L., Ma, S., and Zou, H. (2018). ADMM for high-dimensional sparse penalized quantile regression. Technometrics , 60(3):319-331.

He, B. and Yuan, X. (2012). On the o(1/n) convergence rate of the Douglas-Rachford alternating direction method. SIAM Journal on Numerical Analysis , 50(2):700-709.

Hsieh, C.-J., Chang, K.-W., Lin, C.-J., Keerthi, S. S., and Sundararajan, S. (2008). A dual coordinate descent method for large-scale linear SVM. In International Conference on Machine Learning , pages 408-415.

Huber, P. J. (1964). Robust estimation of a location parameter. The Annals of Mathematical Statistics , 35(1):73-101.

Huber, P. J. (1965). A robust version of the probability ratio test. The Annals of Mathematical Statistics , pages 1753-1758.

Jagannathan, R. and Ma, T. (2003). Risk reduction in large portfolios: Why imposing the wrong constraints helps. Journal of Finance , 58(4):1651-1683.

Johnson, R. and Zhang, T. (2013). Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems , 26.

Koenker, R. and Bassett Jr, G. (1978). Regression quantiles. Econometrica , 46(1):33-50.

Luo, Z.-Q. and Tseng, P. (1992). On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications , 72(1):7-35.

Michel Valladão, D., Veiga, Á., and Street, A. (2018). A linear stochastic programming model for optimal leveraged portfolio selection. Computational Economics , 51(4):1021-1032.

Moreau, T., Massias, M., Gramfort, A., Ablin, P., Bannier, P.-A., Charlier, B., Dagréou, M., Dupre la Tour, T., Durif, G., Dantas, C. F., et al. (2022). Benchopt: Reproducible, efficient and collaborative optimization benchmarks. In Advances in Neural Information Processing Systems , pages 25404-25421.

Nesterov, Y. (2005). Smooth minimization of non-smooth functions. Mathematical Programming , 103:127-152.

Nesterov, Y. (2012). Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362.

O'donoghue, B., Chu, E., Parikh, N., and Boyd, S. (2016). Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications , 169(3):1042-1068.

Rennie, J. D. and Srebro, N. (2005). Loss functions for preference levels: Regression with discrete ordered labels. In Proceedings of the IJCAI Multidisciplinary Workshop on Advances in Preference Handling , volume 1. Citeseer.

Rockafellar, R. T. and Wets, R. J.-B. (2009). Variational analysis . Springer Science & Business Media.

Saha, A. and Tewari, A. (2013). On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization , 23(1):576-601.

Salas, V. and Saurina, J. (2002). Credit risk in two institutional regimes: Spanish commercial and savings banks. Journal of Financial Services Research , 22(3):203-224.

Schmidt, M., Le Roux, N., and Bach, F. (2017). Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112.

Schumaker, L. (2007). Spline Functions: Basic Theory . Cambridge university press.

Shalev-Shwartz, S. and Zhang, T. (2013). Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research , 14(1).

Shalev-Shwartz, S. and Zhang, T. (2014). Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. In International Conference on Machine Learning , pages 64-72. PMLR.

Shen, X., Diamond, S., Gu, Y., and Boyd, S. (2016). Disciplined convex-concave programming. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 1009-1014. IEEE.

Tseng, P. and Yun, S. (2010). A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Computational Optimization and Applications , 47(2):179-206.

Vapnik, V. (1991). Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems , pages 831-838.

Vapnik, V. (1999). The Nature of Statistical Learning Theory . Springer science & business media.

Wright, S. J. (2015). Coordinate descent algorithms. Mathematical Programming , 151(1):3-34.

Yi, C. and Huang, J. (2017). Semismooth newton coordinate descent algorithm for elastic-net penalized Huber loss regression and quantile regression. Journal of Computational and Graphical Statistics , 26(3):547-557.

Zafar, M. B., Valera, I., Gomez-Rodriguez, M., and Gummadi, K. P. (2019). Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research , 20(1):27372778.

Zafar, M. B., Valera, I., Rogriguez, M. G., and Gummadi, K. P. (2017). Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics , pages 962-970.

Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 67(2):301-320.

A Example Models

SVMs with fairness constraints (FairSVM). The formulation for FairSVM [48] is:

where x i ∈ R d is a feature vector, and y i ∈ {-1 , 1 } is a binary label, z i ∈ R d 0 is a collection of centered sensitive features ( ∑ n i =1 z i = 0 ) , such as gender and/or race. The constraints in (A.1) constrain the correlation between the sensitive features z i and the decision function β ⊺ x i , and the constants ρ ∈ R d 0 + trade off predictive accuracy and fairness. Note that the FairSVM (A.1) can be rewritten as a ReHLine optimization of (4) with

where 1 n = (1 , · · · , 1) ⊺ is the lengthn one vector, X ∈ R n × d is the feature matrix, and y = ( y 1 , · · · , y n ) ⊺ is the response vector.

Elastic net regularized quantile regression (ElasticQR). The formulation for the elastic net penalized quantile regression [19] is:

where

is the check loss, κ ∈ (0 , 1) is a prespecified sample quantile, x i ∈ R d is a feature vector, y i ∈ R is a response, and λ 1 , λ 2 ≥ 0 are weights of lasso and ridge penalties. Then, the ElasticQR (A.2) can be rewritten as a ReHLine optimization of (4) with

where I d +1 is the identity matrix.

Elastic net regularized Huber minimization. The formulation for the elastic net penalized Huber minimization [23] is:

where H κ ( · ) is the Huber loss with a given parameter κ :

In this case, (A.4) can be rewritten as a ReHLine optimization with

Smoothed SVM (sSVM). The formulation for the smoothed SVMs [33] is:

where x i ∈ R d is a feature vector, and y i ∈ {-1 , 1 } is a binary label, and V κ ( · ) is the modified Huber loss or the smoothed hinge loss:

In this case, (A.5) can be rewritten as a ReHLine optimization with

B Update Rules for (7)

B.1 Canonical CD updates

Update ξ k given others. For k = 1 , · · · , K ,

̸

̸

Update λ li given others. For l = 1 , · · · , L and i = 1 , · · · , n ,

̸

̸

Update γ hi given others. For h = 1 , · · · , H and i = 1 , · · · , n ,

̸

̸

The canonical CD updates in (B.6) - (B.8) for one full iteration require computation complexity of O ( ( K + nd + nL + nH )( K + nL + nH ) ) .

B.2 ReHLine updates

Consider the Lagrangian function L in (7). Clearly,

̸

Similarly, we can derive that

Then the CD updates in (B.6) can be simplified as:

The CD updates in (B.7) are simplified as:

The CD updates in (B.8) are simplified as:

The computational complexity of the new ReHLine iterations is O ( ( K + nL + nH ) d ) , as shown in (B.9) through (B.11). Notably, this complexity is linear with respect to both the sample size n and the dimensionality d , making ReHLine a highly efficient algorithm.

C Screening rules for ReHLine

We extend the shrinking strategy in Algorithm 3 of [12] to the ReHLine solver. To begin with, the following theorem advocates utilizing 'sparse' updates to optimize the performance of ReHLine.

Theorem 4. Let { ξ ∗ , Λ ∗ , Γ ∗ , β ∗ } be the convergent point of { ξ ( q ) , Λ ( q ) , Γ ( q ) , β ( q ) } based on (B.9) -(B.11) , then

Building upon the insights from Theorem 4, we have formulated Algorithm 2 that involves shrinking certain 'boundary' variables during the implementation of ReHLine.

Algorithm 2: ReHLine solver minimizing (1) with shrinking.

Algorithm 3: Projected gradients in ReHLine.

D Technical proofs

Lemma 1. Suppose f ( z ) is a nonnegative convex PLQ with knots and extreme points ( t k ) K k =1 . Then, there exists k 0 ∈ { 1 , · · · , K } , such that f ( z ) ≥ f ( t k 0 ) for any z ∈ R .

Proof. We prove it by contradiction. Suppose that for there exists z 0 such that f ( z 0 ) < f ( t k ) for all k ∈ { 1 , · · · , K } , then the minimum can only be obtained outside [ t 1 , t K ] , that is

For example, suppose that z 0 ∈ ( t K , ∞ ) , f ( z 0 ) < f ( t k ) for any k ∈ { 1 , · · · , K } . Note that the quadratic function in ( z K , ∞ ) can be written as f ( z ) = 1 2 f ′′ + ( t K )( z -t K ) 2 + f ′ + ( t K )( z -t K )+ f ( t K ) . Based on the fact that f ( z 0 ) < f ( t K ) and the convexity of f ( z ) , we have f ′ + ( t K ) < 0 . Then, since ( t k ) K k =1 is the set of all knots and extreme points, we have f ′ ( z ) < 0 for all z > t K , and lim z →∞ f ′ ( z ) = lim z →∞ ( f ′′ + ( t K )( z -t K ) + f ′ + ( t K )) ≤ 0 . This yields that f ′′ + ( t K ) = 0 , alternatively, f ( z ) is a linear function in ( t K , ∞ ) as f ( z ) = f ′ + ( t K )( z -t K ) + f ( t K ) . Hence, when z →∞ , f ( z ) →-∞ , which contradicts with the fact that f ( z ) is a nonnegative function. The same argument goes for the case when z 0 ∈ ( -∞ , t 1 ) . This completes the proof.

Proof of Proposition 1

Proof. Note that c ReLU( z ) = c max( z, 0) = max( cz, 0) , and

for c > 0 . Then,

This completes the proof.

Proof of Theorem 1

Proof. The sufficiency ( ⇐ = ) directly follows from the definitions of ReLU and ReHU . Next, the necessity ( = ⇒ ) of the theorem is established using induction. Without loss of generality, for a nonnegative convex PLQ function f ( z ) , we assume that (A1) f ′ + ( t k ) f ′ -( t k +1 ) ≥ 0 , otherwise we can add the extreme point between ( t k , t k +1 ] into the set of knots; (A2) min k =1 , ··· ,K f ( t k ) = 0 , otherwise we can add ReLU ( min k =1 , ··· ,K f ( t k ) ) to the final decomposition (Lemma 1 shows that the minimum of f ( z ) can only be obtained within the knots). Next, we conduct the mathematical induction.

K = 1 . Note that f ( z ) can be rewritten as:

where q -1 ( z ) and q + 1 ( z ) are left and right parts of f ( z ) separated at z = t 1 . According to Lemma 1 and Assumption A2, we have f ( t 1 ) = 0 is the minimum of f ( z ) . Combined with Assumption A1, we have f ′ -( t ) ≤ 0 for t < t 1 , and f ′ + ( t ) ≥ 0 for t > t 1 . Then, f ( z ) can be decomposed as:

Hence, f ( z ) = ReHU ∞ ( -√ f ′′ -( t 1 )( z -t 1 )) + ReLU( f ′ -( t 1 )( z -t 1 )) + ReHU ∞ ( √ f ′′ + ( t 1 )( z -t 1 )) + ReLU( f ′ + ( t 1 )( z -t 1 )) . The necessity of the theorem for K = 1 then follows.

K = K 0 +1 . Assume that a convex PLQ f ≥ 0 with K ( K ≤ K 0 ) knots is ReLU-ReHU composite, now we turn to prove the statement still holds for f ( z ) with K 0 +1 .

CASE 1. Suppose the minimum f ( t ∗ ) = 0 is obtained at { t 2 , · · · t K 0 } (not in the boundary). In this case, we can decompose f ( z ) as f ( z ) = f l ( z ) + f r ( z ) where f l ( z ) and f r ( z ) are defined as:

Both f l ( z ) and f r ( z ) are nonnegative convex PLQs with K ≤ K 0 knots, thus can be represented as ReHUs and ReLUs.

CASE 2. Suppose the minimum f ( t ∗ ) = 0 is obtained when t ∗ = t 1 or t ∗ = t K 0 +1 . For example, when t ∗ = t 1 , we conduct the same decomposition f ( z ) = f l ( z ) + f r ( z ) at z = t 1 , where f l ( z ) is a nonnegative convex PLQ with one knot, which is ReHU-ReLU composite. For f r ( z ) , we further decompose it as:

where τ = √ f ′′ + ( t 1 )( t 2 -t 1 ) , and R ( z ) (with at most K 0 knots) is defined as:

Now, we show that R ( z ) is nonnegative convex. Note that f ( z ) is convex, then for z ≥ t 2 ,

where the first equality follows from the fact that f ( z ) is a quadratic function in [ t 1 , t 2 ] , and thus f ′ -( t 2 ) = f ′ + ( t 1 )( t 2 -t 1 ) , and the second equality follows from f ( t 2 ) = f ′′ -( t 1 ) 2 ( t 2 -t 1 ) 2 + f ′ -( t 1 )( t 2 -t 1 ) .

Thus, R ( z ) is a nonnegative convex PLQ with K 0 knots { t 2 , · · · , t K 0 +1 } , which can be decomposed as ReHUs and ReLUs. The same argument goes for the case when t ∗ = t K 0 +1 .

This completes the proof, and the desirable results follow from the induction.

Proof of Proposition 2

Proof. The lemma follows from the fact that:

where e j = (0 , · · · , 0 , 1 , 0 · · · ) is the d -tuple with all components equal to 0, except the j th.

Proof of Theorem 2

Proof. Note that the augmented Lagrangian of the primal problem (6) is:

where ξ = ( ξ k ) ∈ R K , Λ = ( λ li ) ∈ R L × n , Γ = ( γ hi ) ∈ R H × n , ∆ = ( δ li ) ∈ R L × n and Ψ = ( ψ hi ) ∈ R H × n are dual variables. The KKT conditions for the augmented Lagrangian are:

for all ( i, l, h ) . Then, by substituting the KKT conditions into the primal Lagrangian, the (negative) dual objective function is derived as:

where ¯ U (3) and ¯ S (3) are the mode-3 unfolding of the tensors ¯ U ∈ R L × n × d and ¯ S ∈ R H × n × d , and I is an identity matrix.

Proof of Theorem 3

Proof. The treatment for the proof is based on a result of coordinate descent on boxed constrained convex optimization [27]. To proceed, denote µ = ( ξ ⊺ , vec ( Λ ) ⊺ , vec ( Γ ) ⊺ ) ⊺ as the collection of all dual variables, then the dual objective can be rewritten as:

where g ( z ) = 1 2 ∥ z ∥ 2 2 , and E and c are defined as:

Now, we tend to verify assumptions of Theorem 2.1 in [27]. Specifically, (a) the dual problem is a convex box-constrained QP, thus the set of optimal solutions is nonempty; (b) g ( z ) = 1 2 ∥ z ∥ 2 2 is a strictly convex twice continuously differentiable function; (c) ∇ 2 g ( z ) = I d + Hn is always positive definite; (d) E has no zero column based on its definition. The desirable result then follows from Theorem 2.1 of [27].

Proof of Theorem 4

Proof. Note that the primal problem (6) is strictly convex, thus it has the unique minimizer β ∗ . Therefore, the sequence { β ( q ) } ∞ q =1 provided by the proposed Algorithm 1 converges to the unique minimizer β ∗ . Next, we check the gradients of each coordinate.

For ξ -coordinates, with ξ k ≥ 0 ,

̸

Suppose g ( q ) k is the gradient of ξ k based on µ ( q ) , then lim q →∞ g ( q ) k = a ⊺ k β ∗ + b k =: g ∗ k . Therefore, if g ∗ k > 0 , then there exists q 0 , for all q ≥ q 0 , we have g ( q ) k > 0 , which implies that ξ ( q +1) k = 0 .

In the same manner, for Λ -coordinates, with 1 ≥ λ li ≥ 0 ,

Then, lim q →∞ g ( q ) li = -u li x ⊺ i β ∗ -v li =: g ∗ li ,

For Γ -coordinates, with τ hi ≥ γ hi ≥ 0 , thus

Then, lim q →∞ g ( q ) hi = γ ∗ hi -u li x ⊺ i β ∗ -v li =: g ∗ hi , which implies that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dual-coordinate descent solver for a class of ERM (Empricial Risk Minimization) problem with general linear inequality constraint, which is of linear convergence rate and efficient coordinate-update cost of O(n) (n is the number of samples). The main contribution is it extends existing method of dual-coordinate descent to handle problem with additional linear inequality constraints.\n\nSTRENGTHS:\nThe paper has solid theoretical motivation, derivation and analysis, and has presented the idea clearly with solid experiments support on pratical optimization problems.\n\nWEAKNESSES:\nThe paper completely missed discussion on existing works of dual-coordinate ascent on general ERM problem. There is a thread of works such as:\n\nShalev-Shwartz, Shai, and Tong Zhang. \"Stochastic dual coordinate ascent methods for regularized loss minimization.\" Journal of Machine Learning Research 14.1 (2013).\n\non dual-coordinate ascent method for general ERM. The novelty of problem setup proposed in this reviewed paper is it adds an additional linear inequality constraints. The \"piecewise-quadratic loss\" is simply a special case of \"smooth loss\" of the above mentioned paper and SAME convergence rate applies to all this type of problem, so the author chould have made this work more general (by extending the formulation of the above work by simply adding linear inequliaty).\n\nQUESTIONS:\nTo my knowledge, Liblinear is a special case of solver proposed in this paper, speicificaly designed for the SVM optimization problem.\nThen why could solver proposed in this paper perform more efficiently than Liblinear on SVM problem? Is it implementation issue? or their objective functions are different? The author should have made this clear to avoid confusion.\n\nLIMITATIONS:\nThe paper does not need to limit itself to the case of Relu-Rehu function.\nInstead, any smooth loss (in the primal) yields strongly-convex dual problem. Therefore can apply same optimization method convergence rate proposed in the paper. (ref: https://www.jmlr.org/papers/volume14/shalev-shwartz13a/shalev-shwartz13a.pdf)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new optimization algorithm for convex piecewise linear-quadratic objectives with $L_2$ regularization and linear constraints, which achieves the best known iteration complexity simultaneously with the best known per-iteration computational cost, resulting in a smaller total computational complexity than any previous algorithm. Experiments show that the proposed algorithm achieves ~1000x speedup compared to previous solvers and improves over problem-specific solvers despite its generality.\n\nSTRENGTHS:\n1. The considered optimization problem is broad, encompassing training for many linear models (SVM, LAD, SVR, QR) and constraint sets, including fairness.\n2. The technique appears to be novel.\n3. The comparison of total complexity (Table 2) demonstrates a significant theoretical advantage over previous algorithms.\n4. The empirical comparison (Table 5) shows that the proposed algorithm enjoys a significant practical advantage over previous algorithms.\n\nWEAKNESSES:\n1. The experimental evaluation is missing a few details that need clarifying. For example, line 212 states that the BenchOpt framework is used \"to implement optimization benchmarks for all the SOTA solvers\". I find this statement a little unclear. Did the authors implement all of the baseline solvers from scratch? If not, what do they mean by \"implement optimization benchmarks\"? Also, is it common to implement these solvers in Python? My understanding is that the production solvers are most commonly implemented in C/C++, and using a slower language like Python might amplify the difference in speed between algorithms.\n\nQUESTIONS:\n1. Line 123 states that the proposed algorithm is inspired by LibLinear, but there is no further discussion of the relationship between ReHLine and LibLinear. What components of ReHLine are inspired by LibLinear, and what components of ReHLine are novel compared to LibLinear?\n2. The introduction mentions that this work focuses on the case that the dataset size is much larger than the input dimension and the number of constraints, but this large-scale condition is not discussed further. Why is this large-scale condition significant to your approach, and how does ReHLine compare to baseline algorithms when $n$ is small?\n3. Please clarify the questions I posed about the details of experimental evaluation in the Weaknesses section.\n\nLIMITATIONS:\nThe authors provided a reasonable discussion of the limitations of their approach. A discussion of potential negative societal impact is, in my opinion, not necessary for this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the Empirical Risk Minimization(ERM), which is an important framework of machine learning, and focus on a general regularized ERM based on a convex PLQ loss with linear constraints. According to the authors, the existing algorithms are faced with the problem of slow convergence or high computational cost. The authors leverage the linear property of Karush-Kuhn-Tucker conditions and propose ReHLine, whose total complexity is lower than existing algorithms.


## Comments
It is recommended to indicate the unit of running time on Table 5.

STRENGTHS:
1. The authors cleverly decompose convex PLQ into a series of ReLU and the so-called rectified Huber units to deal with the problem that the standard definition of convex PLQ is not convenient to optimization algorithms.When solving the decomposed optimization problem, the authors prove that their ReHLine algorithm can achieve linear convergence rate.
2. After comparing with other state-of-the-art algorithms on multiple datasets, the experimental results show that the ReHLine algorithm can solve a variety of domain-specific problems, presenting excellent flexibility, and on large-scale datasets, the algorithm shows excellent acceleration ratio.

WEAKNESSES:
Actually, rather than those solvers tested in the experiment part, there are a number of general optimization solvers, including CPLEX, GUROBI, SCIP, etc. Those optimization solvers have exhibited great success in solving linear/nonlinear continuous/discrete optimization problems. Have you compared your proposed method against those general optimization solvers?

QUESTIONS:
Please see my question in the "Weakness" section.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

comments: SUMMARY:
This paper introduces an algorithm, ReHLine, which has a linear convergence rate on minimizing convex piecewise linear-quadratic loss functions. Experiments on several tasks show great performance gains over existing algorithms.

STRENGTHS:
General: The paper is clean well-written and easy to follow. Definitions are Proposition are well explained with examples.


Pros:

* Introduces ReLU-ReHU decomposition and show commonly-used loss functions have their ReLU-ReHU representation.
* Finds the dual formulation and shows that proposed algorithm has linear convergence rate by Coordinates Descent.
* Experimental results greatly outperform existing algorithms.

Cons:

* The main contribution of this paper is proposal of ReLU-ReHU decomposition and its resulting formulation. Its convergence result is shown by classical result. I feel the contribution is limited and marginally above the borderline.


In all, this paper is well written and the framework ReLU-ReHU is promising. Although the contribution seems limited but I am happy to vote for accept.

----------

Main body checked but did not throughly go through appendix.

WEAKNESSES:
See above.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider tackling the optimization fo finite sum of piecewise linear quadratic functions (PLQ) arising from e.g. robust empirical risk minimization by means of a reformulation of PLQ functions as sums of ReLU and smoothed ReLU functions combined with a stochastic dual ascent algorithm. The authors start by presenting how any PLQ function can be formulated as sums of of ReLU and smoothed ReLU functions with a table summarizing some common ones. Then the proposed algorithm is presented whose convergence is ensured by previous work. Experimental results advocate for the superiority of the approach compared to old all-purpose solvers in various classification tasks.\n\nSTRENGTHS:\n- The generality of the approach is well defended by the flexibility of the parameterization with ReLU and smoothed ReLU functions.\n- The proposed algorithm captures efficiently the underlying structure of the algorithm. In particular, compared to primal-based approaches (which could easily be implemented using the proximal operators of the losses), the approach can handle constraints. \n- The proposed algorithm is hyper-parameter free. \n- The experiments could be a strength, although they are simply misleading in their current state.\n\nWEAKNESSES:\n- The paper lacks some important related work. In particular, the proposed algorithm appears to simply be a form of stochastic coordinate dual ascent. Such a note would help gain perspective on the contributions of the authors.\n- A \n- Numerous algorithms have been developed to tackle empirical risk minimization problems. For example, s-SVM or SVM^2 losses are smooth and could be tackled by fast incremental solvers such as SAGA, SVRG or SDCA or even their accelerated versions such as [1]. see also [2]. Non-smooth losses can also generally easily be smoothed, see e.g. [3], to be amenable theoretically and in practice to fast resolution by fast incremental algorithms. True, most of these algorithms require stepsizes, which make the proposed approach friendlier at first. However, SDCA for example does not require any and could a priori be formulated like the proposed algorithm (using that the proximal operator of the conjugate of the losses is available here). A thorough discussion and comparison to all of the aforementioned algorithms is necessary to held this paper to the claims the authors make. Additional \n- Non-negativity, box and monotonicity constraints can be handled by simple projections and primal algorithms could also be considered in those cases.\n- The current approach cannot handle non-strongly convex penalty such as sparsity inducing penalties. \n- The experimental results and claims are misleading: no one uses all-purpose solvers such as MOSEK. In particular no one uses interior point methods for empirical risk minimization. Claiming 1000 times improvement is overselling the method which in turns is detrimental to the contributions of the paper. Usual comparisons consist in claiming improvements over state of the art method, which appear to hold but at much smaller scale. The proposed algorithm may provide quantitative benefits, but a comprehensive comparison would help the paper.\n\nMinor details:\n- E appears not have been defined in Theorem 2. Similarly recalling what a mode-3 unfolding of a tensor is, would improve the readability of the paper. \n\n[1] A. Defazio. A Simple Practical Accelerated Method for Finite Sums. NIPS 2016\n[2] S Shalev-Shwartz, T Zhang. Accelerated Mini-Batch Stochastic Dual Coordinate Ascent. NIPS 2013\n[3] A Beck. Smoothing and First Order Methods: A Unified Framework. SIOPT 2012\n\nQUESTIONS:\n- Can the authors do comparisons with state of the art algorithms for empirical risk minimization for each of the proposed problems rather than using all-purpose algorithms? A list of potential candidates has been given in the weakness section. \n- In particular, can the authors discuss and compare empirically their algorithm against SDCA in the absence of constraints?\n- How can the proposed algorithms handle nonlinear models such as kernel methods?\n\nLIMITATIONS:\n- No support for sparsity inducing norms", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a new function class called composite ReLU-ReHU which is shown to be equivalent to the class of convex PLQ functions. Based on the ReLU-ReHU decomposition of convex PLQ functions, the authors then formulated a new box-constrained quadratic programming optimization problem, named ReHLine optimization which can be solved efficiently by the ReHLine algorithm. The proposed solver has a provable linear convergence rate and a linear per-iteration computation complexity. Benchmarking on various tasks and datasets, the ReHLine showed significant improvement over both generic and specific solvers, especially on large-scale datasets.\n\nSTRENGTHS:\n- This paper is well-written and easy to follow.\n- To the best of my knowledge, the composite ReLU-ReHU functions and the ReHLine algorithm are novel.\n- The effectiveness of the proposed method is adequately backed by both theoretical results and empirical results.\n- The ReHLine algorithm successfully tackles any convex PLQ loss functions, thus it shows great potential for many machine learning tasks.\n\nWEAKNESSES:\n- While authors claimed that the simplification of Canonical CD updates is highly non-trivial, it seems to be a straight application of [18] to the ReHLine optimization problem.\n\nQUESTIONS:\n1. Are the experimental results using Algorithm 1 or Algorithm 2 in the Appendix? Could the authors provide the running time of both algorithms for comparison?\n2. Could the authors provide an ablation study to demonstrate the objective gaps of different solvers when increasing the number of iterations?\n\nMinor:\n- There is a typo in Proposition 2: $T \\leftarrow \\begin{pmatrix} \\sqrt{\\frac{2}{\\lambda_2}} \\boldsymbol{T} & \\boldsymbol{0}_d^T \\end{pmatrix}$.\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}