SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{P}_{[a,b]}(x) = \max (a,\min (b,x))$  means projecting a real number  $x$  to the interval  $[a,b]$ .
Clearly, assuming the values  $\mathbf{x}_i^\top \mathbf{a}_k$  and  $\| \mathbf{x}_i\| _2^2$  are cached, updating one  $\lambda_{li}$  value requires  $\mathcal{O}(K + nd + nL + nH)$  of computation, and updating the whole  $\Lambda$  matrix requires  $\mathcal{O}\big(nL(K + nd + nL + nH)\big)$ . Adding all variables together, the canonical CD update rule for one full cycle has a computational complexity of  $\mathcal{O}\big((K + nd + nL + nH)(K + nL + nH)\big)$ .
ReHLine updates. The proposed ReHLine algorithm, on the other hand, significantly reduces the computational complexity of canonical CD by updating  $\beta$  according to the KKT condition (9) after each update of a dual variable. To see this, let  $\pmb{\mu} \coloneqq (\pmb{\xi}, \pmb{\Lambda}, \pmb{\Gamma})$  denote all the dual variables, and define
$$
\boldsymbol {\beta} (\boldsymbol {\mu}) = \sum_ {k = 1} ^ {K} \xi_ {k} \mathbf {a} _ {k} - \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \left(\sum_ {l = 1} ^ {L} \lambda_ {l i} u _ {l i} + \sum_ {h = 1} ^ {H} \gamma_ {h i} s _ {h i}\right).
$$
Then it can be proved that  $(\nabla_{\lambda}\mathcal{L}_{li})(\lambda_{li}) = -(u_{li}\mathbf{x}_i^\top \boldsymbol {\beta}(\boldsymbol {\mu}) + v_{li})$ . Therefore, when  $\pmb{\mu}$  is fixed at  $\pmb{\mu}^{\mathrm{old}} = (\pmb{\xi}^{\mathrm{old}},\pmb{\Lambda}^{\mathrm{old}},\pmb{\Gamma}^{\mathrm{old}})$  and let  $\beta^{\mathrm{old}} = \beta (\pmb{\mu}^{\mathrm{old}})$ , (10) can be rewritten as
$$
\lambda_ {l i} ^ {\mathrm {n e w}} = \mathcal {P} _ {[ 0, 1 ]} \left(\lambda_ {l i} ^ {\mathrm {o l d}} - \frac {(\nabla_ {\lambda_ {l i}} \mathcal {L}) (\lambda^ {\mathrm {o l d}})}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right) = \mathcal {P} _ {[ 0, 1 ]} \left(\lambda_ {l i} ^ {\mathrm {o l d}} + \frac {u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} ^ {\mathrm {o l d}} + v _ {l i}}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right).
$$
Accordingly, the primal variable  $\beta$  is updated as
$$
\pmb {\beta} ^ {\mathrm {n e w}} = \pmb {\beta} ^ {\mathrm {o l d}} - (\lambda_ {l i} ^ {\mathrm {n e w}} - \lambda_ {l i} ^ {\mathrm {o l d}}) u _ {l i} \mathbf {x} _ {i},
$$
which can then be used for the next dual variable update. Simple calculations show that this scheme only costs  $\mathcal{O}(d)$  of computation for one  $\lambda_{li}$  variable.
The update rules for other dual variables are similar, with the complete algorithm shown in Algorithm 1 and the full derivation details given in Appendix B. Overall, the ReHLine update rule has a computational complexity of  $\mathcal{O}\big((K + nL + nH)d\big)$  for one full cycle. Given that  $(K,L,H)$  are all small numbers, the complexity is linear with respect to both the sample size  $n$  and the dimension  $d$ .
We emphasize that the linear relationship (9) between the primal and dual variables, as well as its consequence of reducing the computational cost, is highly non-trivial. This technique has been proposed for specialized problems such as SVMs via the LIBLINEAR solver [21], and here we show that its success can be greatly generalized to any convex PLQ loss functions.
Algorithm 1: The ReHLine algorithm that solves (4).
Input  $\mathbf{:X}\in \mathbb{R}^{n\times d},\mathbf{U},\mathbf{V}\in \mathbb{R}^{L\times n},\mathbf{S},\mathbf{T},\pmb {\tau}\in \mathbb{R}^{H\times n},\mathbf{A}\in \mathbb{R}^{K\times d},\mathbf{b}\in \mathbb{R}^{K}$  1 Initialize  $\pmb {\xi}\geq \mathbf{0},\mathbf{E}\geq \pmb {\Lambda}\geq \mathbf{0},\pmb {\tau}\geq \pmb{\Gamma}\geq \mathbf{0};$    
2 Compute  $\beta = \mathbf{A}^{\intercal}\pmb {\xi} - \sum_{i = 1}^{n}\mathbf{x}_{i}\bigl (\sum_{l = 1}^{L}\lambda_{li}u_{li} + \sum_{h}^{H}\gamma_{hi}s_{hi}\bigr);$    
3 Compute and store  $\mathbf{r} = (r_i)\in \mathbb{R}^n,r_i = \| \mathbf{x}_i\| _2^2$  , and  $\mathbf{p} = (p_k)\in \mathbb{R}^K,p_k = \| \mathbf{a}_k\| _2^2;$    
4 while maximum iteration do   
5 for  $k\in \mathcal{K} = [K]$  do   
6  $\begin{array}{rl} & {\varepsilon^{*}\gets \max \left(-\xi_{k}, - p_{k}^{-1}(\mathbf{a}_{k}^{\intercal}\boldsymbol {\beta} + b_{k})\right);\quad \xi_{k}\gets \xi_{k} + \varepsilon^{*};\quad \beta \gets \beta +\varepsilon^{*}\mathbf{a}_{k};}\\ & {\mathrm{for~}(l,i)\in \mathcal{L} = [L]\otimes [n]~do}\\ & {\varepsilon^{*}\gets \max \left(-\lambda_{li},\min \left(1 - \lambda_{li},u_{li}^{-2}r_{i}^{-1}(v_{li} + u_{li}\mathbf{x}_{i}^{\intercal}\boldsymbol {\beta})\right)\right);}\\ & {\lambda_{li}\gets \lambda_{li} + \varepsilon^{*};\quad \beta \gets \beta -\varepsilon^{*}u_{li}\mathbf{x}_{i};}\\ & {\mathrm{for~}(h,i)\in \mathcal{H} = [H]\otimes [n]~do}\\ & {\varepsilon^{*}\gets \max \left(-\gamma_{hi},\min \left(\tau_{hi} - \gamma_{hi},(s_{hi}^{2}r_{i} + 1)^{-1}(t_{hi} + s_{hi}\mathbf{x}_{i}^{\intercal}\boldsymbol {\beta} - \gamma_{hi})\right)\right);}\\ & {\gamma_{hi}\gets \gamma_{hi} + \varepsilon^{*};\quad \beta \gets \beta -\varepsilon^{*}s_{hi}\mathbf{x}_{i};} \end{array}$    
9 Output:  $\xi ,\Lambda ,\Gamma ,\beta$
3.3 Global convergence rate
Finally, we show that the ReHLine algorithm has a fast global linear convergence rate.
Theorem 3. Let  $\pmb{\mu}^{(q)}\coloneqq (\pmb{\xi}^{(q)},\pmb{\Lambda}^{(q)},\pmb{\Gamma}^{(q)})$  be a sequence of iterates generated by Algorithm 1, and  $\pmb{\mu}^*$  be a minimizer of the dual objective (8). Then  $\pmb{\mu}^{(q)}$  is feasible, and the dual objective value converges at least linearly to that of  $\pmb{\mu}^*$ , that is, there exist  $0 < \eta < 1$  and  $q_{0}$  such that for all  $q\geq q_0$
$$
\mathcal {L} \left(\boldsymbol {\mu} ^ {(q + 1)}\right) - \mathcal {L} \left(\boldsymbol {\mu} ^ {*}\right) \leq \eta \left(\mathcal {L} \left(\boldsymbol {\mu} ^ {(q)}\right) - \mathcal {L} \left(\boldsymbol {\mu} ^ {*}\right)\right).
$$
Table 1 summarizes the existing methods in solving the primal or dual problem of (1). Note that Column #ITERATIONS is the best-fit theoretical results of convergence analysis for each algorithm based on our literature search, which may be improved with further developments of convergence analysis. These algorithms can be broadly divided into the following categories.
Generic QP solvers. For instance, a projected sub-gradient descent is readily applicable to the primal form (1) by computing the sub-gradient of each loss and the projection under affine constraints, although the projection operator itself is already difficult. As for the box-QP dual problem (7), CD converges linearly to the optimal solution [27, 31, 35, 42]. Box-QP can also be efficiently solved using IPM, which leverages second-order cone programming techniques [13, 14]. Moreover, ADMM was employed to iteratively minimize the augmented Lagrangian of (1).
ERM solvers. There are multiple existing works developed to tackle ERM problems, such as SAG [37], SVRG [25], SAGA [8], proximal SDCA [40] and SDCA [39]. However, SAG, SVRG, SAGA and proximal SDCA can only handle smooth loss functions, with an optional non-smooth term that has an easy proximal operator. SDCA applies to general loss functions, but it requires the convex conjugate of loss functions, which is not necessarily simple to compute, also see discussion in Section 3.1 in [40]. Moreover, it only guarantees a sublinear convergence rate for non-smooth loss functions. In contrast, ReHLine supports all convex PLQ loss functions with optional general linear constraints, which could be non-smooth by construction, and they all enjoy a linear computational complexity and provable linear convergence.
For non-smooth loss functions, another existing method to solve ERM is the smoothing technique [40, 3, 30], which approximates the non-smooth terms by smooth functions, and then uses smooth-based algorithms to solve the smooth problem. However, the choice of the smoothing function and smoothing parameter typically requires additional knowledge, and the linear convergence rate is not always guaranteed.
Despite the successful use of the existing algorithms in solving (1), these methods either suffer from slow convergence (P-GD), or high computational costs (CD, IPM, and ADMM). This is partly because these general methods do not fully exploit the inherent structure of the problem. In contrast, ReHLine leverages the KKT condition in (9) to substantially improve numerical performance, thus achieving a linear computational complexity per iteration with respect to the sample size, as well as a linear convergence rate.
In addition to generic algorithms, specialized algorithms have been developed for specific types of (1). For example, the dual coordinate descent of LIBLINEAR in solving SVMs [21], the semismooth Newton coordinate descent algorithm of HQREG in solving the ridge regularized Huber minimization [46], and SAGA, SAG, proximal SDCA, and SVRG of LIGHTNING in solving smooth ERMs [4], such as smoothed SVMs. Our algorithm is also compared to these specialized methods in Section 5.
Software. Below are some commonly-used software for solving (1) based on the algorithms available.
- CVX/CVXPY [17, 10]: a modeling language for convex optimization problems and supports a wide range of solvers such as ECOS, SCS, OSQP, and MOSEK.
- MOSEK [2]: a commercial convex optimization solver based on IPM.
- ECOS [11]: an open-source solver of second-order cone programming based on IPM.
- SCS [32]: an open-source convex quadratic cone solver based on ADMM.
- DCCP [41]: a disciplined convex-concave programming solver built on top of CVXPY, which has been used to solve classification problems with fairness constraints [47].
- LIBLINEAR [21]: a library for large linear classification.
- HQREG [46]: an R package solving Lasso or elastic net penalized Huber loss regression and quantile regression.
- LIGHTNING [4]: a Python library solving various ERMs based on primal SDCA, SGD, AdaGrad, SAG, SAGA, SVRG.
5 Experiments and benchmarks
In this section we demonstrate the performance of ReHLine compared with the existing state-of-the-art (SOTA) solvers on various machine learning tasks. The source code of ReHLine, along with Python/R interface, is readily accessible on our GitHub repository (https://github.com/softmin/ReHLine) and our project page (https://rehline.github.io/). Our experiments involve five prominent machine learning tasks that draw from diverse sets of data. Specifically, we focus on four classification datasets and five regression datasets sourced from OpenML (https://www.openml.org/), with various scales and dimensions. The experiment settings are summarized in Table 4. To achieve a fair comparison, we use a well-organized toolset and framework, the BENCHOPT library [29], to implement optimization benchmarks for all the SOTA solvers.
Due to representation limits, we only present the results using a variant of the ReHLine algorithm that supports variable shrinkage (refer to Algorithm 2). Results utilizing ReHLine with or without shrinkage are reported on our benchmark repository (https://github.com/softmin/ReHLine-benchmark). Note that there is no substantial difference between these variations, and they do not influence the conclusions drawn from our experiments. Simultaneously, in the benchmark repository, we provide a more detailed "time versus objective" plot for each task/dataset and each solver produced by BENCHOPT.
Table 4: The classification and regression datasets with different scales used in our experiments.
CLASSIFICATION DATASET (n × d)	REGRESSION DATASET (n × d)
Steel-plates-fault (SPF): 1941 × 34	Liver-disorders (LD): 345 × 6
Philippine: 5832 × 309	Kin8nm: : 8191 × 9
Sylva-prior: 14395 × 109	Topo-2-1: 8885 × 267
Creditcard: 284807 × 31	House-8L: 22784 × 9
Buzzin-Twitter (BT): 583250 × 78
For the FairSVM task, ReHLine is compared with the original solver [48] based on DCCP and other generic solvers in CVXPY. In this case, both feasibility and optimality are examined, where the
optimality is measured by the objective function, and the feasibility is measured by the violation of constraints:  $\max \left(n^{-1}\Big|\sum_{i = 1}^{n}z_{i}\beta^{\mathsf{T}}\mathbf{x}_{i}\Big| - \rho ,10^{-6}\right)$ . Moreover, we examine the performance of ElasticQR by considering the model defined in (A.2) with  $\lambda_1 = \lambda_2 = 1$ , RidgeHuber of (A.4) with  $\lambda_1 = 0,\lambda_2 = 1$ , and SVM of (A.1) and sSVM of (A.5) with  $C = 1$ . Table 5 presents the running times in seconds of all solvers that converge based on a relative/absolute tolerance of  $10^{-5}$ .
Table 5: The averaged running times (± standard deviation) of SOTA solvers on machine learning tasks. “X” indicates cases where the solver produced an invalid solution or exceeded the allotted time limit. Speed-up refers to the speed-up in the averaged running time (on the largest dataset) achieved by ReHLine, where “∞” indicates that the solver fails to solve the problem.
TASK	DATASET	ECOS	MOSEK	SCS	DCCP	REHLINE
FairSVM	SPF (×1e-4)	X	X	X	X	4.25 (±0.5)
Philippine (×1e-2)	1550 (±0.6)	87.4 (±0.2)	130 (±42)	1137 (±9.2)	1.03 (±0.2)
Sylva-prior (×1e-2)	X	X	X	X	0.47 (±0.1)
Creditcard (×1e-1)	175 (±0.2)	64.2 (±0.1)	161 (±405)	X	0.64 (±0.2)
Fail/Succeed	2/2	2/2	2/2	3/1	0/4
Speed-up (on Creditcard)	273x	100x	252x	∞	-
TASK	DATASET	ECOS	MOSEK	SCS	REHLINE
ElasticQR	LD (×1e-4)	X	106 (±7)	34.9 (±25.0)	2.60 (±0.30)
Kin8nm (×1e-3)	X	92.0 (±1.0)	63.1 (±58.5)	4.12 (±0.95)
House-8L (×1e-3)	887 (±161)	277 (±34)	X	7.21 (±1.99)
Topo-2-1 (×1e-2)	4752 (±2015)	X	X	3.04 (±0.49)
BT (×1e-0)	7079 (±2517)	X	X	2.49 (±0.56)
Fail/Succeed	3/2	2/3	3/2	0/5
Speed-up (on BT)	2843x	∞	∞	-
TASK	DATASET	ECOS	MOSEK	SCS	HQREG	REHLINE
RidgeHuber	LD (×1e-4)	X	X	X	4.90 (±0.00)	1.40 (±0.20)
Kin8nm (×1e-3)	X	X	X	1.58 (±0.21)	2.04 (±0.30)
House-8L (×1e-3)	X	925 (±2)	X	2.42 (±0.34)	0.80 (±0.21)
Topo-2-1 (×1e-2)	2620 (±1040)	267 (±1)	213 (±2)	3.53 (±0.67)	1.78 (±0.32)
BT (×1e-1)	X	2384 (±433)	X	12.5 (±1.8)	5.28 (±1.31)
Fail/Succeed	4/1	2/3	4/1	0/5	0/5
Speed-up (on BT)	∞	452x	∞	2.37x	-
TASK	DATABASET	ECOS	MOSEK	SCS	LIBLINEAR	REHLINE
SVM	SPF (×1e-4)	X	372 (±1)	237 (±27)	12.7 (±0.1)	3.90 (±0.10)
Philippine (×1e-2)	1653 (±41)	86.5 (±0.2)	153 (±146)	1.80 (±0.02)	0.82 (±0.02)
Sylva-prior (×1e-3)	X	731 (±2)	843 (±1006)	16.0 (±0.6)	4.08 (±0.84)
Creditcard (×1e-2)	2111 (±804)	X	1731 (±4510)	23.1 (±2.5)	5.08 (±1.45)
Fail/Succeed	2/2	1/3	0/4	0/4	0/4
Speed-up (on Creditcard)	415x	∞	340x	4.5x	-
TASK	DATASET	SAGA	SAG	SDCA	SVRG	REHLINE
sSVM	SPF (×1e-4)	39.9 (±4.6)	28.3 (±5.0)	15.0 (±2.4)	41.4 (±3.9)	4.80 (±1.20)
Philippine (×1e-2)	24.3 (±27.8)	5.53 (±9.8)	1.47 (±0.19)	15.8 (±6.8)	0.89 (±0.10)
Sylva-prior (×1e-2)	3.37 (±9.81)	3.00 (±0.56)	1.57 (±0.23)	3.40 (±0.84)	0.86 (±0.14)
Creditcard (×1e-2)	10.4 (±1.4)	15.0 (±2.0)	14.0 (±1.9)	11.2 (±1.4)	6.36 (±1.92)
Fail/Succeed	0/4	0/4	0/4	0/4	0/4
Speed-up (on Creditcard)	1.6x	2.3x	2.2x	1.7x	-
The results in Table 5 indicate that the proposed ReHLine algorithm/software achieves visible improvements over existing solvers. The major empirical findings are as follows. First, the amount of improvement achieved by ReHLine in terms of running time is substantial over generic solvers, including ECOS, SCS, and MOSEK based on CVX, with the largest improvement  $100\mathrm{x}-1000\mathrm{x}$  speed-up on the largest-scale dataset. The improvement continues to expand even further as the scale of the problem grows. Second, ReHLine also yields considerable improvements for specialized algorithms, including LIBLINEAR in SVM, HQREG in Huber minimization, and LIGHTNING in sSVM. Third, given that the generic solvers fail or time out in multiple tasks and datasets, ReHLine has shown remarkable flexibility when it comes to solving various domain-specific problems.
In this paper, we present a new algorithm ReHLine designed to efficiently solve general regularized ERM with the convex PLQ loss function and optional linear constraints. Through the transformation of a PLQ loss into a composite ReLU-ReHU function, we have developed a novel CD algorithm that updates primal and dual variables simultaneously. This approach has proven to be highly effective in reducing the computational complexity while at the same time achieving a fast linear convergence rate. Based on our experiments, ReHLine demonstrates remarkable flexibility and computational efficiency, outperforming existing generic and specialized methods in a range of problems.
Limitation. Although our approach has exhibited substantial versatility in addressing a diverse range of problems, it is important to acknowledge that it may not be universally applicable to all convex optimization problems. The main limit of our approach stems from the restriction imposed by the formulation (1). For instance, (1) requires a convex PLQ loss function and a strict  $l_{2}$ -regularization. We recognize this limitation and intend to address it in future research pursuits. Moreover, we are interested in exploiting additional loss function properties, such as symmetry.
Ben Dai's work was supported in part by HK GRF-24302422 and GRF-14304823. Yixuan Qiu's work was supported in part by National Natural Science Foundation of China (12101389), Shanghai Pujiang Program (21PJC056), MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD110001), and Shanghai Research Center for Data Science and Decision Technology. The authors confirm contribution to the paper as follows. The main algorithm: Ben Dai and Yixuan Qiu; theory and proofs: Ben Dai;  $\mathrm{C + + }$  implementation: Yixuan Qiu; Python interface: Ben Dai and Yixuan Qiu; R interface: Yixuan Qiu; manuscript preparation: Ben Dai and Yixuan Qiu, equally.
[1] Alekseeva, E., Brotcorne, L., Lepaul, S., and Montmeat, A. (2018). A bilevel approach to optimize electricity prices. *Yugoslav Journal of Operations Research*, 29(1):9-30.
[2] ApS, M. (2019). The MOSEK Optimization Toolbox for MATLAB Manual. Version 9.0.
[3] Beck, A. and Teboulle, M. (2012). Smoothing and first order methods: A unified framework. SIAM Journal on Optimization, 22(2):557-580.
[4] Blondel, M. and Pedregosa, F. (2016). Lightning: large-scale linear classification, regression and ranking in Python.
[5] Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning, 3(1):1-122.
[6] Bubeck, S. et al. (2015). Convex optimization: algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231-357.
[7] Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3):273-297.
[8] Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems, 27.
[9] Deng, W., Lai, M.-J., Peng, Z., and Yin, W. (2017). Parallel multi-block ADMM with o(1/k) convergence. Journal of Scientific Computing, 71:712-736.
[10] Diamond, S. and Boyd, S. (2016). CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(1):2909–2913.
[11] Domahidi, A., Chu, E., and Boyd, S. (2013). Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC), pages 3071-3076. IEEE.
[33] Rennie, J. D. and Srebro, N. (2005). Loss functions for preference levels: Regression with discrete ordered labels. In Proceedings of the IJCAI Multidisciplinary Workshop on Advances in Preference Handling, volume 1. Citeseer.
[34] Rockafellar, R. T. and Wets, R. J.-B. (2009). Variational analysis. Springer Science & Business Media.
[35] Saha, A. and Tewari, A. (2013). On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization, 23(1):576-601.
[36] Salas, V. and Saurina, J. (2002). Credit risk in two institutional regimes: Spanish commercial and savings banks. Journal of Financial Services Research, 22(3):203-224.
[37] Schmidt, M., Le Roux, N., and Bach, F. (2017). Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162:83-112.
[38] Schumaker, L. (2007). Spline Functions: Basic Theory. Cambridge university press.
[39] Shalev-Shwartz, S. and Zhang, T. (2013). Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14(1).
[40] Shalev-Shwartz, S. and Zhang, T. (2014). Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. In International Conference on Machine Learning, pages 64-72. PMLR.
[41] Shen, X., Diamond, S., Gu, Y., and Boyd, S. (2016). Disciplined convex-concave programming. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 1009-1014. IEEE.
[42] Tseng, P. and Yun, S. (2010). A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Computational Optimization and Applications, 47(2):179-206.
[43] Vapnik, V. (1991). Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, pages 831-838.
[44] Vapnik, V. (1999). The Nature of Statistical Learning Theory. Springer science & business media.
[45] Wright, S. J. (2015). Coordinate descent algorithms. Mathematical Programming, 151(1):3-34.
[46] Yi, C. and Huang, J. (2017). Semismooth newton coordinate descent algorithm for elastic-net penalized Huber loss regression and quantile regression. Journal of Computational and Graphical Statistics, 26(3):547-557.
[47] Zafar, M. B., Valera, I., Gomez-Rodriguez, M., and Gummadi, K. P. (2019). Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research, 20(1):2737-2778.
[48] Zafar, M. B., Valera, I., Rodriguez, M. G., and Gummadi, K. P. (2017). Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pages 962-970.
[49] Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301-320.
A Example Models
SVMs with fairness constraints (FairSVM). The formulation for FairSVM [48] is:
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} \quad \frac {C}{n} \sum_ {i = 1} ^ {n} (1 - y _ {i} \boldsymbol {\beta} ^ {\intercal} \mathbf {x} _ {i}) _ {+} + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2},
$$
$$
\text {s . t .} \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {z} _ {i} \boldsymbol {\beta} ^ {\intercal} \mathbf {x} _ {i} \leq \boldsymbol {\rho}, \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {z} _ {i} \boldsymbol {\beta} ^ {\intercal} \mathbf {x} _ {i} \geq - \boldsymbol {\rho}, \tag {A.1}
$$
where  $\mathbf{x}_i\in \mathbb{R}^d$  is a feature vector, and  $y_{i}\in \{-1,1\}$  is a binary label,  $\mathbf{z}_i\in \mathbb{R}^{d_0}$  is a collection of centered sensitive features  $(\sum_{i = 1}^{n}\mathbf{z}_{i} = \mathbf{0})$ , such as gender and/or race. The constraints in (A.1) constrain the correlation between the sensitive features  $\mathbf{z}_i$  and the decision function  $\beta^{\intercal}\mathbf{x}_i$ , and the constants  $\rho \in \mathbb{R}_{+}^{d_{0}}$  trade off predictive accuracy and fairness. Note that the FairSVM (A.1) can be rewritten as a ReHLine optimization of (4) with
$$
\mathbf {U} \leftarrow - C \mathbf {y} ^ {\intercal} / n, \quad \mathbf {V} \leftarrow C \mathbf {1} _ {n} ^ {\intercal} / n, \quad \mathbf {A} \leftarrow \left( \begin{array}{c} \mathbf {Z} ^ {\intercal} \mathbf {X} / n \\ - \mathbf {Z} ^ {\intercal} \mathbf {X} / n \end{array} \right), \quad \mathbf {b} \leftarrow \left( \begin{array}{c} \boldsymbol {\rho} \\ \boldsymbol {\rho} \end{array} \right),
$$
where  $\mathbf{1}_n = (1, \dots, 1)^\top$  is the length- $n$  one vector,  $\mathbf{X} \in \mathbb{R}^{n \times d}$  is the feature matrix, and  $\mathbf{y} = (y_1, \dots, y_n)^\top$  is the response vector.
Elastic net regularized quantile regression (ElasticQR). The formulation for the elastic net penalized quantile regression [19] is: