SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ben Dai*
bendai@cuhk.edu.hk
Yixuan Qiu*†
qiuyixuan@sufe.edu.cn
Empirical risk minimization (ERM) is a crucial framework that offers a general approach to handling a broad range of machine learning tasks. In this paper, we propose a novel algorithm, called ReHLine, for minimizing a set of regularized ERMs with convex piecewise linear-quadratic loss functions and optional linear constraints. The proposed algorithm can effectively handle diverse combinations of loss functions, regularization, and constraints, making it particularly well-suited for complex domain-specific problems. Examples of such problems include FairSVM, elastic net regularized quantile regression, Huber minimization, etc. In addition, ReHLine enjoys a provable linear convergence rate and exhibits a per-iteration computational complexity that scales linearly with the sample size. The algorithm is implemented with both Python and R interfaces, and its performance is benchmarked on various tasks and datasets. Our experimental results demonstrate that ReHLine significantly surpasses generic optimization solvers in terms of computational efficiency on large-scale datasets. Moreover, it also outperforms specialized solvers such as LIBLINEAR in SVMs, HQREG in Huber minimization and LIGHTNING (SAGA, SAG, SDCA, SVRG) in smoothed SVMs, exhibiting exceptional flexibility and efficiency. The source code, project page, accompanying software, and the Python/R interface can be accessed through the link: https://github.com/softmin/ReHLine.
Empirical risk minimization (ERM) with different losses is a fundamental framework in statistical machine learning [43], which has demonstrated a significant impact on many application areas, such as artificial intelligence, computational biology, computer vision, natural language processing, electronic engineering, and finance. In fact, many common practical ERMs are associated with a piecewise linear-quadratic (PLQ; [34]) loss function, including support vector machines (SVMs; [7]), the least absolute deviation regression, quantile regression [26], Huber regression [22], borrowing cost function in portfolio selection [28], and pricing/planning problems in electricity and operational research [16, 1], among many others. In addition, ERMs are frequently blended with linear constraints, for instance, the fairness constraints in fair classification [48], monotonicity constraints in classification or regression, etc. More linear constraints according to the prior domain knowledge, such as the sum-to-one constraints in the portfolio variance minimization problem [24], negative or positive constraints in non-performing loan [36], are considered in various real applications. Therefore, it has become of utmost importance to develop fast and scalable numerical algorithms to solve ERMs with PLQ losses based on large-scale datasets and flexible linear constraints.
*Co-first authorship, and the order is determined by coin toss. † Corresponding author.
In this paper, we consider a general regularized ERM based on a convex PLQ loss with linear constraints:
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} \sum_ {i = 1} ^ {n} L _ {i} \left(\mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta}\right) + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \quad \text {s . t .} \mathbf {A} \boldsymbol {\beta} + \mathbf {b} \geq \mathbf {0}, \tag {1}
$$
where  $\mathbf{x}_i\in \mathbb{R}^d$  is the feature vector for the  $i$ -th observation,  $\beta \in \mathbb{R}^d$  is an unknown coefficient vector,  $\mathbf{A}\in \mathbb{R}^{K\times d}$  and  $\mathbf{b}\in \mathbb{R}^K$  are defined as linear inequality constraints for  $\beta$ , and  $L_{i}(\cdot)\geq 0$  is a convex PLQ loss function. The PLQ function class greatly generalizes existing popular loss functions, and the convexity of  $L_{i}(\cdot)$  guarantees the global convergence of optimization. Here, we focus on working with a large-scale dataset, where the dimension of the coefficient vector and the total number of constraints are comparatively much smaller than the sample size, that is,  $d\ll n$  and  $K\ll n$ .
Although (1) is a strongly convex problem with affine constraints, thus admitting a unique global optimum, directly solving (1) by a generic solver may encounter various challenges. For example,  $L_{i}(\cdot)$  is in general non-smooth, so gradient-based methods may fail, and hence slow-convergent subgradient methods need to be used. In addition, the constraint set is a polyhedron, whose projection operator is non-trivial to compute, making various projection-based methods difficult to apply.
Of course, for some specific forms of  $L_{i}(\cdot)$ , highly efficient solvers for (1) indeed exist. One remarkable example is the LIBLINEAR solver [12], which has gained great success in solving large-scale SVMs. However, LIBLINEAR highly relies on the hinge loss function, so its success does not directly transfer to more general loss functions.
Outline. To this end, in this article we develop a novel algorithm to solve (1) with the greatest generality. The proposed algorithm, named ReHLine, has three key ingredients. First, we show that any convex PLQ loss function can be decomposed as the sum of a finite number of rectified linear units (ReLU, [15]) and rectified Huber units (ReHU, defined in Section 2). Second, based on this decomposition, the dual problem of (1) is shown to be a box-constrained quadratic programming (box-QP) problem. Finally, a special paired primal-dual coordinate descent algorithm is developed to simultaneously solve the primal and dual problems, with a computational cost linear in  $n$ .
We emphasize that for the second point, the dual box-QP problem is highly structured. Therefore, a general-purpose solver may not fully exploit the structure, thus leading to slower convergence rates or higher computational costs. As a direct comparison between the proposed ReHLine solver and existing algorithms, including the projected (sub-)gradient descent (P-GD; [6]), the coordinate descent (CD; [45]), the interior-point methods (IPMs; [13, 14, 18]), the stochastic dual coordinate ascent (SDCA; [39, 40]) related methods, and the alternating direction method of multipliers (ADMM; [5]), Table 1 summarizes the required number of iterations and per-iteration cost of each method. More details and discussions for this comparison are given in Section 4.
Table 1: Overview of existing algorithms for solving (1). Column COMPLEXITY (PER ITERATION) shows the computational complexity of the algorithm per iteration. Here, we focus only on the order of  $n$  since  $d \ll  n$  is assumed in our setting. Column #ITERATIONS shows the number of iterations needed to achieve an accuracy of  $\varepsilon$  to the minimizer.
ALGORITHM	COMPLEXITY (PER ITERATION)	#ITERATIONS	COMPLEXITY (TOTAL)
P-GD	O(n)	O(ε-1)[6]	O(nε-1)
CD	O(n2)	O(log(ε-1))[31]	O(n2log(ε-1))
IPM	O(n2)	O(log(ε-1))[18]	O(n2log(ε-1))
ADMM	O(n2)	o(ε-1)[9, 20]	o(n2ε-1)
SDCA	O(n)	O(ε-1)[39]	O(nε-1)
ReHLine (ours)	O(n)	O(log(ε-1))	O(n log(ε-1))
Contribution. Compared with other specialized ERM solvers or general-purpose box-QP solvers, the proposed ReHLine solver has four appealing "linear properties":
1. It applies to any convex piecewise linear-quadratic loss function (potential for non-smoothness included), including the hinge loss, the check loss, the Huber loss, etc.
2. In addition, it supports linear equality and inequality constraints on the parameter vector.
3. The optimization algorithm has a provable linear convergence rate.
4. The per-iteration computational complexity is linear in the sample size.
Moreover, ReHLine is designed to be a computationally efficient and practically useful software package for large-scale ERM problems. As an example, our experiments in Section 5 have shown that ReHLine significantly surpasses generic solvers in terms of computational efficiency on large-scale datasets, and provides reasonable improvements over specialized solvers and problems, such as LIBLINEAR, HQREG and LIGHTNING, given that LIBLINEAR, HQREG and LIGHTNING are one of the fastest and most widely-used solvers for SVM, Huber minimization, and smoothed SVM, respectively. Finally, ReHLine is available as user-friendly software in both Python and R, which facilitates ease of use for researchers, practitioners, and a wider audience.
2 The ReLU-ReHU decomposition
As is introduced in Section 1, we attempt to solve ERM with a general loss function, and one sensible choice is the class of convex PLQ functions. A univariate loss function  $f: \mathbb{R} \to \mathbb{R}_{\geq 0}$  is PLQ if there exist a finite number of knots  $t_1 < t_2 < \dots < t_K$  such that  $f$  is equal to a linear or quadratic function on each of the intervals  $(-\infty, t_1], [t_1, t_2], \ldots, [t_{K-1}, t_K], [t_K, \infty)$ . In what follows we merely consider nonnegative convex PLQ functions, which also imply the continuity of the loss function. It is well known that PLQ functions are universal approximators of continuous functions [38], but the canonical definition of PLQ is not convenient for optimization algorithms. Instead, in this section we prove that convex PLQ functions can actually be decomposed into a series of simple and basic units.
Figure 1: The ReLU function and the ReHU function with  $\tau = 1,2,3,\infty$
Before proceeding, we introduce a generalized quadratic version of ReLU, the so-called rectified Huber unit, ReHU:
$$
\operatorname {R e H U} _ {\tau} (z) = \left\{ \begin{array}{l l} 0, & z \leq 0 \\ z ^ {2} / 2, & 0 <   z \leq \tau , \\ \tau (z - \tau / 2), & z > \tau \end{array} \right. \tag {2}
$$
where  $\tau \in [0,\infty ]$  is a hyperparameter of ReHU switching from quadratic to linear form, see examples in Figure 1. ReHU also incorporates the rectified quadratic unit (ReQU) as a special case, that is,  $\mathrm{ReHU}_{\infty}(z) = \mathrm{ReQU}(z) / 2 = (z_{+})^{2} / 2$  when  $\tau = \infty$ , where  $z_{+} = \max \{z,0\}$ . Then we give the characterization of a composite ReLU-ReHU function.
Definition 1 (Composite ReLU-ReHU function). A function  $L(z)$  is composite ReLU-ReHU, if there exist  $\mathbf{u}, \mathbf{v} \in \mathbb{R}^L$  and  $\boldsymbol{\tau}, \mathbf{s}, \mathbf{t} \in \mathbb{R}^H$  such that
$$
L (z) = \sum_ {l = 1} ^ {L} \operatorname {R e L U} \left(u _ {l} z + v _ {l}\right) + \sum_ {h = 1} ^ {H} \operatorname {R e H U} _ {\tau_ {h}} \left(s _ {h} z + t _ {h}\right), \tag {3}
$$
where  $\mathrm{ReLU}(z) = z_{+}$ , and  $\mathrm{ReHU}_{\tau_h}(z)$  is defined in (2).
Clearly, the composite ReLU-ReHU function class is closed under affine transformations, as is indicated by Proposition 1.
Proposition 1 (Closure under affine transformation). If  $L(z)$  is a composite ReLU-ReHU function as in (3), then for any  $c > 0$ ,  $p \in \mathbb{R}$ , and  $q \in \mathbb{R}$ ,  $cL(pz + q)$  is also composite ReLU-ReHU, that is,
$$
c L (p z + q) = \sum_ {l = 1} ^ {L} \mathrm {R e L U} (u _ {l} ^ {\prime} z + v _ {l} ^ {\prime}) + \sum_ {h = 1} ^ {H} \mathrm {R e H U} _ {\tau_ {h} ^ {\prime}} (s _ {h} ^ {\prime} z + t _ {h} ^ {\prime}),
$$
where  $u_{l}^{\prime} = c p u_{l}$ ,  $v_{l}^{\prime} = c u_{l} q + c v_{l}$ ,  $\tau_{h}^{\prime} = \sqrt{c}\tau_{h}$ ,  $s_{h}^{\prime} = \sqrt{c} p s_{h}$ , and  $t_{h}^{\prime} = \sqrt{c}(s_{h}q + t_{h})$ .
More importantly, we show that the composite ReLU-ReHU function class is equivalent to the class of convex PLQ functions.
Theorem 1. A loss function  $L: \mathbb{R} \to \mathbb{R}_{\geq 0}$  is convex PLQ if and only if it is composite ReLU-ReHU.
Based on the fact that the loss functions of various SVMs (the hinge loss, the squared hinge loss, and the smoothed hinge loss), quantile regression (the check loss), and least absolute deviation (the absolute loss) are all convex PLQs, Theorem 1 suggests that Definition 1 incorporates numerous widely-used losses in statistics and machine learning applications. Table 2 provides a convenient translation from some commonly-used loss functions to their ReLU-ReHU representation.
Table 2: Some widely-used composite ReLU-ReHU losses as in (3). Here SVM is weighted SVMs based on the hinge loss [7], sSVM is smoothed SVMs based on the smoothed hinge loss [33],  $\mathrm{SVM}^2$  is weighted squared SVMs based on the squared hinge loss [7], LAD is the least absolute deviation regression, SVR is support vector regression with the  $\varepsilon$ -insensitive loss [44], and QR is quantile regression with the check loss [26].
PROBLEM	LOSS (Li(zi))	COMPOSITE RELU-REHU PARAMETERS
SVM	ci(1 - yi zi)_+	u1i = -ciyi, v1i = ci
sSVM	ciReLU1(-(yi zi - 1))	s1i = -√ciyi, t1i = √ci, τ = √ci
SVM²	ci((1 - yi zi)_+)²	s1i = -√2ciyi, t1i = √2ci, τ = ∞
LAD	ci|yi - zi|	u1i = ci, v1i = -ciyi, u2i = -ci, v2i = ciyi
SVR	ci(|yi - zi| - ε)_+	u1i = ci, v1i = -(yi + ε), u2i = -ci, v2i = yi - ε
QR	ciρκ(yi - zi) (A.3)	u1i = -ciκ, v1i = κciyi, u2i = ci(1 - κ), v2i = -ci(1 - κ)yi
Table 3: Some widely-used linear constraints of the form  $\mathbf{A}\beta +\mathbf{b}\geq 0$  , where U-bidiag([-1,1]) is an upper bidiagonal matrix with main diagonal being -1 and upper diagonal being 1.
CONSTRAINT	FORM	REHLINE PARAMETERS
Nonnegative	β≥0	A = -I, b = 0
Box	l ≤ β ≤ u	A = [I; -I], b = [-l; u]
Monotonicity	β1 ≤··· ≤ βd	A = B1:d-1, B = U-bidiag([-1, 1]), b = 0d-1
Fairness (A.1)	|Z^T X β/n| ≤ ρ	A = Z^T X [I; -I]/n, b = [ρ; ρ]
Taken together, our algorithm is designed to address the empirical regularized ReLU-ReHU minimization problem, named ReHLine optimization, of the following form:
$$
\begin{array}{l} \min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} \sum_ {i = 1} ^ {n} \sum_ {l = 1} ^ {L} \operatorname {R e L U} (u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} + v _ {l i}) + \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \operatorname {R e H U} _ {\tau_ {h i}} (s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} + t _ {h i}) + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \\ \text {s . t .} \mathbf {A} \boldsymbol {\beta} + \mathbf {b} \geq \mathbf {0}, \tag {4} \\ \end{array}
$$
where  $\mathbf{U} = (u_{li}),\mathbf{V} = (v_{li})\in \mathbb{R}^{L\times n}$  and  $\mathbf{S} = (s_{hi}),\mathbf{T} = (t_{hi}),\pmb {\tau} = (\tau_{hi})\in \mathbb{R}^{H\times n}$  are the ReLU-ReHU loss parameters, as illustrated in Table 2, and (A, b) are the constraint parameters, as illustrated in Table 3. This formulation has a wide range of applications, including statistics, machine learning, computational biology, and social studies. Some popular examples include SVMs with fairness constraints (FairSVM), elastic net regularized quantile regression (ElasticQR), ridge Huber minimization (RidgeHuber), and smoothed SVMs (sSVM). See Appendix A for details.
Despite appearing to impose limitations on the  $l_{2}$ -regularization in (4), our ReHLine formulation can actually be extended to more general regularization forms, including the elastic net penalty [49].
Proposition 2. If  $L_{i}$  is a composite ReLU-ReHU loss function with parameters  $(u_{li}, v_{li})_{l=1}^{L}$  and  $(s_{hi}, t_{hi}, \tau_{hi})_{h=1}^{H}$ , then its elastic net regularized minimization with  $\lambda_{1} \geq 0$  and  $\lambda_{2} > 0$ ,
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} \sum_ {i = 1} ^ {n} L _ {i} \left(\mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta}\right) + \lambda_ {1} \| \boldsymbol {\beta} \| _ {1} + \frac {\lambda_ {2}}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \quad \text {s . t .} \quad \mathbf {A} \boldsymbol {\beta} + \mathbf {b} \geq \mathbf {0},
$$
can be rewritten as the form of ReHLine optimization (4) with
$$
\mathbf {U} \leftarrow \left( \begin{array}{c c} \frac {1}{\lambda_ {2}} \mathbf {U} & \mathbf {0} _ {L \times d} \\ \mathbf {0} _ {n} ^ {\intercal} & \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & - \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d} ^ {\intercal} \end{array} \right), \quad \mathbf {V} \leftarrow \left( \begin{array}{c c} \frac {1}{\lambda_ {2}} \mathbf {V} & \mathbf {0} _ {L \times d} \\ \mathbf {0} _ {n + d} ^ {\intercal} & \\ \mathbf {0} _ {n + d} ^ {\intercal} & \end{array} \right), \quad \mathbf {X} \leftarrow \left( \begin{array}{c} \mathbf {X} \\ \mathbf {I} _ {d} \end{array} \right),
$$
$$
\mathbf {S} \leftarrow \left( \begin{array}{c c} \sqrt {\frac {1}{\lambda_ {2}}} \mathbf {S} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right), \qquad \mathbf {T} \leftarrow \left( \begin{array}{c c} \sqrt {\frac {1}{\lambda_ {2}}} \mathbf {T} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right), \qquad \boldsymbol {\tau} \leftarrow \left( \begin{array}{c c} \sqrt {\frac {1}{\lambda_ {2}}} \boldsymbol {\tau} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right),
$$
where  $\leftarrow$  indicates assigning a value to a parameter.
3 Optimization algorithm
In this section, we present the ReHLine algorithm to tackle the optimization in (4). The proposed algorithm is inspired by LIBLINEAR [12], which has gained great success in solving standard SVMs. The key idea of ReHLine is to leverage the linear property of Karush-Kuhn-Tucker (KKT) conditions, thus simplifying the iterative complexity of CD and achieving rapid convergence.
3.1 Primal and dual formulations
To proceed, we first demonstrate equivalent formulations for ReLU and ReHU, respectively, where the equivalence means that the value on the left is equal to the minimum value of the right-hand side:
$$
\operatorname {R e L U} (z) \iff \begin{array}{c c c} \min  _ {\pi} & \pi \\ \text {s . t .} & \pi \geq z, \\ & \pi \geq 0. \end{array} \quad \operatorname {R e H U} (z) \iff \begin{array}{c c c} \min  _ {\theta , \sigma} & \frac {1}{2} \theta^ {2} + \tau \sigma \\ \text {s . t .} & \theta + \sigma \geq z, \\ & \sigma \geq 0. \end{array} \tag {5}
$$
It is worth noting that when  $\tau = \infty$ , the optimal solution for  $\sigma$  is 0, and hence the last constraint  $(\sigma \geq 0)$  can be removed. Then according to the definition of the composite ReLU-ReHU loss function and the equivalent forms in (5), the ReHLine optimization problem (4) can be rewritten as:
$$
\begin{array}{l} \min  _ {\boldsymbol {\beta}, \boldsymbol {\Pi}, \boldsymbol {\Theta}, \boldsymbol {\Sigma}} \sum_ {i = 1} ^ {n} \sum_ {l = 1} ^ {L} \pi_ {l i} + \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \frac {1}{2} \theta_ {h i} ^ {2} + \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \tau_ {h i} \sigma_ {h i} + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2} \\ \mathrm {s . t .} \quad \mathbf {A} \boldsymbol {\beta} + \mathbf {b} \geq \mathbf {0}, \quad \pi_ {l i} \geq u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} + v _ {l i}, \quad \theta_ {h i} + \sigma_ {h i} \geq s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} + t _ {h i}, \\ \pi_ {l i} \geq 0, \quad \sigma_ {h i} \geq 0, \quad \text {f o r a l l} (i, l, h), \tag {6} \\ \end{array}
$$
where  $\Pi = (\pi_{li})\in \mathbb{R}^{L\times n}$ ,  $\Theta = (\theta_{hi})\in \mathbb{R}^{H\times n}$  and  $\Sigma = (\sigma_{hi})\in \mathbb{R}^{H\times n}$  are slack variables. Theorem 2 shows the dual form of the primal formulation (6) by using the Lagrangian duality theory, and the relation between primal and dual variables is also provided.
Theorem 2. The Lagrangian dual problem of (6) is:
$$
\begin{array}{l} (\widehat {\boldsymbol {\xi}}, \widehat {\boldsymbol {\Lambda}}, \widehat {\boldsymbol {\Gamma}}) = \operatorname * {a r g m i n} _ {\boldsymbol {\xi}, \boldsymbol {\Lambda}, \boldsymbol {\Gamma}} \mathcal {L} (\boldsymbol {\xi}, \boldsymbol {\Lambda}, \boldsymbol {\Gamma}) \\ \mathrm {s . t .} \quad \boldsymbol {\xi} \geq \mathbf {0}, \quad \mathbf {E} \geq \boldsymbol {\Lambda} \geq \mathbf {0}, \quad \boldsymbol {\tau} \geq \boldsymbol {\Gamma} \geq \mathbf {0}, \tag {7} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {\xi}, \boldsymbol {\Lambda}, \boldsymbol {\Gamma}) = \frac {1}{2} \boldsymbol {\xi} ^ {\intercal} \mathbf {A} \mathbf {A} ^ {\intercal} \boldsymbol {\xi} + \frac {1}{2} v e c (\boldsymbol {\Lambda}) ^ {\intercal} \bar {\mathbf {U}} _ {(3)} ^ {\intercal} \bar {\mathbf {U}} _ {(3)} v e c (\boldsymbol {\Lambda}) + \frac {1}{2} v e c (\boldsymbol {\Gamma}) ^ {\intercal} \left(\bar {\mathbf {S}} _ {(3)} ^ {\intercal} \bar {\mathbf {S}} _ {(3)} + \mathbf {I}\right) v e c (\boldsymbol {\Gamma}) \\ - \boldsymbol {\xi} ^ {\mathsf {T}} \mathbf {A} \bar {\mathbf {U}} _ {(3)} v e c (\boldsymbol {\Lambda}) - \boldsymbol {\xi} ^ {\mathsf {T}} \mathbf {A} \bar {\mathbf {S}} _ {(3)} v e c (\boldsymbol {\Gamma}) + v e c (\boldsymbol {\Lambda}) ^ {\mathsf {T}} \bar {\mathbf {U}} _ {(3)} ^ {\mathsf {T}} \bar {\mathbf {S}} _ {(3)} v e c (\boldsymbol {\Gamma}) \\ + \boldsymbol {\xi} ^ {\mathsf {T}} \mathbf {b} - \operatorname {T r} (\boldsymbol {\Lambda} \mathbf {V} ^ {\mathsf {T}}) - \operatorname {T r} (\boldsymbol {\Gamma} \mathbf {T} ^ {\mathsf {T}}), \tag {8} \\ \end{array}
$$
where  $\pmb{\xi} \in \mathbb{R}^{K}$ ,  $\pmb{\Lambda} = (\lambda_{li}) \in \mathbb{R}^{L \times n}$ , and  $\Gamma = (\gamma_{hi}) \in \mathbb{R}^{H \times n}$  are dual variables,  $\bar{\mathbf{U}}_{(3)} \in \mathbb{R}^{d \times nL}$  and  $\bar{\mathbf{S}}_{(3)} \in \mathbb{R}^{d \times nH}$  are the mode-3 unfolding of the tensors  $\bar{\mathbf{U}} = (u_{lij}) \in \mathbb{R}^{L \times n \times d}$  and  $\bar{\mathbf{S}} = (s_{hij}) \in \mathbb{R}^{H \times n \times d}$ , respectively,  $u_{lij} = u_{li}x_{ij}$ ,  $s_{hij} = s_{hi}x_{ij}$ ,  $\mathbf{I}$  is the identity matrix, and all inequalities are elementwise.
Moreover, the optimal point  $\hat{\beta}$  of (6) can be recovered as:
$$
\widehat {\boldsymbol {\beta}} = \sum_ {k = 1} ^ {K} \widehat {\xi} _ {k} \mathbf {a} _ {k} - \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \left(\sum_ {l = 1} ^ {L} \widehat {\lambda} _ {l i} u _ {l i} + \sum_ {h = 1} ^ {H} \widehat {\gamma} _ {h i} s _ {h i}\right) = \mathbf {A} ^ {\intercal} \widehat {\boldsymbol {\xi}} - \bar {\mathbf {U}} _ {(3)} v e c (\widehat {\boldsymbol {\Lambda}}) - \bar {\mathbf {S}} _ {(3)} v e c (\widehat {\boldsymbol {\Gamma}}). \tag {9}
$$
3.2 ReHLine update rules
Indeed, (9) is a KKT condition of (6), and the main idea of the proposed ReHLine solver is to update the dual variables  $(\pmb {\xi},\pmb {\Lambda},\pmb {\Gamma})$  by CD on (7) with an analytic solution, and simultaneously update the primal variable  $\beta$  by the KKT condition (9). Most importantly, the computational complexity of the CD updates on (7) can be significantly reduced by using the information of  $\beta$ .
Canonical CD updates. As a first step, we consider the canonical CD update rule that directly optimizes the dual problem (7) with respect to a single variable. For brevity, in this section we only illustrate the result for  $\lambda_{li}$  variables, and the full details are given in Appendix B.
By excluding the terms unrelated to  $\lambda_{li}$ , we have  $\lambda_{li}^{\mathrm{new}} = \operatorname*{argmin}_{0\leq \lambda \leq 1}\mathcal{L}_{li}(\lambda)$ , where
$$
\begin{array}{l} \mathcal {L} _ {l i} (\lambda) = \frac {1}{2} u _ {l i} ^ {2} \left(\mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i}\right) \lambda^ {2} + \sum_ {\left(l ^ {\prime}, i ^ {\prime}\right) \neq (l, i)} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} u _ {l i} \left(\mathbf {x} _ {i ^ {\prime}} ^ {\intercal} \mathbf {x} _ {i}\right) \lambda - \sum_ {k = 1} ^ {K} \xi_ {k} u _ {l i} \left(\mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i}\right) \lambda \\ + \sum_ {h ^ {\prime}, i ^ {\prime}} u _ {l i} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\mathsf {T}} \mathbf {x} _ {i ^ {\prime}} \lambda - v _ {l i} \lambda . \\ \end{array}
$$
Therefore, by simple calculations we obtain
$$
\lambda_ {l i} ^ {\text {n e w}} = \mathcal {P} _ {[ 0, 1 ]} \left(\frac {u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \left(\sum_ {k = 1} ^ {K} \xi_ {k} \mathbf {a} _ {k} - \sum_ {\left(l ^ {\prime} , i ^ {\prime}\right) \neq (l , i)} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}} - \sum_ {h ^ {\prime} , i ^ {\prime}} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}}\right) + v _ {l i}}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right), \tag {10}
$$