SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d + 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {\kappa} \left(y _ {i} - \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} _ {1: d} - \beta_ {d + 1}\right) + \lambda_ {1} \| \boldsymbol {\beta} \| _ {1} + \frac {\lambda_ {2}}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \tag {A.2}
$$
$$
\rho_ {\kappa} (u) = u \left(\kappa - \mathbf {1} (u <   0)\right) \tag {A.3}
$$
is the check loss,  $\kappa \in (0,1)$  is a prespecified sample quantile,  $\mathbf{x}_i\in \mathbb{R}^d$  is a feature vector,  $y_{i}\in \mathbb{R}$  is a response, and  $\lambda_1,\lambda_2\geq 0$  are weights of lasso and ridge penalties. Then, the ElasticQR (A.2) can be rewritten as a ReHLine optimization of (4) with
$$
\mathbf {U} \leftarrow \left( \begin{array}{c c} - \frac {\kappa}{n \lambda_ {2}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \\ \frac {1 - \kappa}{n \lambda_ {2}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d + 1} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & - \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d + 1} ^ {\intercal} \end{array} \right), \quad \mathbf {V} \leftarrow \left( \begin{array}{c c} \frac {\kappa}{n \lambda_ {2}} \mathbf {y} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \\ - \frac {1 - \kappa}{n \lambda_ {2}} \mathbf {y} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & \mathbf {0} _ {d + 1} ^ {\intercal} \end{array} \right), \quad \mathbf {X} \leftarrow \left( \begin{array}{c c} \mathbf {X} & \mathbf {1} _ {n} \\ \mathbf {I} _ {d + 1} & \end{array} \right),
$$
where  $\mathbf{I}_{d + 1}$  is the identity matrix.
Elastic net regularized Huber minimization. The formulation for the elastic net penalized Huber minimization [23] is:
$$
\min  _ {\beta} \frac {1}{n} \sum_ {i = 1} ^ {n} H _ {\kappa} \left(y _ {i} - \mathbf {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) + \lambda_ {1} \| \boldsymbol {\beta} \| _ {1} + \frac {\lambda_ {2}}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \tag {A.4}
$$
where  $H_{\kappa}(\cdot)$  is the Huber loss with a given parameter  $\kappa$
$$
H _ {\kappa} (z) = \left\{ \begin{array}{l l} z ^ {2} / 2, & 0 <   | z | \leq \kappa , \\ \kappa (| z | - \kappa / 2), & | z | > \kappa . \end{array} \right.
$$
In this case, (A.4) can be rewritten as a ReHLine optimization with
$$
\mathbf {S} \leftarrow \left( \begin{array}{c c} - \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \\ \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right), \quad \mathbf {T} \leftarrow \left( \begin{array}{c c} \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {y} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \\ - \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {y} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right), \quad \boldsymbol {\tau} \leftarrow \left( \begin{array}{c c} \kappa \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \\ \kappa \sqrt {\frac {1}{n \lambda_ {2}}} \mathbf {1} _ {n} ^ {\intercal} & \mathbf {0} _ {d} ^ {\intercal} \end{array} \right),
$$
$$
\mathbf {U} \leftarrow \left( \begin{array}{c c} \mathbf {0} _ {n} ^ {\intercal} & \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d} ^ {\intercal} \\ \mathbf {0} _ {n} ^ {\intercal} & - \frac {\lambda_ {1}}{\lambda_ {2}} \mathbf {1} _ {d} ^ {\intercal} \end{array} \right), \quad \mathbf {V} \leftarrow \mathbf {0}, \quad \mathbf {X} \leftarrow \left( \begin{array}{c} \mathbf {X} \\ \mathbf {I} _ {d} \end{array} \right).
$$
Smoothe d SVM (sSVM). The formulation for the smoothed SVMs [33] is:
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n} \sum_ {i = 1} ^ {n} V \left(y _ {i} \boldsymbol {\beta} ^ {\intercal} \mathbf {x} _ {i}\right) + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2}, \tag {A.5}
$$
where  $\mathbf{x}_i\in \mathbb{R}^d$  is a feature vector, and  $y_{i}\in \{-1,1\}$  is a binary label, and  $V_{\kappa}(\cdot)$  is the modified Huber loss or the smoothed hinge loss:
$$
V (z) = \left\{ \begin{array}{l l} 0, & z \geq 1, \\ (1 - z) ^ {2} / 2, & 0 <   z \leq 1, \\ (1 / 2 - z), & z <   0. \end{array} \right.
$$
In this case, (A.5) can be rewritten as a ReHLine optimization with
$$
\mathbf {S} \leftarrow - \sqrt {C / n} \mathbf {y} ^ {\intercal}, \quad \mathbf {T} \leftarrow \sqrt {C / n} \mathbf {1} _ {n} ^ {\intercal}, \quad \boldsymbol {\tau} \leftarrow \sqrt {C / n} \mathbf {1} _ {n} ^ {\intercal}.
$$
B Update Rules for (7)
B.1 Canonical CD updates
Update  $\xi_{k}$  given others. For  $k = 1,\dots ,K$
$$
\begin{array}{l} \xi_ {k} ^ {\text {n e w}} = \underset {\xi_ {k} \geq 0} {\operatorname {a r g m i n}} \frac {1}{2} \mathbf {a} _ {k} ^ {\intercal} \mathbf {a} _ {k} \xi_ {k} ^ {2} + \sum_ {k ^ {\prime} \neq k} \xi_ {k ^ {\prime}} \mathbf {a} _ {k ^ {\prime}} ^ {\intercal} \mathbf {a} _ {k} \xi_ {k} - \sum_ {l, i} \lambda_ {l i} u _ {l i} \xi_ {k} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} - \sum_ {h, i} \gamma_ {h i} s _ {h i} \xi_ {k} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} + b _ {k} \xi_ {k} \\ = \max  \left(0, - \frac {\mathbf {a} _ {k} ^ {\mathsf {T}} \left(\sum_ {k ^ {\prime} \neq k} \xi_ {k ^ {\prime}} \mathbf {a} _ {k ^ {\prime}} - \sum_ {l , i} \lambda_ {l i} u _ {l i} \mathbf {x} _ {i} - \sum_ {h , i} \gamma_ {h i} s _ {h i} \mathbf {x} _ {i}\right) + b _ {k}}{\| \mathbf {a} _ {k} \| _ {2} ^ {2}}\right). \tag {B.6} \\ \end{array}
$$
Update  $\lambda_{li}$  given others. For  $l = 1, \dots, L$  and  $i = 1, \dots, n$ ,
$$
\begin{array}{l} \lambda_ {l i} ^ {\mathrm {n e w}} = \underset {1 \geq \lambda_ {l i} \geq 0} {\operatorname {a r g m i n}} \frac {1}{2} u _ {l i} ^ {2} \lambda_ {l i} ^ {2} \mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i} + \sum_ {(l ^ {\prime}, i ^ {\prime}) \neq (l, i)} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} u _ {l i} \mathbf {x} _ {i ^ {\prime}} ^ {\intercal} \mathbf {x} _ {i} \lambda_ {l i} \\ - \sum_ {k = 1} ^ {K} \xi_ {k} u _ {l i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} \lambda_ {l i} + \sum_ {h ^ {\prime}, i ^ {\prime}} u _ {l i ^ {\prime}} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i ^ {\prime}} \lambda_ {l i} - v _ {l i} \lambda_ {l i} \\ = \mathcal {P} _ {[ 0, 1 ]} \left(\frac {u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \left(\sum_ {k = 1} ^ {K} \xi_ {k} \mathbf {a} _ {k} - \sum_ {\left(l ^ {\prime} , i ^ {\prime}\right) \neq (l , i)} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}} - \sum_ {h ^ {\prime} , i ^ {\prime}} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}}\right) + v _ {l i}}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right). \tag {B.7} \\ \end{array}
$$
Update  $\gamma_{hi}$  given others. For  $h = 1,\dots ,H$  and  $i = 1,\dots ,n$
$$
\begin{array}{l} \gamma_ {h i} ^ {\mathrm {n e w}} = \operatorname * {a r g m i n} _ {\tau_ {h i} \geq \gamma_ {h i} \geq 0} \frac {1}{2} (s _ {h i} ^ {2} \mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i} + 1) \gamma_ {h i} ^ {2} + \sum_ {(h ^ {\prime}, i ^ {\prime}) \neq (h, i)} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} s _ {h i} \mathbf {x} _ {i ^ {\prime}} ^ {\intercal} \mathbf {x} _ {i} \gamma_ {h i} \\ - \sum_ {k = 1} ^ {K} \xi_ {k} s _ {h i} \mathbf {a} _ {k} ^ {\mathsf {T}} \mathbf {x} _ {i} \gamma_ {h i} + \sum_ {l ^ {\prime}, i ^ {\prime}} s _ {h i} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\mathsf {T}} \mathbf {x} _ {i ^ {\prime}} \gamma_ {h i} - t _ {h i} \gamma_ {h i} \\ = \mathcal {P} _ {[ 0, \tau_ {h i} ]} \left(\frac {s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \left(\sum_ {k = 1} ^ {K} \xi_ {k} \mathbf {a} _ {k} - \sum_ {\left(h ^ {\prime} , i ^ {\prime}\right) \neq (h , i)} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}} - \sum_ {l ^ {\prime} , i ^ {\prime}} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}}\right) + t _ {h i}}{s _ {h i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 1}\right). \tag {B.8} \\ \end{array}
$$
The canonical CD updates in (B.6) - (B.8) for one full iteration require computation complexity of  $\mathcal{O}\big((K + nd + nL + nH)(K + nL + nH)\big)$ .
B.2 ReHLine updates
Consider the Lagrangian function  $\mathcal{L}$  in (7). Clearly,
$$
\begin{array}{l} \nabla_ {\xi_ {k}} \mathcal {L} = \mathbf {a} _ {k} ^ {\intercal} \mathbf {a} _ {k} \xi_ {k} + \sum_ {k ^ {\prime} \neq k} \xi_ {k ^ {\prime}} \mathbf {a} _ {k ^ {\prime}} ^ {\intercal} \mathbf {a} _ {k} - \sum_ {l, i} \lambda_ {l i} u _ {l i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} - \sum_ {h, i} \gamma_ {h i} s _ {h i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} + b _ {k} \\ = \sum_ {k ^ {\prime}} \xi_ {k ^ {\prime}} \mathbf {a} _ {k ^ {\prime}} ^ {\intercal} \mathbf {a} _ {k} - \sum_ {l, i} \lambda_ {l i} u _ {l i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} - \sum_ {h, i} \gamma_ {h i} s _ {h i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} + b _ {k} \\ = \mathbf {a} _ {k} ^ {\intercal} \boldsymbol {\beta} (\boldsymbol {\mu}) + b _ {k}. \\ \end{array}
$$
Similarly, we can derive that
$$
\nabla_ {\lambda_ {l i}} \mathcal {L} = - \left(u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} (\boldsymbol {\mu}) + v _ {l i}\right), \quad \nabla_ {\gamma_ {h i}} \mathcal {L} = \gamma_ {h i} - \left(s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} (\boldsymbol {\mu}) + t _ {h i}\right).
$$
Then the CD updates in (B.6) can be simplified as:
$$
\begin{array}{l} \xi_ {k} ^ {\mathrm {n e w}} = \mathcal {P} _ {[ 0, + \infty)} \left(\xi_ {k} ^ {\mathrm {o l d}} - \frac {\nabla_ {\xi_ {k}} \mathcal {L} (\xi^ {\mathrm {o l d}})}{\| \mathbf {a} _ {k} \| _ {2} ^ {2}}\right) = \max \left(0, \xi_ {k} ^ {\mathrm {o l d}} - \frac {\mathbf {a} _ {k} ^ {\intercal} \boldsymbol {\beta} ^ {\mathrm {o l d}} + b _ {k}}{\| \mathbf {a} _ {k} \| _ {2} ^ {2}}\right), \\ \boldsymbol {\beta} ^ {\text {n e w}} = \boldsymbol {\beta} ^ {\text {o l d}} + \left(\xi_ {k} ^ {\text {n e w}} - \xi_ {k} ^ {\text {o l d}}\right) \mathbf {a} _ {k}. \tag {B.9} \\ \end{array}
$$
The CD updates in (B.7) are simplified as:
$$
\begin{array}{l} \lambda_ {l i} ^ {\mathrm {n e w}} = \mathcal {P} _ {[ 0, 1 ]} \left(\lambda_ {l i} ^ {\mathrm {o l d}} - \frac {\nabla_ {\lambda_ {l i}} \mathcal {L} (\lambda^ {\mathrm {o l d}})}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right) = \max  \left(0, \min  \left(1, \lambda_ {l i} ^ {\mathrm {o l d}} + \frac {u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} ^ {\mathrm {o l d}} + v _ {l i}}{u _ {l i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2}}\right)\right), \\ \boldsymbol {\beta} ^ {\text {n e w}} = \boldsymbol {\beta} ^ {\text {o l d}} - \left(\lambda_ {l i} ^ {\text {n e w}} - \lambda_ {l i} ^ {\text {o l d}}\right) u _ {l i} \mathbf {x} _ {i}. \tag {B.10} \\ \end{array}
$$
The CD updates in (B.8) are simplified as:
$$
\gamma_ {h i} ^ {\mathrm {n e w}} = \mathcal {P} _ {[ 0, \tau_ {h i} ]} \left(\gamma_ {h i} ^ {\mathrm {o l d}} - \frac {\nabla_ {\gamma_ {h i}} \mathcal {L} (\gamma^ {\mathrm {o l d}})}{s _ {h i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 1}\right) = \max  \left(0, \min  \left(\tau_ {h i}, \gamma_ {h i} ^ {\mathrm {o l d}} + \frac {s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} ^ {\mathrm {o l d}} + t _ {h i} - \gamma_ {h i} ^ {\mathrm {o l d}}}{s _ {h i} ^ {2} \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 1}\right)\right),
$$
$$
\boldsymbol {\beta} ^ {\text {n e w}} = \boldsymbol {\beta} ^ {\text {o l d}} - \left(\gamma_ {h i} ^ {\text {n e w}} - \gamma_ {h i} ^ {\text {o l d}}\right) s _ {h i} \mathbf {x} _ {i}. \tag {B.11}
$$
The computational complexity of the new ReHLine iterations is  $\mathcal{O}\big((K + nL + nH)d\big)$ , as shown in (B.9) through (B.11). Notably, this complexity is linear with respect to both the sample size  $n$  and the dimensionality  $d$ , making ReHLine a highly efficient algorithm.
C Screening rules for ReHLine
We extend the shrinking strategy in Algorithm 3 of [12] to the ReHLine solver. To begin with, the following theorem advocates utilizing "sparse" updates to optimize the performance of ReHLine.
Theorem 4. Let  $\{\pmb {\xi}^*,\pmb {\Lambda}^*,\pmb {\Gamma}^*,\pmb {\beta}^*\}$  be the convergent point of  $\{\pmb {\xi}^{(q)},\pmb {\Lambda}^{(q)},\pmb {\Gamma}^{(q)},\pmb {\beta}^{(q)}\}$  based on (B.9)-(B.11), then
(A) For  $\pmb{\xi}$  updates, let  $g_{k}^{*} = \mathbf{a}_{k}^{\intercal}\boldsymbol{\beta}^{*} + b_{k},$
$$
- i f g _ {k} ^ {*} > 0, t h e n \exists q _ {0} s u c h t h a t \forall q \geq q _ {0}, \xi_ {k} ^ {(q)} = \xi_ {k} ^ {*} = 0.
$$
(B) For  $\pmb{\Lambda}$  updates,  $g_{li}^{*} = u_{li}\mathbf{x}_{i}^{\top}\boldsymbol{\beta}^{*} + v_{li},$
- if  $g_{li}^{*} < 0$ , then  $\exists q_{0}$  such that  $\forall q \geq q_{0}$ ,  $\lambda_{li}^{(q)} = \lambda_{li}^{*} = 0$ ;
- if  $g_{li}^{*} > 0$ , then  $\exists q_{0}$  such that  $\forall q \geq q_{0}$ ,  $\lambda_{li}^{(q)} = \lambda_{li}^{*} = 1$ .
(C) For  $\Gamma$  updates, denote  $g_{hi}^{*} = s_{hi}\mathbf{x}_{i}^{\intercal}\boldsymbol {\beta}^{*} + t_{hi},$
- if  $\gamma_{hi}^{*} = 0$  and  $g_{hi}^{*} < 0$ , then  $\exists q_0$  such that  $\forall q \geq q_0$ ,  $\gamma_{hi}^{(q)} = 0$ ;
- if  $\gamma_{hi}^{*} = \tau_{hi}$  and  $g_{hi}^{*} > \tau_{hi}$ , then  $\exists q_0$  such that  $\forall q\geq q_0$ ,  $\gamma_{hi}^{(q)} = \tau_{hi}$ .
Building upon the insights from Theorem 4, we have formulated Algorithm 2 that involves shrinking certain "boundary" variables during the implementation of ReHLine.
Algorithm 2: ReHLine solver minimizing (1) with shrinking.
Input  $\mathbf{:X}\in \mathbb{R}^{n\times d},\mathbf{U},\mathbf{V}\in \mathbb{R}^{L\times n};\mathbf{S},\mathbf{T},\tau \in \mathbb{R}^{H\times n},\mathbf{A}\in \mathbb{R}^{K\times d},\mathbf{b}\in \mathbb{R}^{K},\mathrm{tol}\varepsilon >0$    
1 Initialize  $\pmb {\xi}\geq \mathbf{0},\mathbf{E}\geq \pmb {\Lambda}\geq \mathbf{0},\pmb {\tau}\geq \pmb{\Gamma}\geq \mathbf{0};$    
2 Compute  $\beta = \sum_{k = 1}^{K}\xi_{k}\mathbf{a}_{k} - \sum_{i = 1}^{n}\mathbf{x}_{i}\left(\sum_{l = 1}^{L}\lambda_{li}u_{li} + \sum_{h = 1}^{H}\gamma_{hi}s_{hi}\right);$    
3 Compute and store  $\mathbf{r} = (r_i)\in \mathbb{R}^n,r_i = \| \mathbf{x}_i\| _2^2$  ,and  $\mathbf{p} = (p_k)\in \mathbb{R}^K,p_k = \| \mathbf{a}_k\| _2^2;$    
4 Initialize  $\bar{M}_s = \infty ,\bar{m}_s = -\infty ,s = \{\xi ,\lambda ,\gamma \} ,\mathcal{K} = [K],\mathcal{L} = [L]\otimes [n],\mathcal{H} = [H]\otimes [n];$    
5 while maximum iteration do   
6 Initialize  $M_s = \infty ,m_s = -\infty ,s = \{\xi ,\lambda ,\gamma \}$  . // CD updates for  $\pmb{\xi}$    
7 for  $k\in \mathrm{PERM}(\mathcal{K})$  do   
8  $g_{k} = \mathbf{a}_{k}^{\intercal}\beta +b_{k};$  (is_shrink,  $\bar{g}_k)\gets$  PGfeasible(  $\xi_k,g_k,\bar{M}_\xi)$  . if is_shrink then  $\begin{array}{rl}{\bigtriangledown}&{\boldsymbol{\mathcal{K}}\leftarrow\boldsymbol{\mathcal{K}}\setminus\{k\};}\\ &{\boldsymbol{\mathcal{M}}_k\leftarrow\max\{M_\xi,\bar{g}_k\};}\\ &{\varepsilon^*=\max(-\xi_k,-p_k^{-1}g_k);}\\ &{\beta\leftarrow\beta+\varepsilon^*\alpha_k;}\end{array}$    
11   
12   
13   
// CD updates for A   
for  $(l,i)\in \mathrm{PERM}(\mathcal{L})$  do  
gl  $= -(u_{li}\mathbf{x}_i^\top \beta +v_{li})$  . (is_shrink,  $\bar{g}_l)\gets$  PG_ReLU(λli, gli, Mλ, mλ); if is_shrink then L L {l,i}; continue;   
 $M_{\lambda}\gets \max \{M_{\lambda},\bar{g}_{li}\} ;$  mλ←min{mλ, gli};   
ε* = max(-λli, min(1 - λli, uLI-2ri-1gl)); λli←λli+ε*; β←β-ε*uIxxi;   
// CD updates for F   
for  $(h,i)\in \mathrm{PERM}(\mathcal{H})$  do  
ghi  $= \gamma_{hi} - (s_{hi}\mathbf{x}_i^\top \beta +t_{hi})$  . (is_shrink, ghi) PPG_ReHU(γhi, ghi, Mγ, mγ); if is_shrink then H H {h,i}; continue;   
 $M_{\gamma}\gets \max \{M_{\gamma},g_{hi}\} ;$  mγ←min{mγ, ghi};   
ε* = max(-γhi, min(τhi - γhi, (s²hi ri + 1)-1(ghi - γhi));   
γhi←γhi+ε*; β←β-ε*sHi xi;   
if  $M_s - m_s\leq \varepsilon$  and  $|M_s|\leq \varepsilon$  and  $|m_s|\leq \varepsilon$  s={ξ,λ,γ} then if  $|\mathcal{K}| + |\mathcal{L}| + |\mathcal{H}| = K + n(L + H)$  then break; else Take all shrunken variables back, i.e., K=[K], L=[L]⊗[n], H=[H]⊗[n]; M-s<∞,m-s<-∞,s={ξ,λ,γ};   
if  $M_s\leq 0$  then  $M_s\gets \infty$  else  $M_s\gets M_s,s = \{\xi ,\lambda ,\gamma \}$  . if  $m_s\geq 0$  then  $m_s\gets \infty$  else  $m_s,s = \{\xi ,\lambda ,\gamma \}$
Output:  $\xi, \Lambda, \Gamma, \beta$
Algorithm 3: Projected gradients in ReHLine.
1 Function PGfeasible  $(\xi_{k},g_{k},\bar{M})$  ..   
2  $\bar{g}_k\gets g_k$  is_shrink  $\leftarrow$  False;   
3 if  $\xi_{k} = 0$  and  $g_{k} > \bar{M}$  then   
4 is_shrink  $\leftarrow$  True   
5 if  $\xi_{k} = 0$  and  $g_{k}\geq 0$  then   
6  $\bar{g}_k = 0$  Output:is_shrink,  $\bar{g}_k$    
7 Function PGReLU  $(\lambda_{li},g_{li},\bar{M},\bar{m})$  .   
8  $\bar{g}_{li}\gets g_{li}$  is_shrink  $\leftarrow$  False;   
9 if  $(\lambda_{li} = 0$  and  $g_{li} > \bar{M})$  or  $(\lambda_{li} = 1$  and  $g_{li} <   \bar{m})$  then   
10 is_shrink  $\leftarrow$  True   
11 if  $(\lambda_{li} = 0$  and  $g_{li}\geq 0)$  or  $(\lambda_{li} = 1$  and  $g_{li}\leq 0)$  then   
12  $\bar{g}_{li} = 0$  Output:is_shrink,  $\bar{g}_{li}$    
13 Function PG_ReHU(  $\gamma_{hi},g_{hi},\tau_{hi},\bar{M},\bar{m})$  .   
14  $\bar{g}_{hi}\gets g_{hi}$  is_shrink  $\leftarrow$  False;   
15 if  $(\gamma_{hi} = 0$  and  $g_{hi} > \bar{M})$  or  $(\gamma_{hi} = \tau_{hi}$  and  $g_{hi} <   \bar{m})$  then   
16 is_shrink  $\leftarrow$  True   
17 if  $(\gamma_{hi} = 0$  and  $g_{hi}\geq 0)$  or  $(\gamma_{hi} = \tau_{hi}$  and  $g_{hi}\leq 0)$  then   
18  $\bar{g}_{hi} = 0$  Output:is_shrink,  $\bar{g}_{hi}$
D Technical proofs
Lemma 1. Suppose  $f(z)$  is a nonnegative convex PLQ with knots and extreme points  $(t_k)_{k=1}^K$ . Then, there exists  $k_0 \in \{1, \dots, K\}$ , such that  $f(z) \geq f(t_{k_0})$  for any  $z \in \mathbb{R}$ .
Proof. We prove it by contradiction. Suppose that for there exists  $z_0$  such that  $f(z_0) < f(t_k)$  for all  $k \in \{1, \dots, K\}$ , then the minimum can only be obtained outside  $[t_1, t_K]$ , that is
$$
\exists z _ {0} <   t _ {1}, f (z _ {0}) <   f (t _ {k}), \quad \text {o r} \quad \exists z _ {0} > t _ {K}, f (z _ {0}) <   f (t _ {k}) \quad \text {f o r a n y} k \in \{1, \dots , K \}.
$$
For example, suppose that  $z_0 \in (t_K, \infty)$ ,  $f(z_0) < f(t_k)$  for any  $k \in \{1, \dots, K\}$ . Note that the quadratic function in  $(z_K, \infty)$  can be written as  $f(z) = \frac{1}{2} f_{+}^{\prime \prime}(t_K)(z - t_K)^2 + f_{+}^{\prime}(t_K)(z - t_K) + f(t_K)$ . Based on the fact that  $f(z_0) < f(t_K)$  and the convexity of  $f(z)$ , we have  $f_{+}^{\prime}(t_K) < 0$ . Then, since  $(t_k)_{k=1}^{K}$  is the set of all knots and extreme points, we have  $f'(z) < 0$  for all  $z > t_K$ , and  $\lim_{z \to \infty} f'(z) = \lim_{z \to \infty} (f_{+}^{\prime \prime}(t_K)(z - t_K) + f_{+}^{\prime}(t_K)) \leq 0$ . This yields that  $f_{+}^{\prime \prime}(t_K) = 0$ , alternatively,  $f(z)$  is a linear function in  $(t_K, \infty)$  as  $f(z) = f_{+}^{\prime}(t_K)(z - t_K) + f(t_K)$ . Hence, when  $z \to \infty$ ,  $f(z) \to -\infty$ , which contradicts with the fact that  $f(z)$  is a nonnegative function. The same argument goes for the case when  $z_0 \in (-\infty, t_1)$ . This completes the proof.
Proof. Note that  $c\mathrm{ReLU}(z) = c\max (z,0) = \max (cz,0)$ , and
$$
e \mathrm {R e H U} _ {\tau} (z) = \left\{ \begin{array}{l l} 0, & \sqrt {c c} z \leq 0 \\ (\sqrt {c c}) ^ {2} / 2, & 0 <   \sqrt {c c} z \leq \sqrt {c c} \tau \\ \sqrt {c c} (\sqrt {c c} z - \sqrt {c c} \tau / 2), & \sqrt {c c} z > \sqrt {c c} \tau \end{array} = \mathrm {R e H U} _ {\sqrt {c c}} (\sqrt {c c}), \right.
$$
for  $c > 0$ . Then,
$$
\begin{array}{l} c L (p z + q) = \sum_ {l = 1} ^ {L} c \operatorname {R e L U} \left(u _ {l} (p z + q) + v _ {l}\right) + \sum_ {h = 1} ^ {H} c \operatorname {R e H U} _ {\tau_ {h}} \left(s _ {h} (p z + q) + t _ {h}\right) \\ = \sum_ {l = 1} ^ {L} \operatorname {R e L U} \left(c u _ {l} (p z + q) + c v _ {l}\right) + \sum_ {h = 1} ^ {H} \operatorname {R e H U} _ {\sqrt {c} \tau_ {h}} \left(\sqrt {c} s _ {h} (p z + q) + \sqrt {c} t _ {h}\right) \\ = \sum_ {l = 1} ^ {L} \mathrm {R e L U} (u _ {l} ^ {\prime} z + v _ {l} ^ {\prime}) + \sum_ {h = 1} ^ {H} \mathrm {R e H U} _ {\tau_ {h} ^ {\prime}} (s _ {h} ^ {\prime} z + t _ {h} ^ {\prime}). \\ \end{array}
$$
Proof. The sufficiency  $(\Longleftrightarrow)$  directly follows from the definitions of ReLU and ReHU. Next, the necessity  $(\Rightarrow)$  of the theorem is established using induction. Without loss of generality, for a nonnegative convex PLQ function  $f(z)$ , we assume that (A1)  $f_{+}^{\prime}(t_{k})f_{-}^{\prime}(t_{k + 1})\geq 0$ , otherwise we can add the extreme point between  $(t_k,t_{k + 1}]$  into the set of knots; (A2)  $\min_{k = 1,\dots ,K}f(t_k) = 0$  otherwise we can add ReLU  $\left(\min_{k = 1,\dots ,K}f(t_k)\right)$  to the final decomposition (Lemma 1 shows that the minimum of  $f(z)$  can only be obtained within the knots). Next, we conduct the mathematical induction.
$\underline{K} = 1$ . Note that  $f(z)$  can be rewritten as:
$$
f (z) = \left\{ \begin{array}{l l} q _ {1} ^ {-} (z) = \frac {f _ {-} ^ {\prime \prime} (t _ {1})}{2} (z - t _ {1}) ^ {2} + f _ {-} ^ {\prime} (t _ {1}) (z - t _ {1}), & z \leq t _ {1}, \\ q _ {1} ^ {+} (z) = \frac {f _ {+} ^ {\prime \prime} (t _ {1})}{2} (z - t _ {1}) ^ {2} + f _ {+} ^ {\prime} (t _ {1}) (z - t _ {1}), & z > t _ {1}, \end{array} \right.
$$
where  $q_1^- (z)$  and  $q_1^+ (z)$  are left and right parts of  $f(z)$  separated at  $z = t_1$ . According to Lemma 1 and Assumption A2, we have  $f(t_{1}) = 0$  is the minimum of  $f(z)$ . Combined with Assumption A1, we have  $f_{-}^{\prime}(t)\leq 0$  for  $t < t_1$ , and  $f_{+}^{\prime}(t)\geq 0$  for  $t > t_1$ . Then,  $f(z)$  can be decomposed as:
$$
f (z) = \left\{ \begin{array}{l l} q _ {1} ^ {-} (z) = \mathrm {R e H U} _ {\infty} (- \sqrt {f _ {-} ^ {\prime \prime} (t _ {1})} (z - t _ {1})) + \mathrm {R e L U} (f _ {-} ^ {\prime} (t _ {1}) (z - t _ {1})), & z \leq t _ {1}, \\ q _ {1} ^ {+} (z) = \mathrm {R e H U} _ {\infty} (\sqrt {f _ {+} ^ {\prime \prime} (t _ {1})} (z - t _ {1})) + \mathrm {R e L U} (f _ {+} ^ {\prime} (t _ {1}) (z - t _ {1})), & z > t _ {1}. \end{array} \right.
$$
Hence,  $f(z) = \mathrm{ReHU}_{\infty}(-\sqrt{f_{-}^{\prime\prime}(t_1)} (z - t_1)) + \mathrm{ReLU}(f_{-}^{\prime}(t_1)(z - t_1)) + \mathrm{ReHU}_{\infty}(\sqrt{f_{+}^{\prime\prime}(t_1)} (z - t_1)) + \mathrm{ReLU}(f_{+}^{\prime}(t_1)(z - t_1))$ . The necessity of the theorem for  $K = 1$  then follows.
$K = K_{0} + 1$ . Assume that a convex PLQ  $f \geq 0$  with  $K(K \leq K_{0})$  knots is ReLU-ReHU composite, now we turn to prove the statement still holds for  $f(z)$  with  $K_{0} + 1$ .
- CASE 1. Suppose the minimum  $f(t^{*}) = 0$  is obtained at  $\{t_2, \dots, t_{K_0}\}$  (not in the boundary). In this case, we can decompose  $f(z)$  as  $f(z) = f_l(z) + f_r(z)$  where  $f_l(z)$  and  $f_r(z)$  are defined as:
$$
f _ {l} (z) = \left\{ \begin{array}{l l} f (z), & z \leq t ^ {*}, \\ 0, & z > t ^ {*}. \end{array} \right. \qquad \qquad f _ {r} (z) = \left\{ \begin{array}{l l} 0, & z \leq t ^ {*}, \\ f (z), & z > t ^ {*}. \end{array} \right.
$$
Both  $f_{l}(z)$  and  $f_{r}(z)$  are nonnegative convex PLQs with  $K \leq K_{0}$  knots, thus can be represented as ReHUs and RLUs.
- CASE 2. Suppose the minimum  $f(t^{*}) = 0$  is obtained when  $t^{*} = t_{1}$  or  $t^{*} = t_{K_{0} + 1}$ . For example, when  $t^{*} = t_{1}$ , we conduct the same decomposition  $f(z) = f_{l}(z) + f_{r}(z)$  at  $z = t_{1}$ , where  $f_{l}(z)$  is a nonnegative convex PLQ with one knot, which is ReHU-ReLU composite. For  $f_{r}(z)$ , we further decompose it as: