SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {r} (z) = \mathrm {R e L U} (f _ {+} ^ {\prime} (t _ {1}) (z - t _ {1})) + \mathrm {R e H U} _ {\tau} \big (\sqrt {f _ {+} ^ {\prime \prime} (t _ {1})} (z - t _ {1}) \big) + R (z),
$$
where  $\tau = \sqrt{f_{+}^{\prime\prime}(t_1)} (t_2 - t_1)$ , and  $R(z)$  (with at most  $K_{0}$  knots) is defined as:
$$
R (z) = \left\{ \begin{array}{l l} 0, & z \leq t _ {2}, \\ f (z) - f _ {+} ^ {\prime} (t _ {1}) (z - t _ {1}) - f _ {+} ^ {\prime \prime} (t _ {1}) (t _ {2} - t _ {1}) (z - (t _ {1} + t _ {2}) / 2), & z > t _ {2}. \end{array} \right.
$$
Now, we show that  $R(z)$  is nonnegative convex. Note that  $f(z)$  is convex, then for  $z \geq t_2$ ,
$$
\begin{array}{l} f (z) \geq f (t _ {2}) + f _ {-} ^ {\prime} (t _ {2}) (z - t _ {2}) = f (t _ {2}) + \left(f _ {+} ^ {\prime \prime} (t _ {1}) \left(t _ {2} - t _ {1}\right) + f _ {+} ^ {\prime} (t _ {1})\right) (z - t _ {2}) \\ = f _ {+} ^ {\prime} \left(t _ {1}\right) \left(z - t _ {1}\right) + f _ {+} ^ {\prime \prime} \left(t _ {1}\right) \left(t _ {2} - t _ {1}\right) \left(z - \left(t _ {1} + t _ {2}\right) / 2\right) \\ = \operatorname {R e L U} \left(f _ {+} ^ {\prime} \left(t _ {1}\right) \left(z - t _ {1}\right)\right) + \operatorname {R e H U} _ {\tau} \left(\sqrt {f _ {+} ^ {\prime \prime} \left(t _ {1}\right)} \left(z - t _ {1}\right)\right), \\ \end{array}
$$
where the first equality follows from the fact that  $f(z)$  is a quadratic function in  $[t_1, t_2]$ , and thus  $f_{-}^{\prime}(t_2) = f_{+}^{\prime}(t_1)(t_2 - t_1)$ , and the second equality follows from  $f(t_2) = \frac{f_{-}^{\prime\prime}(t_1)}{2} (t_2 - t_1)^2 + f_{-}^{\prime}(t_1)(t_2 - t_1)$ .
Thus,  $R(z)$  is a nonnegative convex PLQ with  $K_{0}$  knots  $\{t_{2},\dots ,t_{K_{0} + 1}\}$ , which can be decomposed as ReHUs and RLUs. The same argument goes for the case when  $t^{*} = t_{K_{0} + 1}$ .
This completes the proof, and the desirable results follow from the induction.
Proof. The lemma follows from the fact that:
$$
\left\| \boldsymbol {\beta} \right\| _ {1} = \sum_ {j = 1} ^ {d} \left| \beta_ {j} \right| = \sum_ {j = 1} ^ {d} \left(\operatorname {R e L U} \left(\mathbf {e} _ {j} ^ {\intercal} \boldsymbol {\beta}\right) + \operatorname {R e L U} \left(- \mathbf {e} _ {j} ^ {\intercal} \boldsymbol {\beta}\right)\right),
$$
where  $\mathbf{e}_j = (0,\dots ,0,1,0\dots)$  is the  $d$ -tuple with all components equal to 0, except the  $j$ th.
Proof. Note that the augmented Lagrangian of the primal problem (6) is:
$$
\begin{array}{l} \mathcal {L} _ {P} = \sum_ {i = 1} ^ {n} \sum_ {l = 1} ^ {L} \pi_ {l i} + \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \frac {1}{2} \theta_ {h i} ^ {2} + \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \tau_ {h i} \sigma_ {h i} + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2} - \sum_ {k = 1} ^ {K} \xi_ {k} (\mathbf {a} _ {k} ^ {\intercal} \boldsymbol {\beta} + b _ {k}) \\ - \sum_ {i = 1} ^ {n} \sum_ {l = 1} ^ {L} \lambda_ {l i} \left(\pi_ {l i} - u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} - v _ {l i}\right) - \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \gamma_ {h i} \left(\theta_ {h i} + \sigma_ {h i} - s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} - t _ {h i}\right) \\ - \sum_ {i = 1} ^ {n} \sum_ {l = 1} ^ {L} \delta_ {l i} \pi_ {l i} - \sum_ {i = 1} ^ {n} \sum_ {h = 1} ^ {H} \psi_ {h i} \sigma_ {h i} \\ = \mathbf {e} ^ {\intercal} \boldsymbol {\Pi} \mathbf {e} + \frac {1}{2} \| \boldsymbol {\Theta} \| _ {F} ^ {2} + \operatorname {T r} (\boldsymbol {\tau} \boldsymbol {\Sigma} ^ {\intercal}) + \frac {1}{2} \| \boldsymbol {\beta} \| _ {2} ^ {2} - \boldsymbol {\xi} ^ {\intercal} (\mathbf {A} \boldsymbol {\beta} + \mathbf {b}) \\ - \operatorname {T r} \left(\boldsymbol {\Lambda} (\boldsymbol {\Pi} - \mathbf {U} \operatorname {d i a g} (\mathbf {X} \boldsymbol {\beta}) - \mathbf {V}) ^ {\intercal}\right) - \operatorname {T r} \left(\boldsymbol {\Gamma} (\boldsymbol {\Theta} + \boldsymbol {\Sigma} - \mathbf {S} \operatorname {d i a g} (\mathbf {X} \boldsymbol {\beta}) - \mathbf {T}) ^ {\intercal}\right) \\ - \operatorname {T r} \left(\boldsymbol {\Delta} \boldsymbol {\Pi} ^ {\mathsf {T}}\right) - \operatorname {T r} \left(\boldsymbol {\Psi} \boldsymbol {\Sigma} ^ {\mathsf {T}}\right), \tag {D.12} \\ \end{array}
$$
where  $\pmb {\xi} = (\xi_{k})\in \mathbb{R}^{K}$ $\pmb {\Lambda} = (\lambda_{li})\in \mathbb{R}^{L\times n}$ $\Gamma = (\gamma_{hi})\in \mathbb{R}^{H\times n}$ $\Delta = (\delta_{li})\in \mathbb{R}^{L\times n}$  and  $\Psi = (\psi_{hi})\in \mathbb{R}^{H\times n}$  are dual variables. The KKT conditions for the augmented Lagrangian are:
$$
\frac {\partial \mathcal {L} _ {P}}{\partial \boldsymbol {\beta}} = \boldsymbol {\beta} - \sum_ {k = 1} ^ {K} \xi_ {k} \mathbf {a} _ {k} + \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \left(\sum_ {l = 1} ^ {L} \lambda_ {l i} u _ {l i} + \sum_ {h = 1} ^ {H} \gamma_ {h i} s _ {h i}\right) = \mathbf {0},
$$
$$
\frac {\partial \mathcal {L} _ {P}}{\partial \pi_ {l i}} = 1 - \lambda_ {l i} - \delta_ {l i} = 0, \quad \frac {\partial \mathcal {L} _ {P}}{\partial \theta_ {h i}} = \theta_ {h i} - \gamma_ {h i} = 0, \quad \frac {\partial \mathcal {L} _ {P}}{\partial \sigma_ {h i}} = \tau_ {h i} - \gamma_ {h i} - \psi_ {h i} = 0,
$$
for all  $(i,l,h)$ . Then, by substituting the KKT conditions into the primal Lagrangian, the (negative) dual objective function is derived as:
$$
\begin{array}{l} \mathcal {L} _ {D} = \frac {1}{2} \Big (\sum_ {k = 1} ^ {K} \sum_ {k ^ {\prime} = 1} ^ {K} \xi_ {k} \xi_ {k ^ {\prime}} \mathbf {a} _ {k} ^ {\intercal} \mathbf {a} _ {k ^ {\prime}} + \sum_ {l, i} \sum_ {l ^ {\prime}, i ^ {\prime}} \lambda_ {l i} \lambda_ {l ^ {\prime} i ^ {\prime}} u _ {l i} u _ {l ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i ^ {\prime}} + \sum_ {h, i} \sum_ {h ^ {\prime}, i ^ {\prime}} \gamma_ {h i} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h i} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\intercal} \mathbf {x} _ {i ^ {\prime}} \Big) \\ - \sum_ {k = 1} ^ {K} \sum_ {l, i} \xi_ {k} \lambda_ {l i} u _ {l i} \mathbf {a} _ {k} ^ {\mathsf {T}} \mathbf {x} _ {i} - \sum_ {k = 1} ^ {K} \sum_ {h, i} \xi_ {k} \gamma_ {h i} s _ {h i} \mathbf {a} _ {k} ^ {\mathsf {T}} \mathbf {x} _ {i} + \sum_ {l, i} \sum_ {h ^ {\prime}, i ^ {\prime}} \lambda_ {l i} u _ {l i} \gamma_ {h ^ {\prime} i ^ {\prime}} s _ {h ^ {\prime} i ^ {\prime}} \mathbf {x} _ {i} ^ {\mathsf {T}} \mathbf {x} _ {i ^ {\prime}} \\ + \frac {1}{2} \sum_ {h, i} \gamma_ {h i} ^ {2} + \sum_ {k = 1} ^ {K} \xi_ {k} b _ {k} - \sum_ {l, i} \lambda_ {l i} v _ {l i} - \sum_ {h, i} \gamma_ {h i} t _ {h i} \\ = \frac {1}{2} \boldsymbol {\xi} ^ {\mathsf {T}} \mathbf {A} \mathbf {A} ^ {\mathsf {T}} \boldsymbol {\xi} + \frac {1}{2} \operatorname {v e c} (\boldsymbol {\Lambda}) ^ {\mathsf {T}} \bar {\mathbf {U}} _ {(3)} ^ {\mathsf {T}} \bar {\mathbf {U}} _ {(3)} \operatorname {v e c} (\boldsymbol {\Lambda}) + \frac {1}{2} \operatorname {v e c} (\boldsymbol {\Gamma}) ^ {\mathsf {T}} \left(\bar {\mathbf {S}} _ {(3)} ^ {\mathsf {T}} \bar {\mathbf {S}} _ {(3)} + \mathbf {I}\right) \operatorname {v e c} (\boldsymbol {\Gamma}) \\ - \xi^ {\intercal} \mathbf {A} \bar {\mathbf {U}} _ {(3)} \operatorname {v e c} (\boldsymbol {\Lambda}) - \xi^ {\intercal} \mathbf {A} \bar {\mathbf {S}} _ {(3)} \operatorname {v e c} (\boldsymbol {\Gamma}) + \operatorname {v e c} (\boldsymbol {\Lambda}) ^ {\intercal} \bar {\mathbf {U}} _ {(3)} ^ {\intercal} \bar {\mathbf {S}} _ {(3)} \operatorname {v e c} (\boldsymbol {\Gamma}) \\ + \boldsymbol {\xi} ^ {\mathsf {T}} \mathbf {b} - \operatorname {T r} (\boldsymbol {\Lambda} \mathbf {V} ^ {\mathsf {T}}) - \operatorname {T r} (\boldsymbol {\Gamma} \mathbf {T} ^ {\mathsf {T}}), \\ \end{array}
$$
where  $\bar{\mathbf{U}}_{(3)}$  and  $\bar{\mathbf{S}}_{(3)}$  are the mode-3 unfolding of the tensors  $\bar{\mathbf{U}}\in \mathbb{R}^{L\times n\times d}$  and  $\bar{\mathbf{S}}\in \mathbb{R}^{H\times n\times d}$ , and  $\mathbf{I}$  is an identity matrix.
Proof. The treatment for the proof is based on a result of coordinate descent on boxed constrained convex optimization [27]. To proceed, denote  $\pmb{\mu} = (\pmb{\xi}^{\mathsf{T}},\mathrm{vec}(\pmb{\Lambda})^{\mathsf{T}},\mathrm{vec}(\pmb{\Gamma})^{\mathsf{T}})^{\mathsf{T}}$  as the collection of all dual variables, then the dual objective can be rewritten as:
$$
\mathcal {L} _ {D} (\boldsymbol {\mu}) = g (\mathbf {E} \boldsymbol {\mu}) + \mathbf {c} ^ {\intercal} \boldsymbol {\mu},
$$
where  $g(\mathbf{z}) = \frac{1}{2}\| \mathbf{z}\| _2^2$ , and  $\mathbf{E}$  and  $\mathbf{c}$  are defined as:
$$
\mathbf {E} = \left[ \begin{array}{c c c} \mathbf {A} ^ {\intercal} & - \bar {\mathbf {U}} _ {(3)} & - \bar {\mathbf {S}} _ {(3)} \\ \mathbf {0} & \mathbf {0} & \mathbf {I} _ {H \times n} \end{array} \right] \in \mathbb {R} ^ {(d + H n) \times (K + L n + H n)},
$$
$$
\mathbf {c} = \left(\mathbf {b} ^ {\intercal}, - \operatorname {v e c} (\mathbf {V}) ^ {\intercal}, \operatorname {v e c} (\mathbf {T}) ^ {\intercal}\right) ^ {\intercal} \in \mathbb {R} ^ {K + L n + H n}.
$$
Now, we tend to verify assumptions of Theorem 2.1 in [27]. Specifically, (a) the dual problem is a convex box-constrained QP, thus the set of optimal solutions is nonempty; (b)  $g(\mathbf{z}) = \frac{1}{2}\| \mathbf{z}\| _2^2$  is a strictly convex twice continuously differentiable function; (c)  $\nabla^2 g(\mathbf{z}) = \mathbf{I}_{d + Hn}$  is always positive definite; (d)  $\mathbf{E}$  has no zero column based on its definition. The desirable result then follows from Theorem 2.1 of [27].
Proof of Theorem 4
Proof. Note that the primal problem (6) is strictly convex, thus it has the unique minimizer  $\beta^{*}$ . Therefore, the sequence  $\{\beta^{(q)}\}_{q=1}^{\infty}$  provided by the proposed Algorithm 1 converges to the unique minimizer  $\beta^{*}$ . Next, we check the gradients of each coordinate.
- For  $\xi$ -coordinates, with  $\xi_k \geq 0$
$$
\begin{array}{l} g _ {k} ^ {\mathrm {o l d}} := \nabla_ {\xi_ {k}} \mathcal {L} _ {D} (\boldsymbol {\mu} ^ {\mathrm {o l d}}) = \mathbf {a} _ {k} ^ {\intercal} \mathbf {a} _ {k} \xi_ {k} ^ {\mathrm {o l d}} + \sum_ {k ^ {\prime} \neq k} \xi_ {k ^ {\prime}} ^ {\mathrm {o l d}} \mathbf {a} _ {k ^ {\prime}} ^ {\intercal} \mathbf {a} _ {k} - \sum_ {l, i} \lambda_ {l i} ^ {\mathrm {o l d}} u _ {l i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} - \sum_ {h, i} \gamma_ {h i} ^ {\mathrm {o l d}} s _ {h i} \mathbf {a} _ {k} ^ {\intercal} \mathbf {x} _ {i} + b _ {k} \\ = \mathbf {a} _ {k} ^ {\intercal} \left(\sum_ {k = 1} ^ {K} \xi_ {k} ^ {\text {o l d}} \mathbf {a} _ {k} - \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \left(\sum_ {l = 1} ^ {L} \lambda_ {l i} ^ {\text {o l d}} u _ {l i} + \sum_ {h = 1} ^ {H} \gamma_ {h i} ^ {\text {o l d}} s _ {h i}\right)\right) + b _ {k} = \mathbf {a} _ {k} ^ {\intercal} \boldsymbol {\beta} ^ {\text {o l d}} + b _ {k}. \\ \end{array}
$$
Suppose  $g_{k}^{(q)}$  is the gradient of  $\xi_{k}$  based on  $\pmb{\mu}^{(q)}$ , then  $\lim_{q\to \infty}g_k^{(q)} = \mathbf{a}_k^\intercal \pmb{\beta}^* + b_k =: g_k^*$ . Therefore, if  $g_{k}^{*} > 0$ , then there exists  $q_0$ , for all  $q\geq q_0$ , we have  $g_{k}^{(q)} > 0$ , which implies that  $\xi_{k}^{(q + 1)} = 0$ .
- In the same manner, for  $\Lambda$ -coordinates, with  $1 \geq \lambda_{li} \geq 0$
$$
g _ {l i} ^ {\text {o l d}} := \nabla_ {\lambda_ {l i}} \mathcal {L} _ {D} (\boldsymbol {\mu} ^ {\text {o l d}}) = - u _ {l i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} ^ {\text {o l d}} - v _ {l i}.
$$
Then,  $\lim_{q\to \infty}g_{li}^{(q)} = -u_{li}\mathbf{x}_i^\intercal\boldsymbol{\beta}^* - v_{li} =: g_{li}^*$ ,
- if  $g_{li}^{*} < 0$ , then  $\exists q_{0}$  such that  $\forall q \geq q_{0}, \lambda_{li}^{(q + 1)} = \lambda_{li}^{*} = 0$ ;
- if  $g_{li}^{*} > 0$ , then  $\exists q_{0}$  such that  $\forall q \geq q_{0}, \lambda_{li}^{(q + 1)} = \lambda_{li}^{*} = 1$ .
- For  $\Gamma$ -coordinates, with  $\tau_{hi} \geq \gamma_{hi} \geq 0$ , thus
$$
g _ {h i} ^ {\text {o l d}} := \nabla_ {\gamma_ {h i}} \mathcal {L} _ {D} (\boldsymbol {\mu} ^ {\text {o l d}}) = \gamma_ {h i} ^ {\text {o l d}} - s _ {h i} \mathbf {x} _ {i} ^ {\intercal} \boldsymbol {\beta} ^ {\text {o l d}} - t _ {h i}.
$$
Then,  $\lim_{q\to \infty}g_{hi}^{(q)} = \gamma_{hi}^{*} - u_{li}\mathbf{x}_{i}^{\intercal}\boldsymbol{\beta}^{*} - v_{li} =: g_{hi}^{*}$ , which implies that
- if  $\gamma_{hi}^{*} = 0$  and  $g_{hi}^{*} > 0$ , then  $\exists q_0$  such that  $\forall q \geq q_0, \gamma_{hi}^{(q)} = 0$ ;
- if  $\gamma_{hi}^{*} = \tau_{hi}$  and  $g_{hi}^{*} < 0$ , then  $\exists q_0$  such that  $\forall q \geq q_0, \gamma_{hi}^{(q)} = \tau_{hi}$ .