SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem 2. Let  $\pmb{f}^*$  be any min-cost solution. Applying both Lemma 6 and 7 we see it must be the case that  $\pmb{f}^*(\pmb{y}) = \pmb{P}_S \pmb{f}^*(\pmb{P}_S \pmb{y})$  for all  $\pmb{y} \in \mathbb{R}^d$ , since otherwise the representation cost of  $\pmb{f}^*$  could be reduced.
Proof of Corollary 1. Suppose  $\mathcal{S}$  is one-dimensional, i.e.,  $\mathcal{S} = \operatorname{span}\{\pmb{u}\}$  for some unit vector  $\pmb{u} \in \mathbb{R}^d$ . Theorem 2 shows we can express any min-cost solution  $\pmb{f}^*$  as
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \sum_ {k = 1} ^ {K} a _ {k} \boldsymbol {u} \left[ s _ {k} \boldsymbol {u} ^ {\top} \boldsymbol {y} + b _ {k} \right] _ {+} + v \boldsymbol {u} \boldsymbol {u} ^ {\top} \boldsymbol {y} + c \boldsymbol {u} = \boldsymbol {u} \phi \left(\boldsymbol {u} ^ {\top} \boldsymbol {y}\right)
$$
$$
\phi (t) = \sum_ {k} a _ {k} [ s _ {k} t + b _ {k} ] _ {+} + v t + c
$$
is such that  $R(\pmb{f}^{*}) = R(\phi)$ . Therefore, minimizing the representation cost subject to norm-ball interpolation constraints is reduces to a univariate problem:
$$
\min  _ {\phi} R (\phi) \text {s . t .} \phi ((c _ {n} - \rho , c _ {n} + \rho)) = c _ {n}
$$
where  $c_{n} = \pmb{u}^{\top}\pmb{x}_{n}$ . The minimizing  $\phi^{*}$  is unique and coincides with the 1-D denoiser  $f_{1D}^{*}$  in (17) with  $x_{n} = c_{n}$  and  $\epsilon_n^{\mathrm{max}} = -\epsilon_n^{\mathrm{min}} = \rho$ .
C.2 Data along rays
We begin with a key lemma that is central to the proof of Theorem 3.
Lemma 8. Suppose  $\{\pmb{u}_i\}_{i=1}^n \subset \mathbb{R}^d$  is a collection of  $n > 1$  unit vectors such that  $\pmb{u}_i^\top \pmb{u}_j < 0$  for all  $i \neq j$ , and  $\pmb{w}$  is a unit vector such that  $\pmb{u}_i^\top \pmb{w} > 0$  for all  $i = 1, \dots, n$ . Let  $\pmb{a} \in \mathbb{R}^d$  be any vector. Then,
$$
\sum_ {i = 1} ^ {n} \left| \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} \right| \left(\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {w}\right) <   \| \boldsymbol {a} \|.
$$
Before giving the proof of Lemma 8, we first prove an auxiliary result.
Lemma 9. Let  $\mathbf{a} \in \mathbb{R}^d$  be a unit vector, and suppose  $\{\mathbf{u}_i\}_{i=1}^n \subset \mathbb{R}^d$  is a collection of unit vectors such that  $\mathbf{u}_i^\top \mathbf{a} > 0$  for all  $i$  and  $\mathbf{u}_i^\top \mathbf{u}_j < 0$  for all  $i \neq j$ . Let  $\mathbf{b} = \sum_{i=1}^n \mathbf{u}_i \mathbf{u}_i^\top \mathbf{a}$ . Then  $\|\mathbf{b} - \mathbf{a}/2\| \leq 1/2$  with strict inequality if  $n > 1$ .
Proof. It suffices to show  $\boldsymbol{b}^{\top}\boldsymbol{a} \geq \boldsymbol{b}^{\top}\boldsymbol{b}$ , since if this is the case then
$$
\left\| \boldsymbol {b} - \boldsymbol {a} / 2 \right\| = \sqrt {\left(\boldsymbol {b} - \boldsymbol {a} / 2\right) ^ {\top} \left(\boldsymbol {b} - \boldsymbol {a} / 2\right)} = \sqrt {\boldsymbol {b} ^ {\top} \boldsymbol {b} - \boldsymbol {b} ^ {\top} \boldsymbol {a} + \frac {1}{4} \boldsymbol {a} ^ {\top} \boldsymbol {a}} \leq \frac {1}{2}, \tag {57}
$$
which also holds with strict inequality when  $\pmb{b}^{\top}\pmb{a} > \pmb{b}^{\top}\pmb{b}$ .
First, if  $n = 1$ , then  $\pmb{b} = \pmb{a}^{\top}\pmb{u}_{1}\pmb{u}_{1}^{\top}$ , and so  $\pmb{b}^{\top}\pmb{b} = \pmb{a}^{\top}\pmb{u}_{1}\pmb{u}_{1}^{\top}\pmb{u}_{1}\pmb{u}_{1}^{\top}\pmb{a} = \pmb{a}^{\top}\pmb{u}_{1}\pmb{u}_{1}^{\top}\pmb{a} = \pmb{b}^{\top}\pmb{a}$ . Now assume  $n > 1$ . Then we have
$$
\begin{array}{l} \boldsymbol {b} ^ {\top} \boldsymbol {a} = \sum_ {i = 1} ^ {n} \boldsymbol {a} ^ {\top} \boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} (58) \\ = \sum_ {i = 1} ^ {n} \boldsymbol {a} ^ {\top} \boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} (59) \\ > \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \boldsymbol {a} ^ {\top} \boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j} \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} (60) \\ = \boldsymbol {b} ^ {\top} \boldsymbol {b} (61) \\ \end{array}
$$
The first and last equalities hold by definition. The second equality holds because each  $\pmb{u}_i$  is a unit vector. The last inequality holds because for  $i \neq j$
$$
(\boldsymbol {a} ^ {\top} \boldsymbol {u} _ {i}) (\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j}) (\boldsymbol {u} _ {j} ^ {\top} \boldsymbol {a}) <   0,
$$
since  $\pmb{u}_i^\top \pmb{u}_j < 0$  and  $\pmb{a}^{\top}\pmb{u}_i,\pmb{a}^{\top}\pmb{u}_j > 0$  by assumption.
Now we give the proof of Lemma 8.
Proof of Lemma 8. Let  $\pmb{v} = \sum_{i=1}^{n} |\pmb{u}_i^\top \pmb{a}| \pmb{u}_i$ . Then we have
$$
S = \sum_ {i = 1} ^ {n} | \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} | (\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {w}) = \boldsymbol {v} ^ {\top} \boldsymbol {w}
$$
WLOG, we may assume  $\| \pmb{a}\| = 1$ , in which case the lemma reduces to showing  $S = \pmb{v}^{\top}\pmb{w} < 1$ . By the Cauchy-Schwartz inequality, it suffices to show  $\| \pmb{v}\| < 1$ . Toward this end, let us write  $\pmb{v} = \pmb{v}_{+} + \pmb{v}_{-}$  where  $\pmb{v}_{+} = \sum_{i:\pmb{u}_{i}^{\top}\pmb{a} > 0}|\pmb{u}_{i}^{\top}\pmb{a}|\pmb{u}_{i} = \sum_{i:\pmb{u}_{i}^{\top}\pmb{a} > 0}\pmb{u}_{i}\pmb{u}_{i}^{\top}\pmb{a}$  and  $\pmb{v}_{-} = \sum_{i:\pmb{u}_{i}^{\top}\pmb{a} < 0}|\pmb{u}_{i}^{\top}\pmb{a}|\pmb{u}_{i} = \sum_{i:\pmb{u}_{i}^{\top}\pmb{a} < 0}\pmb{u}_{i}\pmb{u}_{i}^{\top}(-\pmb{a})$ . By Lemma 9 we have  $\| \pmb{v}_{+} - \pmb{a} / 2\| \leq 1 / 2$  and  $\| \pmb{v}_{-} + \pmb{a} / 2\| \leq 1 / 2$ , and so
$$
\begin{array}{l} \left\| \boldsymbol {v} \right\| = \left\| \boldsymbol {v} _ {+} + \boldsymbol {v} _ {-} \right\| (62) \\ = \left\| \boldsymbol {v} _ {+} - \boldsymbol {a} / 2 + \boldsymbol {v} _ {-} + \boldsymbol {a} / 2 \right\| (63) \\ \leq \left\| \boldsymbol {v} _ {+} - \boldsymbol {a} / 2 \right\| + \left\| \boldsymbol {v} _ {-} + \boldsymbol {a} / 2 \right\| (64) \\ \leq 1. (65) \\ \end{array}
$$
Also, if either of the index sets  $I_{+} \coloneqq \{i : \pmb{u}_{i}^{\top}\pmb{a} > 0\}$  or  $I_{-} \coloneqq \{i : \pmb{u}_{i}^{\top}\pmb{a} < 0\}$  has cardinality greater than one then the lemma above guarantees  $\| \pmb{v}_{+} - \pmb{a} / 2 \| < 1 / 2$  or  $\| \pmb{v}_{+} - \pmb{a} / 2 \| < 1 / 2$ , and so  $\| \pmb{v} \| < 1$ , which gives  $S < 1$ .
It remains to show  $S < 1$  when  $I_{+}$  or  $I_{-}$  has cardinality less than or equal to one, i.e.,  $|I_{+}| \leq 1$  or  $|I_{-}| \leq 1$ . We consider the various possibilities:
Case 1:  $|I_{+}| = 0$  &  $|I_{-}| = 0$ . Then  $\mathbf{v} = \mathbf{0}$  and so  $S = \mathbf{v}^{\top}\mathbf{w} = 0 < 1$ .
Case 2:  $|I_{+}| = 1$  &  $|I_{-}| = 0$  or  $|I_{+}| = 0$  &  $|I_{-}| = 1$ . Then  $\pmb{v} = |\pmb{u}_i^\top \pmb{a}| \pmb{u}_i$  for some  $i$ , and  $|\pmb{u}_j^\top \pmb{a}| = 0$  for all  $j \neq i$ . By way of contradiction, assume that  $S = \pmb{v}^\top \pmb{w} = |\pmb{u}_i^\top \pmb{a}| \pmb{u}_i^\top \pmb{w} = 1$ . Since  $\pmb{u}_i, \pmb{a}$ , and  $\pmb{w}$  are all unit vectors, the only way this is possible is if  $|\pmb{u}_i^\top \pmb{a}| = 1$  and  $\pmb{u}_i^\top \pmb{w} = 1$ , which implies  $\pmb{a} = \pm \pmb{w}$  and  $\pmb{u}_i = \pmb{w}$ , and so  $\pmb{a} = \pm \pmb{u}_i$ . However, this shows that  $\pmb{u}_j^\top \pmb{u}_i = 0$  for all  $j \neq i$ , contradicting our assumption that  $\pmb{u}_j^\top \pmb{u}_i < 0$  for all  $i \neq j$ . Therefore,  $S < 1$  in this case as well.
Case 3:  $|I_{+}| = 1$  &  $|I_{-}| = 1$ . Then  $\pmb{v} = |\pmb{u}_i^\top \pmb{a}| \pmb{u}_i + |\pmb{u}_j^\top \pmb{a}| \pmb{u}_j$  for some  $i \in I_{+}$  and  $j \in I_{-}$ , and so
$$
\| \boldsymbol {v} \| = \| | \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {a} | \boldsymbol {u} _ {i} + | \boldsymbol {u} _ {j} ^ {\top} \boldsymbol {a} | \boldsymbol {u} _ {j} \| \tag {66}
$$
$$
= \left\| \left(\boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} - \boldsymbol {u} _ {j} \boldsymbol {u} _ {j} ^ {\top}\right) \boldsymbol {a} \right\| \tag {67}
$$
$$
\leq \left\| \boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} - \boldsymbol {u} _ {j} \boldsymbol {u} _ {j} ^ {\top} \right\| \tag {68}
$$
where the last inequality follows by the fact that  $\| \pmb{a} \| = 1$ . Finally, since the matrix  $\pmb{u}_i \pmb{u}_i^\top - \pmb{u}_j \pmb{u}_j^\top$  is symmetric, its operator norm  $\| \pmb{u}_i \pmb{u}_i^\top - \pmb{u}_j \pmb{u}_j^\top \|$  coincides with the maximum eigenvalue (in absolute value). Eigenvectors in the span of  $\pmb{u}_i$  and  $\pmb{u}_j$  have the form  $\pmb{v} = c_i \pmb{u}_i + c_j \pmb{u}_j$  where  $c_i, c_j$  satisfy the equation
$$
\left(\boldsymbol {u} _ {i} \boldsymbol {u} _ {i} ^ {\top} - \boldsymbol {u} _ {j} \boldsymbol {u} _ {j} ^ {\top}\right) \left(c _ {i} \boldsymbol {u} _ {i} + c _ {j} \boldsymbol {u} _ {j}\right) = \lambda \left(c _ {i} \boldsymbol {u} _ {i} + c _ {j} \boldsymbol {u} _ {j}\right) \tag {69}
$$
where  $\lambda \in \mathbb{R}$  is the corresponding eigenvalue. Expanding and equating coefficients gives the system
$$
\left[ \begin{array}{l l} 1 - \lambda & \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j} \\ \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j} & 1 + \lambda \end{array} \right] \left[ \begin{array}{l} c _ {i} \\ c _ {j} \end{array} \right] = \left[ \begin{array}{l} 0 \\ 0 \end{array} \right] \tag {70}
$$
which has non-trivial solutions iff
$$
(1 - \lambda) (1 + \lambda) - \left(\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j}\right) ^ {2} = 0 \quad \Longleftrightarrow \quad \lambda = \pm \sqrt {1 - \left(\boldsymbol {u} _ {i} ^ {\top} \boldsymbol {u} _ {j}\right) ^ {2}} \tag {71}
$$
and so  $|\lambda| < 1$ . Therefore,  $\| \pmb{v} \| \leq \| \pmb{u}_i \pmb{u}_i^\top - \pmb{u}_j \pmb{u}_j^\top\| < 1$  as claimed. And so  $S \leq \| \pmb{v} \| < 1$ .
Figure 6: Illustration of "unit splitting" technique used in proof of Theorem 3. If a ReLU unit is active along two rays (left panel), it can be split into two units with lower representation cost by projecting its inner-layer weight vector  $\boldsymbol{w}$  onto the two rays separately (right panel).
Proof of Theorem 3. Let  $f^{*}$  be any representation cost minimizer. We prove the claim by showing that if  $f^{*}$  fails to have the form specified in (20) then it is possible to construct a norm-ball interpolant  $h$  whose units are aligned with the rays that has strictly smaller representation cost than  $f^{*}$ .
First, let  $f^{*}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_{k}[\pmb{w}_{k}^{\top} \pmb{y} + b_{k}] + \pmb{V} \pmb{x} + c$  be any minimal representative of  $f^{*}$ . By properties of minimal representatives, none of the ReLU boundary sets  $\{\pmb{y} \in \mathbb{R}^{d}: \pmb{w}_{k}^{\top} \pmb{y} + b_{k} = 0\}$  can intersect any of the norm-balls centered at the training samples, since otherwise  $f^{*}$  would be non-constant on one of the norm-balls. Also, we may alter this parameterization in such a way that the active set of every unit avoids the ball centered at the origin  $B_{0} := B(0, \rho)$  without changing the representation cost. In particular, suppose the active set of the  $k$ th unit contains  $B_{0}$ . Then, using the identity  $[t]_{+} = t - [-t]_{+}$  for all  $t \in \mathbb{R}$ , we have
$$
\boldsymbol {a} _ {k} \left[ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} + b _ {k} \right] _ {+} = \boldsymbol {a} _ {k} \left(\boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} + b _ {k}\right) - \boldsymbol {a} _ {k} \left[ - \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} - b _ {k} \right] _ {+}
$$
for all  $\pmb{y} \in \mathbb{R}^d$ . The active set of the reversed unit  $-\pmb{a}_k[-\pmb{w}_k^\top \pmb{y} - b_k]_+$  does not intersect  $B_0$ . Therefore, after applying this transformation to all units whose active sets contain  $B_0$ , we can write  $\pmb{f}^* = \pmb{g}^* + \ell$  where  $\pmb{g}^*$  is a sum of ReLU units whose active sets do not contain  $B_0$  and  $\ell$  is an affine function that combines the original affine part  $\pmb{V}\pmb{y} + \pmb{c}$  with a sum of linear units of the form  $\pmb{a}_k(\pmb{w}_k^\top \pmb{y} + b_k\pmb{a}_k)$  arising from the transformation above. However, because of the interpolation constraint  $\pmb{f}(B_0) = \{\mathbf{0}\}$ , and by the fact that no units in  $\pmb{g}^*$  are active over  $B_0$ , for all  $\pmb{y} \in B_0$  we have
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \boldsymbol {g} ^ {*} (\boldsymbol {y}) + \boldsymbol {\ell} (\boldsymbol {y}) = \boldsymbol {\ell} (\boldsymbol {y}) = \mathbf {0} \tag {72}
$$
which implies  $\ell \equiv 0$ , i.e.,  $f^{*}(\pmb{y}) = g^{*}(\pmb{y})$  for all  $\pmb{y}$ . Finally, note that the inner- and outer-layer weight vectors on the reversed units change only in sign. Therefore, this re-parameterization does not change the representation cost, and so is also a minimal representative.
Now we construct a new norm-ball interpolant  $\pmb{h}$  as follows. Let  $\pmb{f}_{\ell}$  be the function defined as the sum of all units making up the re-parametrized  $\pmb{f}^*$  that are active on at least one norm-ball centered on the  $\ell$ th ray. Also, let  $P_{\ell} = \pmb{u}_{\ell}\pmb{u}_{\ell}^{\top} \in \mathbb{R}^{d\times d}$  denote the orthogonal projector onto the  $\ell$ th ray. Define  $h = \sum_{\ell=1}^{L} h_{\ell}$  where  $h_{\ell}(\pmb{y}) := P_{\ell}f_{\ell}(P_{\ell}\pmb{y})$ . Put in words,  $h$  is constructed by "splitting" any units active over multiple rays into a sum of multiple units aligned with each ray; see Figure 6 for an illustration.
First, we prove that  $\pmb{h}$  satisfies norm-ball interpolation constraints. Let  $\pmb{a}[\pmb{w}^\top \pmb{y} + b]_+$  denote any unit belonging to  $\pmb{f}_{\ell}$ . Since this unit is active on a norm-ball centered on ray  $\ell$ , this implies that  $\pmb{w}^\top \pmb{u}_{\ell} > 0$  and since the unit is inactive on  $B_0$  we must have  $b < 0$ . Also, because we assume  $\pmb{u}_{\ell}^\top \pmb{u}_m < 0$  for any  $m \neq \ell$ , we see that the projected unit  $P_{\ell} \pmb{a}[\pmb{w}^\top P_{\ell} \pmb{y} + b]_+ = P_{\ell} \pmb{a}[(\pmb{w}^\top \pmb{u}_{\ell}) \pmb{u}_{\ell}^\top \pmb{y} + b]_+$  is active on norm-balls centered on ray  $\ell$  and inactive on norm-balls centered on any other ray. In particular, if  $\pmb{y}$  belongs to a norm-ball centered on ray  $\ell$ , then  $h_m(\pmb{y}) = 0$  for all  $m \neq \ell$ . Therefore, if  $\pmb{y} \in B(\pmb{x}_n^{(\ell)}, \rho)$  where  $x_n^{(\ell)}$  denotes a training sample along ray  $\ell$ , then
$$
\begin{array}{l} \boldsymbol {h} (\boldsymbol {y}) = \sum_ {m = 1} ^ {L} \boldsymbol {h} _ {m} (\boldsymbol {y}) (73) \\ = \boldsymbol {h} _ {\ell} (\boldsymbol {y}) (74) \\ = P _ {\ell} f _ {\ell} \left(P _ {\ell} y\right) (75) \\ = P _ {\ell} f \left(P _ {\ell} y\right) (76) \\ = \boldsymbol {P} _ {\ell} \boldsymbol {x} _ {n} ^ {(\ell)} (77) \\ = \boldsymbol {x} _ {n} ^ {(\ell)} (78) \\ \end{array}
$$
which shows that  $h$  satisfies norm-ball interpolation constraints, as claimed.
Next, we show that if  $\pmb{h} \neq \pmb{f}^*$ , then  $\pmb{h}$  has strictly smaller representation cost. Let  $\pmb{u}(\pmb{y}) = \pmb{a}[\pmb{w}^\top \pmb{y} + b]_+$  denote any ReLU unit belonging to  $\pmb{f}^*$ . Since we assume  $\| \pmb{w} \| = 1$ , the contribution of unit  $\pmb{u}$  to the representation cost of  $\pmb{f}^*$  is  $\| \pmb{a} \|$ . Let  $\mathcal{R} \subset \{1, 2, \dots, L\}$  index the subset of rays  $\ell$  for which unit  $\pmb{u}$  is active over at least one norm-ball centered on that ray. In constructing the interpolant  $\pmb{h}$ , the unit  $\pmb{u}$  gets mapped to a sum of  $|\mathcal{R}|$  units:
$$
\sum_ {\ell \in \mathcal {R}} \boldsymbol {P} _ {\ell} \boldsymbol {a} \left[ \left(\boldsymbol {P} _ {\ell} \boldsymbol {w}\right) ^ {\top} \boldsymbol {y} + b \right] _ {+}
$$
whose contribution to the representation cost of  $h$  is bounded above by
$$
C = \sum_ {\ell \in \mathcal {R}} \| \boldsymbol {P} _ {\ell} \boldsymbol {a} \| \| \boldsymbol {P} _ {\ell} \boldsymbol {w} \| = \sum_ {\ell \in \mathcal {R}} | \boldsymbol {u} _ {\ell} ^ {\top} \boldsymbol {a} | | \boldsymbol {u} _ {\ell} ^ {\top} \boldsymbol {w} |.
$$
(a) Training samples along rays
(b) Perturbed training samples
Figure 7: Change in unit alignment of minimal representation cost solutions under perturbations of training samples belonging to rays. The red lines represent the ReLU boundaries of the representation cost minimizing solution satisfying norm-ball interpolation constraints. For the unperturbed samples in (a), all ReLU boundaries align perpendicular to the rays. For the perturbed samples in (b), the ReLU boundaries align perpendicular to the line segments connecting successive samples.
If  $|\mathcal{R}| > 1$ , then by Lemma 8 guarantees  $C < \| \pmb{a}\|$ . And if  $|\mathcal{R}| = 1$  then  $C = |\pmb{u}_{\ell}^{\top}\pmb{a}||\pmb{u}_{\ell}^{\top}\pmb{w}|\leq \| \pmb{a}\|$  with strict inequality unless  $\pmb{w},\pmb{a}$  belong to the span of  $\pmb{u}_{\ell}$ . This implies that any min-cost solution must have all its units aligned with the rays, i.e.,  $\pmb{f}^{*}$  must have the form  $\pmb{f}^{*}(\pmb {y}) = \sum_{\ell = 1}^{L}\pmb{u}_{\ell}\phi_{\ell}(\pmb{u}_{\ell}^{\top}\pmb {y})$  where each  $\phi_{\ell}$  is a univariate function. The representation cost of any function  $\pmb{f}^{*}$  in this form is the sum of the (1-D) representation costs of the  $\phi_{\ell}$ . Therefore,  $\phi_{\ell}$  must also be the 1-D min-cost solution of the norm-ball constraints projected onto ray  $\ell$ , and so must have the form specified by (17).
C.2.1 Perturbations of samples along rays
As an extension of the above setting, suppose the training samples along rays  $\pmb{x}_n^{(\ell)}$  have been slightly perturbed, i.e., we consider training samples  $\widetilde{\pmb{x}}_n^{(\ell)} = \pmb{x}_n^{(\ell)} + \pmb{\epsilon}_n^{(\ell)}$  for some vectors  $\pmb{\epsilon}_n^{(\ell)}$  with  $\| \pmb{\epsilon}_n^{(\ell)} \| < \delta$  for some sufficiently small  $\delta > 0$ . In particular, we make the following two assumptions:
(A1) Let  $\Delta_{\ell} := \{\widetilde{\boldsymbol{x}}_n^{(\ell)} - \widetilde{\boldsymbol{x}}_{n-1}^{(\ell)}\}_{n=1}^{N_{\ell}}$  be the collection of difference vectors between successive perturbed points along the  $\ell$ th ray (with  $\widetilde{\boldsymbol{x}}_0^{(\ell)} := \mathbf{0}$ ). Assume that for all  $\boldsymbol{v}_{\ell} \in \Delta_{\ell}$  and for all  $\boldsymbol{v}_k \in \Delta_k$  with  $k \neq \ell$ , we have  $\boldsymbol{v}_{\ell}^{\top} \boldsymbol{v}_k < 0$ .
(A2) If  $H$  is any halfspace not containing the origin that contains the ball  $B(\tilde{\pmb{x}}_n^{(\ell)};\rho)$ , then  $H$  also contains all successive balls along that ray, i.e.,  $H$  contains  $B(\tilde{\pmb{x}}_m^{(\ell)};\rho)$  for  $m\geq n$ .
Note that if the original points  $\pmb{x}_n^{(\ell)}$  belong to rays that satisfy the conditions of Theorem 3, then for sufficiently small  $\delta$  the perturbed samples  $\widetilde{\pmb{x}}_n^{(\ell)}$  will satisfy assumptions (A1)&(A2) above.
We show in this case the norm-ball interpolating representation cost minimizer is a perturbed version of the min-cost solution for the unperturbed samples as identified in Theorem 3. In particular, the ReLU boundaries align with the line segments connecting successive points along the rays. See Figure 7 for an illustration in the case of  $L = 2$  rays.
Proposition 3. Suppose the training samples  $X$  are a perturbation of data belong to a union of  $L$  rays plus a sample at the origin, i.e.,  $X = \{\mathbf{0}\} \cup \{\widetilde{\pmb{x}}_n^{(1)}\}_{n=1}^{N_1} \cup \dots \cup \{\widetilde{\pmb{x}}_n^{(L)}\}_{n=1}^{N_L}$  where  $\widetilde{\pmb{x}}_n^{(\ell)} = c_n^{(\ell)}\pmb{u}_\ell + \pmb{\epsilon}_n^{(\ell)}$  for some unit vector  $\pmb{u}_\ell$ , constants  $0 < c_1^{(\ell)} < c_2^{(\ell)} < \dots < c_{N_\ell}^{(\ell)}$ , and vectors  $\pmb{\epsilon}_n^{(\ell)}$ . Assume (A1)&(A2) above hold. Then the minimizer  $\pmb{f}^*$  of (19) is unique and is given by
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \sum_ {\ell = 1} ^ {L} \sum_ {n = 1} ^ {N _ {\ell}} \boldsymbol {u} _ {n} ^ {(\ell)} \phi_ {n} ^ {(\ell)} \left(\left(\boldsymbol {u} _ {n} ^ {(\ell)}\right) ^ {\top} \left(\boldsymbol {y} - \widetilde {\boldsymbol {x}} _ {n - 1} ^ {(\ell)}\right)\right) \tag {79}
$$
$$
\begin{array}{l} \text {w h e r e} \boldsymbol {u} _ {n} ^ {(\ell)} = \frac {\widetilde {\boldsymbol {x}} _ {n} ^ {(\ell)} - \widetilde {\boldsymbol {x}} _ {n - 1} ^ {(\ell)}}{\| \widetilde {\boldsymbol {x}} _ {n} ^ {(\ell)} - \widetilde {\boldsymbol {x}} _ {n - 1} ^ {(\ell)} \|} \text {a n d} \phi_ {n} ^ {(\ell)} (t) = s _ {n} ^ {(\ell)} ([ t - a _ {n} ^ {(\ell)} ] - [ t - b _ {n} ^ {(\ell)} ] _ {+}) \text {w i t h} a _ {n} ^ {(\ell)} = \rho , b _ {n} ^ {(\ell)} = \\ \| \boldsymbol {x} _ {n} ^ {(\ell)} - \boldsymbol {x} _ {n - 1} ^ {(\ell)} \| - \rho , \text {a n d} s _ {n} ^ {(\ell)} = \| \widetilde {\boldsymbol {x}} _ {n} ^ {(\ell)} - \widetilde {\boldsymbol {x}} _ {n - 1} ^ {(\ell)} \| / (b _ {n} ^ {(\ell)} - a _ {n} ^ {(\ell)}). \end{array}
$$
Proof. Let  $\pmb{f}^{*}$  be any min-cost solution. Following the same steps as in the proof of Theorem 3, there is a minimal representative of  $\pmb{f}^{*}$  having the form  $\pmb{f}^{*}(\pmb{y}) = \sum_{k} \pmb{a}_{k} [\pmb{w}_{k}^{\top} \pmb{y} + b_{k}]_{+}$  where the active sets of all units in this representation have empty intersection with the ball  $B(\mathbf{0}, \rho)$ .
Let  $\pmb{f}_n^{(\ell)}$  denote the sum of all units in  $\pmb{f}^*$  whose active set boundary  $\{\pmb{y} \in \mathbb{R}^d : \pmb{w}_k^\top \pmb{y} + b_k = 0\}$  intersects the line segment connecting the training samples  $\widetilde{\pmb{x}}_{n-1}^{(\ell)}$  and  $\widetilde{\pmb{x}}_n^{(\ell)}$ . By assumption (A2), this is equivalent to the sum of units that are active over balls  $B(\widetilde{\pmb{x}}_m^{(\ell)}, \rho)$  for  $m \geq n$ , and inactive for the balls with  $m < n$ . In particular, for  $\pmb{y} \in B(\widetilde{\pmb{x}}_n^{(\ell)}, \rho)$ , we have  $\pmb{f}^*(\pmb{y}) = \sum_{m \leq n} \pmb{f}_m^{(\ell)}(\pmb{y})$ . This implies  $\pmb{f}_1^{(\ell)}(\pmb{y}) = \widetilde{\pmb{x}}_1$  for all  $\pmb{y} \in B(\widetilde{\pmb{x}}_1^{(\ell)}, \rho)$ . Likewise,  $\pmb{f}_2^{(\ell)}(\pmb{y}) = \widetilde{\pmb{x}}_2^{(\ell)} - \widetilde{\pmb{x}}_1^{(\ell)}$  for all  $\pmb{y} \in B(\widetilde{\pmb{x}}_2^{(\ell)}; \rho)$ , and so on, such that for all  $n = 1, ..., N_\ell$  we have  $\pmb{f}_n^{(\ell)}(\pmb{y}) = \widetilde{\pmb{x}}_n^{(\ell)} - \widetilde{\pmb{x}}_{n-1}^{(\ell)}$  for all  $\pmb{y} \in B(\widetilde{\pmb{x}}_n^{(\ell)}; \rho)$ .
Now we show how to construct a new interpolating function  $\pmb{h}$  having representation cost less than or equal to  $\pmb{f}^*$ . Let  $\pmb{u}_n^{(\ell)} = \frac{\widetilde{\pmb{x}}_n^{(\ell)} - \widetilde{\pmb{x}}_{n-1}^{(\ell)}}{\|\widetilde{\pmb{x}}_n^{(\ell)} - \widetilde{\pmb{x}}_{n-1}^{(\ell)}\|}$  and define  $P_n^{(\ell)} = \pmb{u}_n^{(\ell)}(\pmb{u}_n^{(\ell)})^\top$ , the orthogonal projector onto the span of the difference vector  $\widetilde{\pmb{x}}_n^{(\ell)} - \widetilde{\pmb{x}}_{n-1}^{(\ell)}$ . Note that the map  $\pmb{y} \mapsto P_n^{(\ell)}\pmb{y} + \widetilde{\pmb{x}}_{n-1}^{(\ell)}$  is projection onto the affine line connecting samples  $\widetilde{\pmb{x}}_{n-1}^{(\ell)}$  and  $\widetilde{\pmb{x}}_n^{(\ell)}$ . Consider the function
$$
\boldsymbol {h} = \sum_ {\ell = 1} ^ {L} \sum_ {n = 1} ^ {N _ {\ell}} \boldsymbol {h} _ {n} ^ {(\ell)}
$$
$$
\boldsymbol {h} _ {n} ^ {(\ell)} (\boldsymbol {y}) = \boldsymbol {P} _ {n} ^ {(\ell)} \boldsymbol {f} _ {n} ^ {(\ell)} \left(\boldsymbol {P} _ {n} ^ {(\ell)} \boldsymbol {y} + \widetilde {\boldsymbol {x}} _ {n - 1} ^ {(\ell)}\right).
$$
Put in words,  $h$  is constructed by aligning all inner-layer and outer-layer weights of units in  $f^{*}$  with the line segment connecting successive data points over which that unit first activates. See Figure 8 for an illustration.