SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 8: Illustration of "unit alignment" technique used in the proof of Proposition 3. The left panel shows an interpolant  $f$  whose ReLU boundaries (shown here as dark blue and dark red lines) are not aligned with the line segments  $S_{1}$  and  $S_{2}$  connecting successive training samples. The right panel shows how a new interpolant  $h$  can be constructed by projecting units in  $f$  onto the line segments  $S_{1}$  and  $S_{2}$ , which reduces the representation cost.
Now we show that  $\pmb{h}$  satisfies norm-ball interpolation constraints. By assumption (A1), the function  $\pmb{h}_n^{(\ell)}$  is non-zero only on the norm-balls  $B(\widetilde{\pmb{x}}_m^{(\ell)},\rho)$  for  $m\geq n$ . This implies that for all  $\pmb {y}\in B(\widetilde{\pmb{x}}_n^{(\ell)};\rho)$  we have  $\pmb {h}(\pmb {y}) = \sum_{m\leq n}\pmb{h}_{n}^{(\ell)}(\pmb {y})$
Observe that  $\pmb{h}(B(\mathbf{0},\rho)) = \{\mathbf{0}\}$  since all functions  $\pmb{h}_n^{(\ell)}$  are zero on  $B(\mathbf{0},\rho)$ . Also, for all  $\pmb{y} \in B(\widetilde{\pmb{x}}_1^{(\ell)};\rho)$  we have  $P_1^{(\ell)}\pmb{y} \in B(\widetilde{\pmb{x}}_1^{(\ell)};\rho)$ , and so
$$
\boldsymbol {h} (\boldsymbol {y}) = \boldsymbol {h} _ {1} ^ {(\ell)} (\boldsymbol {y}) = \boldsymbol {P} _ {1} ^ {\ell} \boldsymbol {f} _ {1} ^ {(\ell)} (\boldsymbol {P} _ {1} ^ {(\ell)} \boldsymbol {y}) = \boldsymbol {P} _ {1} ^ {(\ell)} \widetilde {\boldsymbol {x}} _ {1} ^ {(\ell)} = \widetilde {\boldsymbol {x}} _ {1} ^ {(\ell)}.
$$
Similarly, the only terms in  $\pmb{h}$  active for  $\pmb{y} \in B(\widetilde{\pmb{x}}_2^{(\ell)};\rho)$  are  $\pmb{h}_1^{(\ell)}$  and  $\pmb{h}_2^{(\ell)}$ , for which  $\pmb{h}_1^{(\ell)}(\pmb{y}) = \pmb{x}_1^{(\ell)}$ , and  $\pmb{h}_2^{(\ell)}(\pmb{y}) = \pmb{P}_2^{(\ell)}\pmb{f}_2^{(\ell)}(\pmb{P}_2^{(\ell)}\pmb{y} + \widetilde{\pmb{x}}_1^{(\ell)}) = \pmb{P}_2^{(\ell)}(\widetilde{\pmb{x}}_2^{(\ell)} - \widetilde{\pmb{x}}_1^{(\ell)}) = \widetilde{\pmb{x}}_2^{(\ell)} - \widetilde{\pmb{x}}_1^{(\ell)}$ . This gives
$$
\boldsymbol {h} (\boldsymbol {y}) = \boldsymbol {h} _ {1} ^ {(\ell)} (\boldsymbol {y}) + \boldsymbol {h} _ {2} ^ {(\ell)} (\boldsymbol {y}) = \widetilde {\boldsymbol {x}} _ {2} ^ {(\ell)}.
$$
Iterating this procedure for all  $n = 1,\dots,N_{\ell}$  we see that if  $\pmb {y}\in B(\widetilde{\pmb{x}}_n^{(\ell)},\rho)$  then  $\pmb{h}_n^{(\ell)}(\pmb {y}) = \widetilde{\pmb{x}}_n^{(\ell)} -$ $\widetilde{\pmb{x}}_{n - 1}^{(\ell)}$ , and so  $\pmb {h}(\pmb {y}) = \sum_{m\leq n}\pmb{h}_{m}^{(\ell)}(\pmb {y}) = \pmb{x}_{n}^{(\ell)}$ , as claimed.
Following steps similar to the proof of Theorem 3, we can show that  $R(\pmb{h}) \leq R(\pmb{f}^*)$  and with strict inequality if  $\pmb{h} \neq \pmb{f}^*$ . First, if any unit  $\pmb{u}(\pmb{y}) = \pmb{a}[\pmb{w}^\top \pmb{y} + b]_+$  in  $\pmb{f}^*$  is active over balls centered on multiple rays  $\mathcal{R} \subset \{1,2,\dots,L\}$ , then in the construction of  $\pmb{h}$  the unit  $\pmb{u}$  gets mapped to a sum of multiple units:
$$
\sum_ {\ell \in \mathcal {R}} \boldsymbol {u} _ {n _ {\ell}} ^ {(\ell)} (\boldsymbol {u} _ {n _ {\ell}} ^ {(\ell)}) ^ {\top} \boldsymbol {a} [ \boldsymbol {w} ^ {\top} \boldsymbol {u} _ {n _ {\ell}} ^ {(\ell)} (\boldsymbol {u} _ {n _ {\ell}} ^ {(\ell)}) ^ {\top} (\boldsymbol {y} + \widetilde {\boldsymbol {x}} _ {n _ {\ell} - 1} ^ {(\ell)}) + b ] _ {+}
$$
for some  $n_{\ell}$  with  $1 \leq n_{\ell} \leq N_{\ell}$ . The contribution of the sum of these units to the representation cost of  $\pmb{h}$  is less than or equal to  $C = \sum_{\ell \in \mathcal{R}} |(\pmb{u}_{n_{\ell}}^{(\ell)})^{\top} \pmb{a}| |(\pmb{u}_{n_{\ell}}^{(\ell)})^{\top} \pmb{w}|$ . By assumption (A1), we know  $(\pmb{u}_m^{(\ell)})^\top \pmb{u}_n^{(p)} < 0$  for all  $p \neq \ell$  and for all  $m, n$ . Therefore, by Lemma 8 we know  $C < \| \pmb{a} \|$ . This shows  $\pmb{f}^*$  cannot have any units active over balls centered on different rays, since otherwise  $\pmb{h}$  has strictly smaller representation cost. Therefore, we have shown  $\pmb{f}^*$  as  $\pmb{f}^* = \sum_{\ell=1}^{L} \sum_{n=1}^{N_{\ell}} \pmb{f}_n^{(\ell)}$ . Additionally, we see that  $\pmb{h}_n^{(\ell)} = \pmb{f}_n^{(\ell)}$ , i.e., all inner- and outer-layer weight vectors of units in  $\pmb{f}_n^{(\ell)}$  are aligned with  $\pmb{u}_n^{(\ell)}$ , since otherwise the representation cost of  $\pmb{f}^*$  could be reduced. Therefore, each  $\pmb{f}_n^{(\ell)}$  must have the form  $\pmb{f}_n^{(\ell)}(\pmb{y}) = \pmb{u}_n^{(\ell)} \psi_n^{(\ell)}((\pmb{u}_n^{(\ell)})^\top \pmb{y})$ . Minimizing the  $\psi_n^{(\ell)}$  separately under interpolation constraints, we arrive at the unique solution given in (79).
C.3 Simplex Data
C.3.1 Simplex with one obtuse vertex
Proof of Proposition 2. Consider training samples  $\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_N \in \mathbb{R}^d$  whose convex hull is a  $(N - 1)$ -simplex such that  $\pmb{x}_1$  makes an obtuse angle with all other vertices, i.e.,  $(\pmb{x}_n - \pmb{x}_1)^\top \pmb{x}_1 < 0$  for all  $n = 2, \dots, N$ . By Lemma 5,  $f^*$  is a norm-ball interpolating min-cost solution if and only if  $g^*(\pmb{y}) = f^*(\pmb{y} - \pmb{x}_1) + \pmb{x}_1$  is a norm-ball interpolating min-cost solution for the translated points  $\pmb{0}, \pmb{x}_2 - \pmb{x}_1, \dots, \pmb{x}_N - \pmb{x}_1 \in \mathbb{R}^d$ . The latter configuration satisfies the hypotheses of Theorem 3 with a single training sample per ray. Therefore, the min-cost solution  $g^*$  of the translated points has units whose inner- and outer-layer weight vectors are aligned with  $\pmb{x}_n - \pmb{x}_1$ ,  $n = 2, \dots, N$ , and likewise for  $f^*$ , since it is translation of  $g^*$ .
C.3.2 Simplex with all acute vertices
Justification of Conjecture 1. For concreteness we focus on the case of three points  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3 \in \mathbb{R}^d$  whose convex hull is an acute triangle, and let  $B_1, B_2, B_3$  be open balls of radius  $\rho$  centered at  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3$  (respectively).
Let  $\pmb{f}$  be any norm-ball interpolating min-cost solution, and let  $\pmb{f}(\pmb{y}) = \sum_{k} \pmb{a}_{k} [\pmb{w}_{k}^{\top} \pmb{y} + b_{k}] + \pmb{V} \pmb{x} + \pmb{c}$  be any minimal representative of  $\pmb{f}$ .
First, by properties of minimal representatives, none of the ReLU boundary sets  $H_{k} = \{\pmb{y} \in \mathbb{R}^{d} : \pmb{w}_{k}^{\top} \pmb{y} + b_{k} = 0\}$  intersect any of the balls centered at the training samples. Also, the active set of each unit in  $\pmb{f}$  must contain either one or two norm-balls, since otherwise the unit is either inactive over all balls or active over all balls, in which cases the unit can be removed or absorbed into unregularized linear part while strictly reducing the representation cost. By "reversing units", as in the proof of Theorem 3, we may transform the parameterization in such a way that the active set of every unit contains exactly one ball, and do this without changing the representation cost.
After this transformation, we may write
$$
\boldsymbol {f} = \boldsymbol {f} _ {1} + \boldsymbol {f} _ {2} + \boldsymbol {f} _ {3} + \ell
$$
where  $\pmb{f}_i$  is a sum of units active only on the ball  $B_i$  and no others, and where  $\ell(\pmb{y}) = \pmb{A}\pmb{y} + \pmb{b}$  is an affine function. Then we have
$$
\forall \mathbf {y} _ {1} \in B _ {1}, \quad \boldsymbol {f} (\mathbf {y} _ {1}) = \boldsymbol {f} _ {1} (\mathbf {y} _ {1}) + \boldsymbol {A} \mathbf {y} _ {1} + \boldsymbol {b} = \boldsymbol {x} _ {1} \tag {80}
$$
$$
\forall \mathbf {y} _ {2} \in B _ {2}, \quad \boldsymbol {f} (\mathbf {y} _ {2}) = \boldsymbol {f} _ {2} (\mathbf {y} _ {2}) + \boldsymbol {A} \mathbf {y} _ {2} + \boldsymbol {b} = \boldsymbol {x} _ {2} \tag {81}
$$
$$
\forall \mathbf {y} _ {3} \in B _ {3}, \quad \boldsymbol {f} (\mathbf {y} _ {3}) = \boldsymbol {f} _ {3} (\mathbf {y} _ {3}) + \boldsymbol {A} \mathbf {y} _ {3} + \boldsymbol {b} = \boldsymbol {x} _ {3} \tag {82}
$$
and so,
$$
\forall \mathbf {y} _ {1} \in B _ {1}, \quad \mathbf {f} _ {1} (\mathbf {y} _ {1}) = \mathbf {x} _ {1} - \mathbf {b} - \mathbf {A} \mathbf {y} _ {1} \tag {83}
$$
$$
\forall \mathbf {y} _ {2} \in B _ {2}, \quad \mathbf {f} _ {2} (\mathbf {y} _ {2}) = \mathbf {x} _ {2} - \mathbf {b} - \mathbf {A} \mathbf {y} _ {2} \tag {84}
$$
$$
\forall \mathbf {y} _ {3} \in B _ {3}, \quad \mathbf {f} _ {3} (\mathbf {y} _ {3}) = \mathbf {x} _ {3} - \mathbf {b} - \mathbf {A} \mathbf {y} _ {3} \tag {85}
$$
Finding the min-cost solution then amounts to minimizing of the representation cost of  $f_{1}, f_{2}, f_{3}$  subject to the above constraints. This is made challenging by the fact that the constraints are coupled together by the parameters  $A$  and  $b$  of the affine part. However, under the assumption  $A = 0$ , we prove below that the min-cost solution must have the conjectured form  $f^{*}$  as given in (86). However, since we cannot a priori assume  $A = 0$ , this does not constitute a full proof.
Before proceeding, we give a lemma that will be used to lower bound  $R(\pmb{f})$  (assuming  $\pmb{A} = \mathbf{0}$ ).
Lemma 10. Suppose  $\mathbf{g}$  is a sum of ReLU units such that  $\mathbf{g} = \mathbf{0}$  on a closed convex region  $C\subset \mathbb{R}^d$  and  $\mathbf{g}$  is constant and equal to  $\mathbf{c}$  on a closed ball  $B\subset \mathbb{R}^d$ , and  $\mathbf{g}$  has a minimal representative where all its units are active on  $B$ . Then
$$
R (\boldsymbol {g}) \geq \frac {2 \| \boldsymbol {c} \|}{\operatorname {d i s t} (B , C)}
$$
where  $dist(B,C) = \min_{\pmb {y}\in B,\pmb {x}\in C}\| \pmb {y} - \pmb {x}\|$
Proof. Let  $\pmb{g}(\pmb{y}) = \sum_{k} \pmb{a}_{k}[\pmb{w}_{k}^{\top} \pmb{y} + b_{k}]_{+}$  be a minimal representative where each unit is active on  $B$ . For  $\pmb{g}$  to be constant on  $B$  it must be the case that  $\sum_{k} \pmb{a}_{k} \pmb{w}_{k}^{\top} = \mathbf{0}$ . Let  $\pmb{y} = \pmb{y}_{0}$  be any value for which  $\| \partial \pmb{g}(\pmb{y}_{0}) \|$ , the operator norm of the Jacobian of  $\pmb{g}$ , is maximized. Since  $\pmb{g}$  is piecewise linear, we see that its Lipschitz constant  $\mathrm{Lip}(\pmb{g}) = \| \partial \pmb{g}(\pmb{y}_{0}) \|$ . Let  $I_{0}$  be the set of indices of active units at  $\pmb{y}_{0}$  so that  $\partial \pmb{g}(\pmb{y}_{0}) = \sum_{k \in I_{0}} \pmb{a}_{k} \pmb{w}_{k}^{\top}$ , and let  $I_{1}$  be the complementary index set. Then because  $\sum_{k} \pmb{a}_{k} \pmb{w}_{k}^{\top} = \mathbf{0}$ , we have  $\sum_{k \in I_{0}} \pmb{a}_{k} \pmb{w}_{k}^{\top} = -\sum_{k \in I_{1}} \pmb{a}_{k} \pmb{w}_{k}^{\top}$ , and so  $\| \sum_{k \in I_{0}} \pmb{a}_{k} \pmb{w}_{k}^{\top} \| = \| \sum_{k \in I_{1}} \pmb{a}_{k} \pmb{w}_{k}^{\top} \|$ . Therefore,
$$
\begin{array}{l} 2 \operatorname {L i p} (\boldsymbol {g}) = 2 \| \partial \boldsymbol {g} (\boldsymbol {y} _ {0}) \| = 2 \left\| \sum_ {k \in I _ {0}} \boldsymbol {a} _ {k} \boldsymbol {w} _ {k} ^ {\top} \right\| = \left\| \sum_ {k \in I _ {0}} \boldsymbol {a} _ {k} \boldsymbol {w} _ {k} ^ {\top} \right\| + \left\| \sum_ {k \in I _ {1}} \boldsymbol {a} _ {k} \boldsymbol {w} _ {k} ^ {\top} \right\| \\ \leq \sum_ {k} \| \boldsymbol {a} _ {k} \| \| \boldsymbol {w} _ {k} \| = R (\boldsymbol {g}). \\ \end{array}
$$
$$
\operatorname {L i p} (\boldsymbol {g}) \geq \max  _ {\boldsymbol {x} \in C, \boldsymbol {y} \in B} \frac {\| \boldsymbol {g} (\boldsymbol {y}) - \boldsymbol {g} (\boldsymbol {x}) \|}{\| \boldsymbol {y} - \boldsymbol {x} \|} = \frac {\| \boldsymbol {c} \|}{\min  _ {\boldsymbol {x} \in C , \boldsymbol {y} \in B} \| \boldsymbol {y} - \boldsymbol {x} \|} = \frac {\| \boldsymbol {c} \|}{\operatorname {d i s t} (B , C)}.
$$
Combining this with the previous inequality gives the claim.
Now, returning to the proof of the main claim, for  $n = 1,2,3$ , let  $C_n$  be the closed convex region given by the intersection of all closed half-planes containing the balls  $B_j$  and  $B_k$ ,  $j \neq n$ , and  $k \neq n$ ,  $j \neq k$ . By assumption,  $f_n$  vanishes on  $C_n$  and  $f_n$  is constant and equal to  $x_n - b$  on the closed ball  $B_n$ . So, by Lemma 10, we see that
$$
R \left(\boldsymbol {f} _ {n}\right) \geq \frac {2 \| \boldsymbol {x} _ {n} - \boldsymbol {b} \|}{\delta_ {n}}
$$
where  $\delta_{n} = \mathrm{dist}(B_{n},C_{n})$ . Therefore, for all  $\pmb {b}\in \mathbb{R}^d$  we have
$$
R (\boldsymbol {f}) = R (\boldsymbol {f} _ {1}) + R (\boldsymbol {f} _ {2}) + R (\boldsymbol {f} _ {3}) \geq \sum_ {n = 1} ^ {3} \frac {2 \| \boldsymbol {x} _ {n} - \boldsymbol {b} \|}{\delta_ {n}}
$$
Let  $\overline{\pmb{x}}\in \mathbb{R}^d$  be the minimizer of
$$
\min  _ {\boldsymbol {b} \in \mathbb {R} ^ {d}} \sum_ {n = 1} ^ {3} \frac {2 \| \boldsymbol {x} _ {n} - \boldsymbol {b} \|}{\delta_ {n}}.
$$
Then plugging in  $b = \overline{x}$  above, we have the lower bound
$$
R (\boldsymbol {f}) \geq \sum_ {n = 1} ^ {3} \frac {2 \| \boldsymbol {x} _ {n} - \overline {{\boldsymbol {x}}} \|}{\delta_ {n}},
$$
which is independent of  $b$ .
Finally, simple calculations show that the conjectured min-cost solution  $f^{*}$  specified in (86) has representation cost achieving this lower bound. Therefore, under the assumption  $A = 0$ ,  $f^{*}$  is a min-cost solution.
Now we prove that, in the special case where the convex hull of the training points is an equilateral triangle, the norm-ball interpolator  $f^{*}$  identified in Conjecture 1 is a min-cost solution (though not necessarily the unique min-cost solution). In this case, we may also give  $f^{*}$  a more explicit form, as detailed below.
Proposition 4. Suppose the convex hull of the training points  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3 \in \mathbb{R}^d$  is an equilateral triangle. Assume the norm-balls  $B_n \coloneqq B(\pmb{x}_n, \rho)$  centered at each training point have radius  $\rho < \| \pmb{x}_n - \pmb{x}_0 \| / 2$ ,  $n = 1, 2, 3$ , where  $\pmb{x}_0 = \frac{1}{3} (\pmb{x}_1 + \pmb{x}_2 + \pmb{x}_3)$  is the centroid of the triangle. Then a minimizer  $f^*$  of (19) is given by
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \boldsymbol {u} _ {1} \phi_ {1} \left(\boldsymbol {u} _ {1} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {x} _ {0}\right)\right) + \boldsymbol {u} _ {2} \phi_ {2} \left(\boldsymbol {u} _ {2} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {x} _ {0}\right)\right) + \boldsymbol {u} _ {3} \phi_ {3} \left(\boldsymbol {u} _ {3} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {x} _ {0}\right)\right) + \boldsymbol {x} _ {0}, \tag {86}
$$
where  $\phi_n(t) = s_n([t - a_n]_+ - [t - b_n]_+)$  with  $\mathbf{u}_n = \frac{\mathbf{x}_n - \mathbf{x}_0}{\|\mathbf{x}_n - \mathbf{x}_0\|}$ ,  $a_n = -\frac{1}{2}\|\mathbf{x}_n - \mathbf{x}_0\| + \rho$ ,  $b_n = \|\mathbf{x}_n - \mathbf{x}_0\| - \rho$ , and  $s_n = \|\mathbf{x}_n - \mathbf{x}_0\| / (b_n - a_n)$ .
Proof. By translation and scale invariance of min-cost solutions, without loss of generality we may assume  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3 \in \mathbb{R}^d$  are unit-norm vectors and mean-zero (i.e., the triangle centroid  $\pmb{x}_0 = \mathbf{0}$ ). In this case, the assumption on  $\rho$  translates to  $\rho < 1/2$ . Additionally, it suffices to prove the claim in the case  $d = 2$ . This is because, if  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3 \in \mathbb{R}^d$  are the vertices of an equilateral triangle whose centroid is at the origin, then these points are contained in a two-dimensional subspace  $S \subset \mathbb{R}^d$ . And if we let  $P \in \mathbb{R}^{d \times 2}$  be a matrix whose columns are an orthonormal basis for  $S$ , then by Theorem 2,  $f$  is a min-cost solution if and only if  $f(y) = Pf_0(P^\top y)$ , where  $f_0: \mathbb{R}^2 \to \mathbb{R}^2$  is a min-cost solution under the constraints  $f_0(B(P^\top x_n, \rho)) = \{P^\top x_n\}$  for all  $n = 1, 2, 3$ . Therefore, the problem reduces to finding a min-cost solution of the projected points  $P^\top x_1, P^\top x_2, P^\top x_3 \in \mathbb{R}^2$  whose convex hull is an equilateral triangle in  $\mathbb{R}^2$ .
So now let  $\pmb{f} : \mathbb{R}^2 \to \mathbb{R}^2$  be any min-cost solution under the assumption  $\pmb{x}_1, \pmb{x}_2, \pmb{x}_3 \in \mathbb{R}^2$  are unit-norm, have zero mean, and  $\rho < 1/2$ . By reversing units as necessary, there exists a minimal representative of  $\pmb{f}$  that can be put in the form  $\pmb{f}(\pmb{y}) = \pmb{f}_1(\pmb{y}) + \pmb{f}_2(\pmb{y}) + \pmb{f}_3(\pmb{y}) + \pmb{A}\pmb{y} + \pmb{c}$ , such that  $\pmb{f}_n(\pmb{y})$  is a sum of ReLU units, all of which are active on  $B_n$ , and all of which are inactive on  $B_j$  for  $j \neq n$ .
Let  $Q$  be the reflection matrix  $Q = 2x_{1}x_{1}^{\top} - I$ , which reflects points across the line spanned by  $x_{1}$ . In particular,  $y \in B_{1}$  then  $Qy \in B_{1}$  while if  $y \in B_{2}$  then  $Qy \in B_{3}$  (and vice versa). Also,  $Qx_{1} = x_{1}$ ,  $Qx_{2} = x_{3}$ , and  $Qx_{3} = x_{2}$ . Define  $g(y) = \frac{1}{2}\left(f(y) + Q^{-1}f(Qy)\right)$ . Then it is easy to check that  $g$  satisfies interpolation constraints, and since  $Q$  is unitary, we have  $R(g) \leq \frac{1}{2}R(f) + \frac{1}{2}R(Q \circ f \circ Q^{-1}) = \frac{1}{2}R(f) + \frac{1}{2}R(f) = R(f)$ . Furthermore, since  $f$  is a min-cost solution, we must have  $R(g) = R(f)$ . Additionally, since none of the units making up  $f$  are active over more than one ball, neither are the units making up  $g$ , which implies no pair of units belonging to  $g$  combine to form an affine function. Therefore, we may write  $g(y) = g_{1}(y) + g_{2}(y) + g_{3}(y) + By + v$ , where  $B = A + Q^{-1}AQ$ , such that each  $g_{n}$  is a sum of ReLU units, all of which are active on  $B_{n}$ , and all of which are inactive on  $B_{j}$  for  $j \neq n$ .
Let  $\pmb{U}$  be a rotation matrix by 120 degrees such that  $\pmb{x}_2 = \pmb{U}\pmb{x}_1$ ,  $\pmb{x}_3 = \pmb{U}\pmb{x}_2$ , and  $\pmb{x}_1 = \pmb{U}\pmb{x}_3$ . Consider the symmetrized version of  $\pmb{g}$  given by  $h(\pmb{y}) := \frac{1}{3} (\pmb{g}(\pmb{y}) + \pmb{U}^{-1}\pmb{g}(\pmb{U}\pmb{y}) + \pmb{U}^{-2}\pmb{g}(\pmb{U}^2\pmb{y}))$ .
Since for all  $\pmb{y} \in B_n$  we have  $\pmb{U}\pmb{y} \in B_{n + 1}$  (with indices understood modulo 3), it is easy to verify that  $\pmb{h}$  satisfies interpolation constraints. Also, since  $\pmb{U}$  is unitary, we have  $R(\pmb{h}) \leq R(\pmb{g}) \leq R(\pmb{f})$  which implies  $R(\pmb{h}) = R(\pmb{f})$  since  $\pmb{f}$  is a min-cost solution. Again, since none of the units making up  $\pmb{g}$  are active over more than one ball, neither are the units making up  $\pmb{h}$ , which implies no pair of units belonging to  $\pmb{h}$  combine to form an affine function. And so, we may write  $\pmb{h}(\pmb{y}) = h_1(\pmb{y}) + h_2(\pmb{y}) + h_3(\pmb{y}) + C + u$ , such that each  $h_n$  is a sum of ReLU units, all of which are active on  $B_n$ , and all of which are inactive on  $B_j$  for  $j \neq n$ .
Observe that  $\pmb{u} = \frac{1}{3} (\pmb{I} + \pmb{U}^{-1} + \pmb{U}^{-2})\pmb{v} = \pmb{0}$ . Also, we have  $C = \frac{1}{3} (\pmb{B} + \pmb{U}^{-1}\pmb{B}\pmb{U} + \pmb{U}^{-2}\pmb{B}\pmb{U}^2)$ . This implies  $C$  commutes with  $U$ , because it is easy to see that  $C = U^{-1}CU$ . Additionally,  $C$  commutes with  $Q$ , because it is easy to see  $B = Q^{-1}BQ$ , and by properties of rotations/reflections we have  $UQ = QU^{-1}$ , which implies
$$
\begin{array}{l} Q ^ {- 1} C Q = \frac {1}{3} \left(Q ^ {- 1} B Q + Q ^ {- 1} U ^ {- 1} B U Q + Q ^ {- 1} U ^ {- 2} B U ^ {2} Q\right) \\ = \frac {1}{3} \left(\boldsymbol {Q} ^ {- 1} \boldsymbol {B} \boldsymbol {Q} + \boldsymbol {Q} ^ {- 1} \boldsymbol {U} ^ {- 1} \boldsymbol {B} \boldsymbol {U} \boldsymbol {Q} + \boldsymbol {Q} ^ {- 1} \boldsymbol {U} ^ {- 2} \boldsymbol {B} \boldsymbol {U} ^ {2} \boldsymbol {Q}\right) \\ = \frac {1}{3} \left(\boldsymbol {Q} ^ {- 1} \boldsymbol {B} \boldsymbol {Q} + (\boldsymbol {U} \boldsymbol {Q}) ^ {- 1} \boldsymbol {B} (\boldsymbol {U} \boldsymbol {Q}) + (\boldsymbol {U} \boldsymbol {Q}) ^ {- 1} \boldsymbol {U} ^ {- 1} \boldsymbol {B} \boldsymbol {U} (\boldsymbol {U} \boldsymbol {Q})\right) \\ = \frac {1}{3} \left(\boldsymbol {Q} ^ {- 1} \boldsymbol {B} \boldsymbol {Q} + \boldsymbol {U} \boldsymbol {Q} ^ {- 1} \boldsymbol {B} \boldsymbol {Q} \boldsymbol {U} ^ {- 1} + \boldsymbol {U} \boldsymbol {Q} ^ {- 1} \boldsymbol {U} ^ {- 1} \boldsymbol {B} \boldsymbol {U} \boldsymbol {Q} \boldsymbol {U} ^ {- 1}\right) \\ = \frac {1}{3} \left(Q ^ {- 1} B Q + U Q ^ {- 1} B Q U ^ {- 1} + U ^ {2} Q ^ {- 1} B Q U ^ {- 2}\right) \\ = \frac {1}{3} \left(\boldsymbol {B} + \boldsymbol {U} \boldsymbol {B} \boldsymbol {U} ^ {- 1} + \boldsymbol {U} ^ {2} \boldsymbol {B} \boldsymbol {U} ^ {- 2}\right) \\ = \frac {1}{3} \left(\boldsymbol {B} + \boldsymbol {U} ^ {- 2} \boldsymbol {B} \boldsymbol {U} ^ {2} + \boldsymbol {U} ^ {- 1} \boldsymbol {B} \boldsymbol {U}\right) \\ = C, \\ \end{array}
$$
which shows  $C$  commutes with  $Q$ . Now we show that any matrix  $C$  that commutes with both  $U$  and  $Q$  is a scaled identity matrix. Let  $x_{1}^{\perp}$  denote a unit-vector perpendicular to  $x_{1}$ , such that  $\{x_{1}, x_{1}^{\perp}\}$  form an orthonormal basis for  $\mathbb{R}^{2}$ . First, we know  $Q$  has eigenvectors  $x_{1}$  and  $x_{1}^{\perp}$  with eigenvalues 1 and -1, respectively. And so  $QC x_{1} = CQ x_{1} = C x_{1}$ , while  $QC x_{1}^{\perp} = CQ x_{1}^{\perp} = -C x_{1}$ , which implies  $C x_{1} = a x_{1}$  and  $C x_{1}^{\perp} = b x_{1}^{\perp}$  for some  $a, b \in \mathbb{R}$ , i.e.,  $x_{1}$  and  $x_{1}^{\perp}$  are eigenvectors of  $C$  with eigenvalues  $a$  and  $b$ , respectively. Furthermore, we have  $CU x_{1} = UC x_{1} = a U x_{1}$  and  $CU x_{1}^{\perp} = UC x_{1}^{\perp} = b U x_{1}$ . This implies  $U x_{1}$  and  $U x_{1}^{\perp}$  are also eigenvectors of  $C$  with eigenvalues  $a$  and  $b$ , respectively. But since  $x_{1}$  and  $U x_{1}$  are linearly independent, and likewise so are  $x_{1}^{\perp}$  and  $U x_{1}^{\perp}$ , the only way this could be true is if  $a = b$ , i.e., every vector in  $\mathbb{R}^{2}$  is an eigenvector of  $C$  for the same eigenvalue  $\lambda \in \mathbb{R}$ , which implies  $C = \lambda I$  for some  $\lambda \in \mathbb{R}$ .
Figure 9: Illustration of ReLU boundaries (in red) of original interpolant  $f$  (left), the function  $g$  (middle) obtained by enforcing reflection symmetry, and the function  $h$  (right) obtained by enforcing both reflection and rotation symmetry.
Therefore, we have shown that every min-cost solution  $\pmb{f}$  maps to a min-cost solution  $\pmb{h}$  of the form
$$
\boldsymbol {h} (\boldsymbol {y}) = \boldsymbol {h} _ {1} (\boldsymbol {y}) + \boldsymbol {h} _ {2} (\boldsymbol {y}) + \boldsymbol {h} _ {3} (\boldsymbol {y}) + \lambda \boldsymbol {I}
$$
for some  $\lambda \in \mathbb{R}$ , where each  $h_n$  is a sum of ReLU units, all of which are active on  $B_n$  and all of which are inactive on  $B_j$ ,  $j \neq n$ . In particular,  $h_n(\pmb{y}) = \pmb{x}_n - \lambda \pmb{y}$  for all  $\pmb{y} \in B_n$  and  $h_n(\pmb{y}) = \pmb{0}$  for all  $\pmb{y} \in C_n$ , where  $C_n$  is the intersection of all half-planes containing the balls  $B_j$ ,  $j \neq n$ .
This means we can write  $\pmb{h}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+ + \lambda \pmb{I}$  and  $\pmb{h}_n(\pmb{y}) = \sum_{k \in \mathcal{A}_n} \pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+$  for some index sets  $\mathcal{A}_1, \mathcal{A}_2, \mathcal{A}_3$  partitioning  $\{1, \dots, K\}$  so that  $R(\pmb{h}) = \sum_{k=1}^{K} \| \pmb{a}_k \| = \sum_{n=1}^{3} \sum_{k \in \mathcal{A}_n} \| \pmb{a}_k \|$ . Let  $R_0(\cdot)$  denote the representation cost of a function computed without an unregularized linear part, i.e., define  $R_0$  analogously to  $R$  except where the model class  $\pmb{h}_{\theta}$  in (6) is constrained to have  $V = 0$ . Then, since the realizations of the  $h_n$  functions considered above do not have a linear part, we see that  $\sum_{k \in \mathcal{A}_n} \| \pmb{a}_k \| \geq R_0(\pmb{h}_n)$  and so  $R(\pmb{h}) = \sum_{n=1}^{3} \sum_{k \in \mathcal{A}_n} \| \pmb{a}_k \| \geq R_0(\pmb{h}_1) + R_0(\pmb{h}_2) + R_0(\pmb{h}_3)$ .