SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {x, \epsilon} \left[ \lim  _ {\sigma \rightarrow 0 ^ {+}} \left(\hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon) - x\right)\left(\hat {x} ^ {\mathrm {e M M S E}} (x + \sigma \epsilon) - \hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon)\right)\right] = 0
$$
Since  $\hat{x}^{\mathrm{eMMSE}}(y)$  and  $\hat{x}^{1 - \mathrm{NN}}(y)$  are bounded and  $\mathbb{E}[x] < \infty$ . Therefore, we get
Next, we prove Theorem 1.
Proof. Assume, without loss of generality, that  $N = 2$ . Let  $p_x(x)$  be a probability density function with bounded second moment such that  $p_x(x) > 0$  for all  $x \in [x_1, x_2]$ . According to Lemma 4
So we need to prove that
$$
\lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(\hat {x} ^ {1 - \mathrm {N N}} (y)\right) > \lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(f _ {1 D} ^ {*} (y)\right)
$$
For the case of  $N = 2$ , the training set includes two data points  $\{x_{1}, x_{2}\}$ . So we get,
$$
\hat {x} ^ {1 - \mathrm {N N}} (y) = \left\{ \begin{array}{l l} x _ {1} & y <   \frac {x _ {1} + x _ {2}}{2} \\ x _ {2} & \frac {x _ {1} + x _ {2}}{2} \leq y \end{array} \right.
$$
$$
f _ {1 D} ^ {*} \left(y\right) = \left\{ \begin{array}{l l} x _ {1} & y <   x _ {1} + \Delta_ {1} \\ \frac {x _ {2} - x _ {1}}{x _ {2} - x _ {1} + \Delta_ {2} - \Delta_ {1}} \left(y - x _ {1} - \Delta_ {1}\right) + x _ {1} & x _ {1} + \Delta_ {1} \leq y \leq x _ {2} + \Delta_ {2} \\ x _ {2} & y > x _ {2} + \Delta_ {2} \end{array} \right.
$$
where  $\max_{m\in [M]}\epsilon_{1,m} = \Delta_1 > 0,\min_{m\in [M]}\epsilon_{2,m} = \Delta_2 < 0$ . Note that  $\hat{x}^{1 - \mathrm{NN}}(y) = f_{1D}^* (y)$  for all  $y\in (-\infty ,x_1 + \Delta_1)\cup (x_2 + \Delta ,\infty)$  and  $\mathbb{E}[x] < \infty, \mathbb{E}[x^2 ] < \infty$ . Therefore,
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {x} ^ {1 - \mathrm {N N}} (y)\right) - \operatorname {M S E} \left(f _ {1 D} ^ {*} (y)\right) = \\ \int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(\hat {x} ^ {1 - N N} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) (25) \\ - \int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(f _ {1 D} ^ {*} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) (26) \\ \end{array}
$$
First, we derive (25):
$$
\int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(\hat {x} ^ {1 - N N} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) =
$$
$$
\int_ {- \infty} ^ {\infty} \int_ {x _ {1} + \Delta_ {1}} ^ {\frac {x _ {1} + x _ {2}}{2}} (x _ {1} - x) ^ {2} p _ {y | x} (y | x) p _ {x} (x) \mathrm {d y} \mathrm {d} x + \int_ {- \infty} ^ {\infty} \int_ {\frac {x _ {1} + x _ {2}}{2}} ^ {x _ {2} + \Delta_ {2}} (x _ {2} - x) ^ {2} p _ {y | x} (y | x) p _ {x} (x) \mathrm {d y} \mathrm {d} x =
$$
$$
\mathbb {E} _ {x} \left[ P \left(y \in \left[ x _ {1} + \Delta_ {1}, \frac {x _ {1} + x _ {2}}{2} \right] \Big | x\right) (x _ {1} - x) ^ {2} \right] + \mathbb {E} _ {x} \left[ P \left(y \in \left[ \frac {x _ {1} + x _ {2}}{2}, x _ {2} + \Delta_ {2} \right] \Big | x\right) (x _ {2} - x) ^ {2} \right].
$$
$$
\begin{array}{l} \mathbb {E} \left[ P \left(y \in \left[ x _ {1} + \Delta_ {1}, \frac {x _ {1} + x _ {2}}{2} \right] \mid x\right) (x _ {1} - x) ^ {2} \right] <   \mathbb {E} \left[ (x _ {1} - x) ^ {2} \right] <   \infty \\ \mathbb {E} \left[ P \left(y \in \left[ \frac {x _ {1} + x _ {2}}{2}, x _ {2} + \Delta_ {2} \right] \mid x\right) (x _ {2} - x) ^ {2} \right] <   \mathbb {E} \left[ (x _ {2} - x) ^ {2} \right] <   \infty \\ \end{array}
$$
thus by the Dominated convergence theorem we obtain
$$
\lim  _ {\sigma \rightarrow 0 ^ {+}} \int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(\hat {x} ^ {1 - \mathrm {N N}} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) =
$$
$$
\mathbb{E}_{x}\left[\lim_{\sigma \to 0^{+}}P\left(y\in \left[x_{1} + \Delta_{1},\frac{x_{1} + x_{2}}{2}\right]\Bigg|x\right)\left(x_{1} - x\right)^{2}\right] + \mathbb{E}_{x}\left[\lim_{\sigma \to 0^{+}}P\left(y\in \left[\frac{x_{1} + x_{2}}{2},x_{2} + \Delta_{2}\right]\Bigg|x\right)\left(x_{2} - x\right)^{2}\right] =
$$
$$
\left. \mathbb {E} _ {x} \left[ (x - x _ {1}) ^ {2} 1 \left\{x \in \left[ x _ {1}, \frac {x _ {1} + x _ {2}}{2} \right] \right\} \right] + \mathbb {E} _ {x} \left[ (x - x _ {2}) ^ {2} 1 \left\{x \in \left[ \frac {x _ {1} + x _ {2}}{2}, x _ {2} \right] \right\} \right] > C > 0 \right.
$$
since  $p_x(x) > 0$  for all  $x \in [x_1, x_2]$ . Next, we derive (26)
$$
\begin{array}{l} \int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(f _ {1 D} ^ {*} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) = \\ \mathbb {E} _ {x, y} \left[ 1 \left\{y \in \left[ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} \right] \right\} \left(f _ {1 D} ^ {*} (y) - x\right) ^ {2} \right] = \\ \mathbb {E} _ {x, \epsilon} \left[ 1 \left\{x + \sigma \epsilon \in [ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} ] \right\} \left(f _ {1 D} ^ {*} (x + \sigma \epsilon) - x\right) ^ {2} \right] \\ \end{array}
$$
$$
\mathbb {E} _ {x, \epsilon} \left[ 1 \left\{x + \sigma \epsilon \in [ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} ] \right\} \left(f _ {1 D} ^ {*} (x + \sigma \epsilon) - x\right) ^ {2} \right] <   \mathbb {E} _ {x, \epsilon} \left[ \left(f _ {1 D} ^ {*} (x + \sigma \epsilon) - x\right) ^ {2} \right] <   \infty
$$
$$
\begin{array}{l} \lim  _ {\epsilon \rightarrow 0 ^ {+}} \int_ {- \infty} ^ {\infty} d x \int_ {x _ {1} + \Delta_ {1}} ^ {x _ {2} + \Delta_ {2}} d y \left(f _ {1 D} ^ {*} (y) - x\right) ^ {2} p _ {y | x} (y | x) p _ {x} (x) = \\ \lim  _ {\epsilon \rightarrow 0 ^ {+}} \mathbb {E} _ {x, \epsilon} \left[ 1 \left\{x + \sigma \epsilon \in [ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} ] \right\}\left(f _ {1 D} ^ {*} (x + \sigma \epsilon) - x\right) ^ {2} \right] = \\ \mathbb {E} _ {x, \epsilon} \left[ \lim  _ {\epsilon \rightarrow 0 ^ {+}} 1 \left\{x + \sigma \epsilon \in [ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} ] \right\}\left(f _ {1 D} ^ {*} (x + \sigma \epsilon) - x\right) ^ {2} \right] = 0 \\ \end{array}
$$
$$
\lim  _ {\epsilon \rightarrow 0 ^ {+}} 1 \left\{x + \sigma \epsilon \in \left[ x _ {1} + \Delta_ {1}, x _ {2} + \Delta_ {2} \right]\right\} = 1 \left\{x \in \left[ x _ {1}, x _ {2} \right]\right\}
$$
$$
\lim  _ {\epsilon \to 0 ^ {+}} f _ {1 D} ^ {*} \left(x + \sigma \epsilon\right) = \left\{ \begin{array}{l l} x _ {1} & x <   x _ {1} \\ x & x _ {1} \leq x \leq x _ {2} \\ x _ {2} & x > x _ {2} \end{array} \right.
$$
Proof. First we prove that for all  $y \in \cup_{n \in [N-1]}\left\{\frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}}\right\} f_{1D}^*(y) = y$ . We find the intersection between the line  $f_{1D}^*(y) = y$  and the linear part of (17) (the third branch)
$$
\frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} (y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)) + x _ {n} = y \tag {27}
$$
$$
\left(x _ {n + 1} - x _ {n}\right) \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) + x _ {n} \left(x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) = y \left(x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) \tag {28}
$$
$$
x _ {n} \left(x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) - \left(x _ {n + 1} - x _ {n}\right) \left(x _ {n} + \epsilon_ {n} ^ {\max }\right) = y \left(\epsilon_ {n + 1} ^ {\min } - \epsilon_ {n} ^ {\max }\right) \tag {29}
$$
$$
x _ {n} \left(x _ {n + 1} + \epsilon_ {n + 1} ^ {\min }\right) - x _ {n + 1} \left(x _ {n} + \epsilon_ {n} ^ {\max }\right) = y \left(\epsilon_ {n + 1} ^ {\min } - \epsilon_ {n} ^ {\max }\right) \tag {30}
$$
$$
x _ {n} \epsilon_ {n + 1} ^ {\min } - x _ {n + 1} \epsilon_ {n} ^ {\max } = y \left(\epsilon_ {n + 1} ^ {\min } - \epsilon_ {n} ^ {\max }\right) \tag {31}
$$
$$
y = \frac {x _ {n + 1} \epsilon_ {n} ^ {\operatorname* {m a x}} - x _ {n} \epsilon_ {n + 1} ^ {\operatorname* {m i n}}}{\epsilon_ {n} ^ {\operatorname* {m a x}} - \epsilon_ {n + 1} ^ {\operatorname* {m i n}}}. \tag {32}
$$
$$
x _ {n} + \epsilon_ {n} ^ {\max } <   \frac {x _ {n + 1} \epsilon_ {n} ^ {\max } - x _ {n} \epsilon_ {n + 1} ^ {\min }}{\epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min }} <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } \tag {33}
$$
Since,
$$
x _ {n} + \epsilon_ {n} ^ {\max } <   \frac {x _ {n + 1} \epsilon_ {n} ^ {\max } - x _ {n} \epsilon_ {n + 1} ^ {\min }}{\epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min }} \tag {34}
$$
$$
\left(x _ {n} + \epsilon_ {n} ^ {\max }\right) \left(\epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min }\right) <   x _ {n + 1} \epsilon_ {n} ^ {\max } - x _ {n} \epsilon_ {n + 1} ^ {\min } \tag {35}
$$
$$
\left(x _ {n} + \epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min }\right) \epsilon_ {n} ^ {\max } <   x _ {n + 1} \epsilon_ {n} ^ {\max } \tag {36}
$$
$$
x _ {n} + \epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min } <   x _ {n + 1} \tag {37}
$$
$$
x _ {n} + \epsilon_ {n} ^ {\max } <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } \tag {38}
$$
which holds according to Assumption 1.
$$
\frac {x _ {n + 1} \epsilon_ {n} ^ {\max } - x _ {n} \epsilon_ {n + 1} ^ {\min }}{\epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min}} <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } \tag {39}
$$
$$
x _ {n + 1} \epsilon_ {n} ^ {\max } - x _ {n} \epsilon_ {n + 1} ^ {\min } <   \left(x _ {n + 1} + \epsilon_ {n + 1} ^ {\min }\right) \left(\epsilon_ {n} ^ {\max } - \epsilon_ {n + 1} ^ {\min }\right) \tag {40}
$$
$$
- x _ {n} \epsilon_ {n + 1} ^ {\min } <   - \epsilon_ {n + 1} ^ {\min } \left(x _ {n + 1} - \epsilon_ {n} ^ {\max } + \epsilon_ {n + 1} ^ {\min }\right) \tag {41}
$$
$$
x _ {n} <   x _ {n + 1} - \epsilon_ {n} ^ {\max } + \epsilon_ {n + 1} ^ {\min } \tag {42}
$$
$$
x _ {n} + \epsilon_ {n} ^ {\max } <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } \tag {43}
$$
Next we prove that  $f_{1D}^{*}(y)$  is contractive toward a set of the clean datapoints  $\{x_{n}\}_{n=1}^{N}$  on  $\mathbb{R} \setminus \cup_{n \in [N-1]} \left\{ \frac{x_{n+1} \epsilon_{n}^{\max} - x_{n} \epsilon_{n+1}^{\min}}{\epsilon_{n}^{\max} - \epsilon_{n+1}^{\min}} \right\}$ . In the case where  $y \in (-\infty, x_{1} + \epsilon_{1}^{\min}]$  or  $y \in \cup_{n \in [N]} [x_{n} + \epsilon_{n}^{\min}, x_{n} + \epsilon_{n}^{\max}]$  or  $y \in [x_{N} + \epsilon_{N}^{\max}, \infty)$ ,  $f_{1D}^{*}(y) \in \{x_{n}\}_{n=1}^{N}$  therefore (18) holds for all  $0 \leq \alpha < 1$ . In the case where  $y \in \cup_{n \in [N-1]} [x_{n} + \epsilon_{n}^{\max}, \frac{x_{n+1} \epsilon_{n}^{\max} - x_{n} \epsilon_{n+1}^{\min}}{\epsilon_{n}^{\max} - \epsilon_{n+1}^{\min}})$  we choose  $i = n$ ,
$$
\left| f _ {1 D} ^ {*} (y) - f \left(x _ {n}\right) \right| = \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) \tag {44}
$$
Figure 5: Illustration of  $f_{1D}^{*}(y)$ .
There exists  $0 < \gamma_{1} < 1$  such that
$$
x _ {n} \leq \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} (y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)) + x _ {n} \leq \gamma_ {1} y \tag {45}
$$
since for  $y\in \cup_{n\in [N - 1]}[x_n + \epsilon_n^{\max},\frac{x_{n + 1}\epsilon_n^{\max} - x_n\epsilon_{n + 1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n + 1}^{\min}})$ $f_{1D}^{*}(y)$  is bellow the line  $f(y) = y$  because  $f_{1D}^{*}(y)$  is an affine function with slope larger than 1 and  $f_{1D}^{*}\left(\frac{x_{n + 1}\epsilon_{n}^{\max} - x_{n}\epsilon_{n + 1}^{\min}}{\epsilon_{n}^{\max} - \epsilon_{n + 1}^{\min}}\right) = \frac{x_{n + 1}\epsilon_{n}^{\max} - x_{n}\epsilon_{n + 1}^{\min}}{\epsilon_{n}^{\max} - \epsilon_{n + 1}^{\min}}.$  Therefore there exists  $0 <   \alpha_{1} <   1$
$$
\left| f _ {1 D} ^ {*} (y) - f \left(x _ {n}\right) \right| = \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) \leq \gamma_ {1} y - x _ {n} \leq \alpha_ {1} \left(y - x _ {n}\right) \tag {46}
$$
since,
$$
\gamma_ {1} y - x _ {n} \leq \alpha_ {1} (y - x _ {n}) \tag {47}
$$
$$
\alpha_ {1} \geq \frac {\gamma_ {1} y - x _ {n}}{y - x _ {n}}. \tag {48}
$$
In the case where  $y \in \cup_{n \in [N-1]} \left( \frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}}, x_{n+1} + \epsilon_{n+1}^{\min} \right]$  we choose  $i = n+1$ ,
$$
\left| f _ {1 D} ^ {*} (y) - f \left(x _ {n + 1}\right) \right| = x _ {n + 1} - \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) - x _ {n}. \tag {49}
$$
There exists  $0 < \gamma_{2} < 1$  such that
$$
x _ {n + 1} \geq \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} (y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)) + x _ {n} \geq \frac {1}{\gamma_ {2}} y \tag {50}
$$
since for  $y \in \cup_{n \in [N-1]} \left( \frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}}, x_{n+1} + \epsilon_{n+1}^{\min} \right] f_{1D}^*(y)$  is above the line  $f(y) = y$  because  $f_{1D}^*(y)$  is an affine function with slope larger than 1 and  $f_{1D}^*(\frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}}) = \frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}}$ . Therefore there exists  $0 < \alpha_2 < 1$
$$
\left| f _ {1 D} ^ {*} (y) - f \left(x _ {n + 1}\right) \right| = x _ {n + 1} - \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) - x _ {n} \tag {51}
$$
$$
\leq x _ {n + 1} - \frac {1}{\gamma_ {2}} y \leq \alpha_ {2} (x _ {n + 1} - y) \tag {52}
$$
$$
x _ {n + 1} - \frac {1}{\gamma_ {2}} y \leq \alpha_ {2} (x _ {n + 1} - y) \tag {53}
$$
$$
\alpha_ {2} \geq \frac {x _ {n + 1} - \frac {1}{\gamma_ {2}} y}{x _ {n + 1} - y}. \tag {54}
$$
Therefore (18) holds for  $\alpha = \max \{\alpha_{1},\alpha_{2}\}$ .
C Proofs of Results in Section 5
Let  $\pmb{f} : \mathbb{R}^d \to \mathbb{R}^d$  be any function realizable as a shallow ReLU network. Consider any parametrization of  $\pmb{f}$  given by  $\pmb{f}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k [\pmb{w}_k^\top \pmb{y} + b_k]_+ + \pmb{V}\pmb{y} + \pmb{c}$ . We say such a parametrization is a minimal representative of  $\pmb{f}$  if  $\| \pmb{w}_k \|_2 = 1$  and  $\pmb{a}_k \neq 0$  for all  $k = 1, \dots, K$ , and  $R(\pmb{f}) = \sum_{k=1}^{K} \| \pmb{a}_k \|_2$ . In particular, the units making up a minimal representative must be distinct in the sense that the hyperplanes describing ReLU boundaries  $H_k = \{\pmb{x} \in \mathbb{R}^d : \pmb{w}_k^\top \pmb{x} + b_k = 0\}$  are distinct, which implies that no units can be cancelled or combined.
We will also make use of the following lemma, which shows that representation costs are invariant to a translation of the training samples, assuming the function is suitably translated.
Lemma 5. Let  $\pmb{f}:\mathbb{R}^d\to \mathbb{R}^d$  be any function realizable as a shallow ReLU net satisfying norm-ball interpolation constraints  $\pmb {f}(B(\pmb {x}_n;\rho)) = \{\pmb {x}_n\}$  for all  $n = 1,\dots,N$ . Let  $\pmb{x}_0\in \mathbb{R}^d$ . Then the function  $\pmb {g}(\pmb {y}) = \pmb {f}(\pmb {y} - \pmb {x}_0) + \pmb {x}_0$  is such that  $R(\pmb {g}) = R(\pmb {f})$  and  $g(B(x_{n} + x_{0};\rho)) = \{x_{n} + x_{0}\}$  for all  $n = 1,\ldots ,N$ .
Proof. First we show  $\pmb{g}(B(\pmb{x}_n + \pmb{x}_0; \rho)) = \{\pmb{x}_n + \pmb{x}_0\}$  for all  $n = 1, \dots, N$ . Fix any  $n$ , and let  $\pmb{y} \in B(\pmb{x}_n + \pmb{x}_0; \rho)$ . Then  $\pmb{y} = \pmb{y}' + \pmb{x}_0$  for some  $\pmb{y}' \in B(\pmb{x}_n; \rho)$ , and so  $\pmb{g}(\pmb{y}) = \pmb{f}(\pmb{y}') + \pmb{x}_0 = \pmb{x}_n + \pmb{x}_0$  as claimed.
To show that  $R(\pmb{g}) = R(\pmb{f})$ , let  $\pmb{f}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k [\pmb{w}_k^\top \pmb{y} + b_k]_+ + \pmb{V}\pmb{y} + \pmb{c}$  be any minimal representative of  $\pmb{f}$ . Then
$$
\begin{array}{l} \boldsymbol {g} (\boldsymbol {y}) = \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} \left[ \boldsymbol {w} _ {k} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {x} _ {0}\right) + b _ {k} \right] _ {+} + \boldsymbol {V} \left(\boldsymbol {y} - \boldsymbol {x} _ {0}\right) + \boldsymbol {c} + \boldsymbol {x} _ {0} (55) \\ = \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} \left[ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} + \tilde {b} _ {k} \right] _ {+} + \boldsymbol {V} \boldsymbol {y} + \tilde {\boldsymbol {c}} (56) \\ \end{array}
$$
with  $\tilde{b}_k = b_k - \boldsymbol{w}_k^\top \boldsymbol{x}_0$  and  $\tilde{\boldsymbol{c}} = \boldsymbol{c} + \boldsymbol{x}_0 - \boldsymbol{V}\boldsymbol{x}_0$ . And so  $R(\boldsymbol{g}) \leq \sum_{k=1}^{K} \| \boldsymbol{a}_k \|_2 = R(\boldsymbol{f})$ . A parallel argument with the roles of  $\boldsymbol{g}$  and  $\boldsymbol{f}$  reversed shows that  $R(\boldsymbol{f}) \leq R(\boldsymbol{g})$ , and so  $R(\boldsymbol{f}) = R(\boldsymbol{g})$ .
In particular, this lemma shows that if  $\pmb{f}^{*}(\pmb{y})$  is a norm-ball interpolating representation cost minimizer of the training samples  $\{\pmb{x}_n\}_{n=1}^N$ , then  $\pmb{g}^{*}(\pmb{y}) = \pmb{f}^{*}(\pmb{y} - \pmb{x}_0) + \pmb{x}_0$  is a norm-ball interpolating min-cost solution of the translated training samples  $\{\pmb{x}_n + \pmb{x}_0\}_{n=1}^N$ .
C.1 Training data belonging to a subspace
The proof of Theorem 2 is a direct consequence of the following two lemmas:
Lemma 6. Suppose the clean training samples  $\{\pmb{x}_n\}_{n=1}^N$  belong to a  $r$ -dimensional subspace  $\mathcal{S} \subset \mathbb{R}^d$ , and let  $\pmb{P_S}$  denote the orthogonal projector onto  $\mathcal{S}$ . Let  $\pmb{f}(\pmb{y})$  be any function realizable as a shallow ReLU network satisfying  $\pmb{f}(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$  for all  $n = 1, \dots, N$ . Define  $\pmb{g}(\pmb{y}) = \pmb{P_S} \pmb{f}(\pmb{x})$ . Then  $\pmb{g}(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$  for all  $n = 1, \dots, N$ , and  $R(\pmb{g}) \leq R(\pmb{f})$ , with strict inequality if  $\pmb{f} \neq \pmb{g}$ .
Proof. First, for any  $\pmb{y} \in B(\pmb{x}_n, \rho)$  we have  $\pmb{g}(\pmb{y}) = \pmb{P_S} \pmb{f}(\pmb{y}) = \pmb{P_S} \pmb{x}_n = \pmb{x}_n$ . Therefore,  $\pmb{g}(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$  for all  $n = 1, \dots, N$ .
Now, let  $\pmb{f}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+ + \pmb{V}\pmb{y} + \pmb{c}$  be a minimal representative of  $\pmb{f}$ . Then  $g(\pmb{y}) = \sum_{k=1}^{K} \pmb{P}\pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+ + \pmb{P}(\pmb{V}\pmb{y} + \pmb{c})$ . Since  $\| \pmb{P}\pmb{a}_k\| \leq \| \pmb{a}_k\|$  for all  $k$ , we have  $R(\pmb{g}) \leq \sum_{k=1}^{K} \| \pmb{P}_{\mathcal{S}}\pmb{a}_k\| \leq \sum_{k=1}^{K} \| \pmb{a}_k\| = R(\pmb{f})$ .
Now we show  $\pmb{f} \neq \pmb{g}$  implies  $R(\pmb{g}) < R(\pmb{f})$ . Observe that if any of the outer-layer weight vectors  $\pmb{a}_k \notin S$  then  $\| P_S \pmb{a}_k \| < \| \pmb{a}_k \|$ , which implies  $R(\pmb{g}) < R(\pmb{f})$ . Hence, it suffices to show that  $\pmb{f} \neq \pmb{g}$  implies some  $\pmb{a}_k \notin S$ .
First, consider the case where  $P_{S}V = V$  and  $P_{S}c = c$ . Then in this case  $f \neq g$  if only if  $\sum_{k=1}^{K}(P_{S}a_{k} - a_{k})[w_{k}^{\top}y + b_{k}]_{+} \neq 0$  for all  $y \in \mathbb{R}^{d}$ , which implies  $P_{S}a_{k} \neq a_{k}$  for some  $k$ , or equivalently  $a_{k} \notin S$ .
Next, assume either  $P_{\mathcal{S}}V \neq V$  or  $P_{\mathcal{S}}c \neq c$ . Fix any training sample index  $n$ , and let  $A_{n}$  denote the index set of units active over the ball  $B(\pmb{x}_n,\rho)$ . Then, since  $f(\pmb{y})$  is constant for all  $\pmb{y} \in B(\pmb{x}_n,\rho)$ , the Jacobian  $\partial f(\pmb{y}) = \sum_{k \in A_n} \pmb{a}_k \pmb{w}_k^\top + V = 0$  for all  $\pmb{y} \in B(\pmb{x}_n,\rho)$ . This gives  $V = -\sum_{k \in A_n} \pmb{a}_k \pmb{w}_k^\top$ . Therefore, if  $P_{\mathcal{S}}V \neq V$  then at least one of the  $\pmb{a}_k \notin S$ . On the other hand, if  $P_{\mathcal{S}}c \neq c$  then for all  $\pmb{y} \in B(\pmb{x}_n,\rho)$  we have
$$
\boldsymbol {f} (\boldsymbol {y}) = \sum_ {k \in A _ {n}} \boldsymbol {a} _ {k} \left(\boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} + b _ {k}\right) + \boldsymbol {V} \boldsymbol {y} + \boldsymbol {c} = \sum_ {k \in A _ {n}} \left(\boldsymbol {a} _ {k} \boldsymbol {w} _ {k} ^ {\top} + \boldsymbol {V}\right) \boldsymbol {y} + \sum_ {k \in A _ {n}} b _ {k} \boldsymbol {a} _ {k} + \boldsymbol {c} = \sum_ {k \in A _ {n}} b _ {k} \boldsymbol {a} _ {k} + \boldsymbol {c} = \boldsymbol {x} _ {n},
$$
which implies  $\pmb{c} = \pmb{x}_n - \sum_{k \in A_n} b_k \pmb{a}_k$ , and since  $\pmb{x}_n \in S$  this implies some  $\pmb{a}_k \notin S$ , proving the claim.
Lemma 7. Suppose the clean training samples  $\{\pmb{x}_n\}_{n=1}^N$  belong to an  $r$ -dimensional subspace  $S \subset \mathbb{R}^d$ , and let  $\pmb{P_S}$  denote the orthogonal projector onto  $S$ . Let  $\pmb{f}$  be any network satisfying  $\pmb{f}(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$ . Define  $\pmb{h}(\pmb{y}) = \pmb{f}(\pmb{P_S}\pmb{y})$ . Then  $\pmb{h}(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$  and  $R(\pmb{h}) \leq R(\pmb{f})$  with strict inequality if  $\pmb{h} \neq \pmb{f}$ .
Proof. Define  $P_{S}^{-1}(B(\pmb{x}_{n},\rho)) \coloneqq \{\pmb{y} \in \mathbb{R}^{d} : P_{S}\pmb{y} \in B(\pmb{x}_{n},\rho)\}$ , i.e., the set of points in  $\mathbb{R}^d$  whose projection onto  $S$  is contained in  $B(\pmb{x}_{n},\rho)$ . Then clearly  $h(P_{S}^{-1}(B(\pmb{x}_{n},\rho))) = \{\pmb{x}_{n}\}$ . Also, by properties of norm-balls, we have  $B(\pmb{x}_{n},\rho) \subset P_{S}^{-1}(B(\pmb{x}_{n},\rho))$ . Therefore,  $h(B(\pmb{x}_{n},\rho)) = \{\pmb{x}_{n}\}$  for all  $n = 1,\dots,N$ .
Next, let  $\pmb{f}(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+ + \pmb{V}\pmb{y} + \pmb{c}$  be a minimal representative of  $\pmb{f}$ . Then  $h(\pmb{y}) = \sum_{k=1}^{K} \pmb{a}_k[(\pmb{P}_{\mathcal{S}}\pmb{w}_k)^\top \pmb{y} + b_k]_+ + \pmb{V}\pmb{P}_{\mathcal{S}}\pmb{y} + \pmb{c}$ . Since  $\| \pmb{P}_{\mathcal{S}}\pmb{w}_k \| \leq \| \pmb{w}_k \|$  for all  $k$ , we have  $R(\pmb{h}) \leq \sum_{k=1}^{K} \| \pmb{a}_k \| \| \pmb{P}_{\mathcal{S}}\pmb{w}_k \| \leq \sum_{k=1}^{K} \| \pmb{a}_k \| \| \pmb{w}_k \| = R(\pmb{f})$ .
Finally, we show that if  $\pmb{f} \neq \pmb{h}$ , then  $R(\pmb{h}) < R(\pmb{f})$ . Observe that if any of the inner-layer weight vectors  $\pmb{w}_k \notin S$  then  $\| \pmb{P}\pmb{w}_k\| < \| \pmb{w}_k\|$ , which implies  $R(\pmb{h}) < R(\pmb{f})$ . Hence, it suffices to show that  $\pmb{f} \neq \pmb{h}$  implies some  $\pmb{w}_k \notin S$ . First, consider the case  $VPS = V$ . Then  $\pmb{f} \neq \pmb{h}$  if and only if  $P_{S}w_{k} \neq w_{k}$  for some  $k$ , or equivalently,  $w_{k} \notin S$ . Next, consider the case  $VPS \neq V$ . Fix any training sample index  $n$ , and let  $A_{n}$  denote the index set of units active over  $B(x_{n},\rho)$ . Then, since  $f(y)$  is constant for all  $y \in B(x_n,\rho)$ , the Jacobian  $\partial f(\pmb{x}) = \sum_{k\in A_n}\pmb{a}_k\pmb{w}_k^\top +V = 0$  for all  $y \in B_n(x_n,\rho)$ . This gives  $V = -\sum_{k}\pmb{a}_{k}\pmb{w}_{k}^{\top}$ . Therefore, if  $VPS \neq V$  (i.e., at least one row of  $V$  is not contained in  $S$ ) then at least one of the  $w_{k} \notin S$ , proving the claim.
Finally, we now give the proof Theorem 2 and Corollary 1.