SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we show how lower bound  $R_0(\pmb{h}_n)$  for all  $n = 1,2,3$ . Let  $\pmb{x}_n^\perp$  denote a unit-vector perpendicular to  $\pmb{x}_n$ , such that  $\{\pmb{x}_n,\pmb{x}_n^\perp\}$  form an orthonormal basis for  $\mathbb{R}^2$ . Consider the univariate functions  $h_n^\parallel (t)\coloneqq \pmb{x}_n^\top \pmb {h}_n(\pmb {x}_nt)$  and  $h_n^\perp (t)\coloneqq (\pmb{x}_n^\perp)^\top \pmb {h}_n(\pmb{x}_n^\perp t + \pmb {x}_n)$ . Here  $h_n^\parallel$  is the projection of  $\pmb{h}$  onto the line spanned by  $\pmb{x}_n$ , and  $h_n^\perp$  is a projection onto the line perpendicular to  $\pmb{x}_n$  passing through the point  $\pmb{x}_n$ . In particular, by the constraints on  $h_n$ , we see that  $h_n^\parallel$  and  $h_n^\perp$  satisfy the constraints
$$
h _ {n} ^ {\parallel} (t) = \left\{ \begin{array}{l l} 0 & \text {i f} t <   - 1 / 2 + \rho \\ 1 - \lambda t & \text {i f} t > 1 - \rho \end{array} \right. \tag {87}
$$
and  $h_n^\perp (t) = -\lambda t$  if  $|t|\leq \rho$
Claim 1: For all  $n = 1,2,3$ ,  $R_0(\pmb{h}_n) \geq R_0(h_n^\parallel) + R_0(h_n^\perp)$ . Proof: Let  $\pmb{h}_n(\pmb{y}) = \sum_k \pmb{a}_k[\pmb{w}_k^\top \pmb{y} + b_k]_+ + c$  be any realization of  $\pmb{h}_n$ , whose representation cost is  $C = \sum_k \frac{1}{2}\left(\|\pmb{a}_k\|^2 + \|\pmb{w}_k\|^2\right)$ . Then realizations of  $h_n^\parallel$  and  $h_n^\perp$  are given by
$$
h _ {n} ^ {\parallel} (t) = \boldsymbol {x} _ {n} ^ {\top} \boldsymbol {h} _ {i} (\boldsymbol {x} _ {i} t) = \sum_ {k} \left(\boldsymbol {x} _ {n} ^ {\top} \boldsymbol {a} _ {k}\right) \left[ \left(\boldsymbol {x} _ {n} ^ {\top} \boldsymbol {w} _ {k}\right) t + b _ {k} \right] _ {+} + \boldsymbol {x} _ {n} ^ {\top} \boldsymbol {c}. \tag {88}
$$
$$
h _ {n} ^ {\perp} (t) = \left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {h} _ {n} \left(\boldsymbol {x} _ {n} ^ {\perp} t + \boldsymbol {x} _ {n}\right) = \sum_ {k} \left(\left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {a} _ {k}\right) \left[ \left(\left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {w} _ {k}\right) t + b _ {k} + \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {x} _ {n} \right] _ {+} + \left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {c}, \tag {89}
$$
whose representation costs  $C^{\parallel}$  and  $C^{\perp}$ , respectively, are given by
$$
C ^ {\parallel} = \sum_ {k} \frac {1}{2} \left(\left(\boldsymbol {x} _ {n} ^ {\top} \boldsymbol {a} _ {k}\right) ^ {2} + \left(\boldsymbol {x} _ {n} ^ {\top} \boldsymbol {w} _ {k}\right) ^ {2}\right) \tag {90}
$$
$$
C ^ {\perp} = \sum_ {k} \frac {1}{2} \left(\left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {a} _ {k}\right) ^ {2} + \left(\left(\boldsymbol {x} _ {n} ^ {\perp}\right) ^ {\top} \boldsymbol {w} _ {k}\right) ^ {2}) \tag {91}
$$
and by the Pythagorean Theorem we see that  $C = C^{\parallel} + C^{\perp}$ . Therefore,  $C \geq R_0(h_n^{\parallel}) + R_0(h_n^{\perp})$  and finally minimizing over all realizations of  $h_n$  gives the claim.
Claim 2: For all  $n = 1,2,3$ ,  $R_0(h_n^{\parallel}) \geq \left|\frac{1 - \lambda\beta}{\beta - \alpha}\right| + \left|\frac{1 - \lambda\alpha}{\beta - \alpha}\right|$ , where  $\beta = 1 - \rho$  and  $\alpha = -1/2 + \rho$ . Proof: By results in Savarese et al. [2019],  $R_0(h_n^{\parallel}) \geq R_0(p)$  where  $p(t)$  is the function satisfying the same constraints as  $h_n^{\parallel}$  given in (87) while linearly interpolating over the interval  $t \in [\alpha, \beta]$ . From the formula  $R_0(p) = \max \left\{\int |p''(t)|dt, |p'(\infty) + p'(-\infty)|\right\}$  as established in Savarese et al. [2019], we can show directly that  $R_0(p) = \left|\frac{1 - \lambda\beta}{\beta - \alpha}\right| + \left|\frac{1 - \lambda\alpha}{\beta - \alpha}\right|$ , which gives the claimed bound.
Claim 3: For all  $n = 1,2,3$ ,  $R_0(h_n^\perp) \geq |\lambda|$ . Proof: The function  $q(t)$  with minimal  $R_0$ -cost satisfying the constraint  $q(t) = -\lambda t$  for  $|t| \leq \rho$  is a single ReLU unit plus a constant:  $q(t) = -\lambda [t + \rho]_+ + \lambda \rho$ , which has  $R_0$ -cost  $|\lambda|$ .
Putting the above claims together, we see that
$$
\begin{array}{l} R _ {0} \left(\boldsymbol {h} _ {n}\right) \geq \left| \frac {1 - \lambda \beta}{\beta - \alpha} \right| + \left| \frac {1 - \lambda \alpha}{\beta - \alpha} \right| + | \lambda | (92) \\ \geq \left| \frac {2 - \lambda (\beta + \alpha)}{\beta - \alpha} \right| + | \lambda | (93) \\ \geq \frac {2 - | \lambda | (\beta + \alpha)}{\beta - \alpha} + | \lambda | (94) \\ = \frac {2}{\beta - \alpha} + \frac {- 2 \alpha}{\beta - \alpha} | \lambda | (95) \\ \geq \frac {2}{\beta - \alpha} (96) \\ \end{array}
$$
where in the last inequality we used the fact that  $\frac{-2\alpha}{\beta - \alpha} > 0$  since  $\alpha = -1/2 + \rho < 0$  and  $\beta - \alpha = 3/2 - 2\rho > 0$ .
Therefore,  $R(\pmb{f}) = R(\pmb{h}) \geq R_0(\pmb{h}_1) + R_0(\pmb{h}_2) + R_0(\pmb{h}_3) \geq \frac{6}{\beta - \alpha}$ . Also, the function  $\pmb{f}^*$  given by
$$
\boldsymbol {f} ^ {*} = \boldsymbol {f} _ {1} ^ {*} + \boldsymbol {f} _ {2} ^ {*} + \boldsymbol {f} _ {3} ^ {*}
$$
$$
\pmb {f} _ {n} ^ {*} (\pmb {y}) = \frac {\pmb {x} _ {n}}{\beta - \alpha} ([ \pmb {x} _ {n} ^ {\top} \pmb {y} - \alpha ] _ {+} - [ \pmb {x} _ {n} ^ {\top} \pmb {y} - \beta ] _ {+})
$$
satisfies norm-ball interpolation constraints and  $R(\pmb{f}^{*}) = \frac{6}{\beta - \alpha}$ . Hence,  $\pmb{f}^{*}$  is a min-cost solution.
D Additional simulations
We train a one-hidden-layer ReLU network without a skip connection using the setting we used in Figure 1. As can be seen from Figure 10 we get a similar result for the NN denoiser with and without the skip connection. Therefore in Appendix D.1 we use one-hidden-layer ReLU network without a skip connection.
Figure 10: NN denoiser vs eMMSE denoiser. We trained a one-hidden-layer ReLU network on a denoising task. The clean dataset has four points equally spaced in the interval  $[-5, 5]$ , and the noisy samples are generated by adding zero-mean Gaussian noise with  $\sigma = 1.5$ . We use  $\lambda = 10^{-5}$  in both setting. The figure shows the denoiser output as a function of its input for: (1) NN denoiser trained online using (7) for  $100K$  iterations, (2) NN denoiser trained offline using (8) with  $M = 9000$  and  $20K$  epochs, and (3) the eMMSE denoiser (4).
D.1 MNIST
We use the MNIST dataset to verify various properties. First, the offline and online solutions achieve approximately the same test MSE when trained on a subset of the MNIST dataset (Figure 11). Second, to show that the fact that NN denoiser does not converge to the eMMSE denoiser is not due to approximation error (Figure 12). Lastly, to present the critical noise level in which representation cost minimizer  $f_{1\mathrm{D}}^{*}$  has strictly lower MSE than the eMMSE, for all smaller noise levels (Figure 13).
Figure 11: Online setting vs offline setting for MNIST denoiser. We train a one-hidden layer ReLU network on a subset of  $N = 100$  MNIST images for 10K iterations. We use a Gaussian noise with zero mean and  $\sigma = 0.1$ .
Figure 12: Test loss vs. layer width for MNIST denoiser. We train a one hidden layer ReLU network on MNIST denoiser task using 7 for  $93K$  iterations with a fixed learning rate. We use a Gaussian noise with zero mean and  $\sigma = 0.1$ . The figure shows the test loss vs. layer width.
Figure 13: MSE vs. Noise std. We train a one-hidden layer ReLU network on a subset of  $N = 100$  MNIST images (the range of each pixel is  $[0, 1]$ ) for 10K iterations with fixed LR. We use a Gaussian noise with zero mean. The figure shows the MSE vs noise std  $(\sigma)$  for NN denoiser (orange line) and for eMMSE denoiser (blue line). Note that the eMMSE is dependent on  $\sigma$  (4). For low noise levels, the eMMSE output is one of the training set images. For moderate noise levels, the eMMSE output is a weighted sum of the training set images. For high noise levels, the eMMSE output is the mean of the training set images.
D.2 Three non-colinear training samples
We show in Figures 14 and 15 that for  $N = 3$  training points from the MNIST dataset that forming a triangle in  $d = 2$  dimensions the empirical minimizer obtained using noisy samples and weight decay regularization agrees well with the form of the exact representation cost minimizer predicted by Proposition 2 and Conjecture 4.
D.3 Empirical validation of the subspace assumption
We validated that the following image datasets are (approximately) low rank:
CIFAR10
CINIC10
- Tiny ImageNet (a lower resolution version of ImageNet, enabling us to use SVD)
- BSD (a denoising benchmark composed from 128X1600 patches of size 40X40 cropped from 400 images [Zhang et al., 2017])
Figure 14: Function space view of a denoiser, trained for 3 MNIST data points (Acute Angle). Here, we compare empirical results (left) with our theoretical results (right). We compare the function-space view for inputs from the data plane, with respect to the model output in each of the data directions. For the empirical results, we choose 3 random MNIST data points under the same label, and under the condition that they form an acute triangle  $(64^{\circ}$ , for this figure). We trained a single-layer FC ReLU network with linear residual connection for 1M epochs, with weight decay of  $1E - 8$  (as described in our model), and ADAM optimizer with learning rate  $1E - 5$ .
Table 1: We applied a Singular Value Decomposition (SVD) for each of the above datasets, and calculated the relative number of Singular Values (SV) needed to achieve a given percentile of the energy (for the average vector).
Dataset	95%	99%	99.9%
CIFAR10	0.8%	7.5%	30%
CINIC10	1%	23%	41%
Tiny ImageNet	1.6%	20%	36%
BSD	0.1%	1.6%	4.5%
As can be seen from Table 1 all the datasets that we used are (approximately) low rank.
Figure 15: Function space view of a denoiser, trained for 3 MNIST data points (Obtuse Angle). Here, we compare empirical results (left) with our theoretical results (right). We compare the function-space view for inputs from the data plane, with respect to the model output in each of the data directions. For the empirical results, we added a single data point to the two previously chosen for Figure 14 under the condition that the three points form an obtuse triangle ( $95^{\circ}$ , for this figure). We trained a single-layer FC as described in Figure 14. As predicted, the function we have converged to for data forming an obtuse triangle is noticeably different from the function we converged to when the data was forming an acute triangle.