SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof (which is based on Theorem 1.2. in [Hanin, 2021]) can be found in Appendix B.1. As can be seen from Figure 1, the empirical simulation matches Proposition 1. Proposition 1 states that (17) is a closed-form solution for (8) with minimal representation cost. Notice that the minimal number of neurons needed to represent  $f_{1D}^{*}$  using  $h_{\theta}(y)$  is  $2N - 2$ , which is less than the number of the total training samples  $NM$  for  $M \geq 2$ .
In the case of univariate data, we can prove that the representation cost minimizer  $f_{1D}^{*}$  (linear interpolation) generalizes better than the optimal estimator over the empirical distribution (eMMSE) for low noise levels.
Theorem 1. Let  $y = x + \epsilon$  where  $x \sim p_x(x)$  and  $\epsilon \sim \mathcal{N}(0, \sigma^2)$  where  $x$  and  $\epsilon$  are statistically independent. Then for all datasets such that Assumption 1 holds, and for all density probability distributions  $p_x(x)$  with bounded second moment such that  $p_x(x) > 0$  for all  $x \in [\min_n x_n, \max_n x_n]$ , the following holds,
$$
\lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(\hat {x} ^ {\mathrm {e M M S E}} (y)\right) > \lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(f _ {1 D} ^ {*} (y)\right).
$$
See Appendix B.2 for the proof. We may deduce from Theorem 1 that for each density probability distribution  $p(x)$  there exists a critical noise level for which the representation cost minimizer  $f_{1D}^{*}$  has strictly lower MSE than the eMMSE for all smaller noise levels (this is because the MSE is a continuous function of  $\sigma$ ). The critical noise level can change significantly depending on  $p(x)$ . For example, if  $p(x)$  has a high "mass" in between the training points then the critical noise level is large. However, if the density function has a low "mass" between the training points then the critical noise level is small. In Appendix D we show the MSE vs. the noise level on MNIST denoiser for NN denoiser and eMMSE denoiser (Figure 13). As can be seen there, the critical noise level in this case is not small ( $\sim 5$ ).
Intuitively, the difference between the NN denoiser and the eMMSE denoiser is how they operate on inputs that are not close to any of the clean samples (compared to the noise standard deviation). For such a point, the eMMSE denoiser does not take into account that the empirical distribution of the clean samples does not approximate well their true distribution. Thus, for small noise, it insists on "assigning" it to the closest clean sample point. By contrast, the NN denoiser generalizes better since it takes into account that, far from the clean samples, the data distribution is not well approximated by the empirical sample distribution. Thus, its operation there is near the identity function, with a small contraction toward the clean points, as we discuss next.
3Notice that the training points in Figure 1 are used in the online setting (7) and in the offline setting (8) we observe less noisy samples.
Minimal norm leads to contractive solutions on univariate data. Radhakrishnan et al. [2018] have empirically shown that Auto-Encoders (AE, i.e. NN denoisers without input noise), are locally contractive toward the training samples. Specifically, they showed that the clean dataset can be recovered when iterating the AE output multiple times until convergence. Additionally, they showed that, as we increase the width or the depth of the NN, the network becomes more contractive toward the training examples. In addition, Radhakrishnan et al. [2018] proved that 2-layer AE models are locally contractive under strong assumptions (the weights of the input layer are fixed and the number of neurons goes to infinity). Next, we prove that a univariate shallow NN denoiser is globally contractive toward the clean data points without using the assumptions used by Radhakrishnan et al. [2018] (i.e., the minimizer optimizes over both layers and has a finite number of neurons).
Definition 2. We say that  $\pmb{f}:\mathbb{R}^d\to \mathbb{R}^d$  is contractive toward a set of points  $\{\pmb {x}_n\}_{n = 1}^N$  on  $\mathcal{V}\subseteq \mathbb{R}^d$  if there exists a real number  $0\leq \alpha < 1$  such that for any  $\pmb {y}\in \mathcal{V}$  there exists  $i\in [N]$  so that
$$
\left\| \boldsymbol {f} (\boldsymbol {y}) - \boldsymbol {f} \left(\boldsymbol {x} _ {i}\right) \right\| \leq \alpha \| \boldsymbol {y} - \boldsymbol {x} _ {i} \|. \tag {18}
$$
Lemma 1.  $f_{1D}^{*}(y)$  is contractive toward the clean training points  $\{\pmb{x}_n\}_{n=1}^N$  on  $\mathcal{Y}$ .  $\mathbb{R} \setminus \cup_{n \in [N-1]} \left\{ \frac{x_{n+1} \epsilon_n^{\max} - x_n \epsilon_{n+1}^{\min}}{\epsilon_n^{\max} - \epsilon_{n+1}^{\min}} \right\}$ .
The proof can be found in Appendix B.3.
5 Minimal norm leads to alignment phenomenon on multivariate data
In the multivariate case, min-cost solutions of (14) are difficult to explicitly characterize. Even in the setting of fitting scalar-valued shallow ReLU networks, explicitly characterizing min-cost solutions under interpolation constraints remains an open problem, except in some basic cases (e.g., co-linear data [Ergen and Pilanci, 2021]).
As an approximation to (14) that is more mathematically tractable, we assume the functions being fit are constant and equal to  $\pmb{x}_n$  on a closed ball of radius  $\rho$  centered at each  $\pmb{x}_n$ , i.e.,  $\pmb{f}(\pmb{y}) = \pmb{x}_n$  for all  $\| \pmb{y} - \pmb{x}_n \| \leq \rho$ , such that the balls do not overlap. Letting  $B(\pmb{x}_n, \rho)$  denote the ball of radius  $\rho$  centered at  $\pmb{x}_n$ , we can write this constraint more compactly as  $f(B(\pmb{x}_n, \rho)) = \{\pmb{x}_n\}$ . Consider minimizing the representation cost under this constraint:
$$
\min  _ {\boldsymbol {f}} R (\boldsymbol {f}) \text {s . t .} \boldsymbol {f} (B \left(\boldsymbol {x} _ {n}, \rho\right)) = \left\{\boldsymbol {x} _ {n} \right\} \forall n \in [ N ]. \tag {19}
$$
However, even with this approximation, explicitly describing minimizers of (19) for an arbitrary collection of training samples remains challenging. Instead, to gain intuition, we describe minimizers of (19) assuming the training samples belong to simple geometric structures that yield explicit solutions. Our results reveal a general alignment phenomenon, such that the weights of the representation cost minimizer align themselves with edges and/or faces connecting data points. We also show that approximate solutions of (14) obtained numerically by training a NN denoiser with weight decay match well with the solutions of (19) having exact closed-form expressions.
5.1 Training data on a subspace
In the event that the clean training samples belong to a subspace, we show the representation cost minimizer depends only on the projection of the inputs onto the subspace containing the training data, and its output is also constrained to this subspace.
Theorem 2. Assume the training samples  $\{\pmb{x}_n\}_{n=1}^N$  belong to a linear subspace  $S \subset \mathbb{R}^d$ , and let  $\pmb{P_S} \in \mathbb{R}^{d \times d}$  denote the orthogonal projector onto  $S$ . Then any minimizer  $\pmb{f}^*$  of (19) satisfies  $\pmb{f}^*(\pmb{y}) = \pmb{P_S} \pmb{f}^*(\pmb{P_S} \pmb{y})$  for all  $\pmb{y} \in \mathbb{R}^d$ .
The proof of this result and all others in this section is given in Appendix C.
Note the assumption that the dataset lies on a subspaces is practically relevant, since, in general, large datasets are (approximately) low rank, i.e., lie on a linear subspace [Udell and Townsend, 2019]. In Appendix D we also validated that common image datasets are (approximately) low rank (Table 1).
Specializing to the case co-linear training data (i.e., training samples belonging to a one-dimensional subspace) the min-cost solution is unique and is described by the following corollary:
Figure 2: Predicted (top row) and empirical (bottom row) min-cost NN denoisers for  $N = 3$  clean training samples in  $d = 2$  dimensions. The empirical NN denoisers were trained with weight decay parameter  $\lambda = 10^{-5}$  and  $M = 100$  noisy samples. As predicted by our theory, the ReLU boundaries align either perpendicular to the triangle edges in the obtuse case (left panel), or parallel to the triangle edges (right panel).
Corollary 1. Assume the training samples  $\{\pmb{x}_n\}_{n=1}^N$  are co-linear, i.e.,  $\pmb{x}_n = c_n \pmb{u}$  for some scalars  $c_1 < c_2 < \dots < c_n$  where  $\pmb{u} \in \mathbb{R}^d$  is a unit-vector. Then the minimizer  $\pmb{f}^*$  of (19) is unique and given by  $\pmb{f}^*(\pmb{y}) = \pmb{u} \phi(\pmb{u}^\top \pmb{y})$  where  $\phi: \mathbb{R} \to \mathbb{R}$  has the same form as the 1-D minimizer (17)  $f_{1D}^*$  with  $x_n = c_n$  and  $\epsilon_n^{\max} = -\epsilon_n^{\min} = \rho$ .
In other words, the min-cost solution has a particularly simple form in this case:  $\pmb{f}^{*}(\pmb{y}) = \pmb{u}\phi(\pmb{u}^{\top}\pmb{y})$ , where  $\phi$  is a monotonic piecewise linear function. We call any function of this form a rank-one piecewise linear interpolator. Below we show that many other min-cost solutions can be expressed as superpositions of rank-one piecewise linear interpolators.
5.2 Training data on rays
As an extension of the previous setting, we now consider training data belonging to a union of one-sided rays sharing a common origin. Assuming the rays are well-separated (in a sense made precise below) we prove that the representation cost minimizer decomposes into a sum of rank-one piecewise linear interpolators aligned with each ray.
Theorem 3. Suppose the training samples  $X$  belong to a union of  $L$  rays plus a sample at the origin:  $X = \{\mathbf{0}\} \cup \{\pmb{x}_n^{(1)}\}_{n=1}^{N_1} \cup \dots \cup \{\pmb{x}_n^{(L)}\}_{n=1}^{N_L}$  where  $\pmb{x}_n^{(\ell)} = c_n^{(\ell)}\pmb{u}_\ell$  for some unit vector  $\pmb{u}_\ell$  and constants  $0 < c_1^{(\ell)} < c_2^{(\ell)} < \dots < c_{N_\ell}^{(\ell)}$ . Assume that the rays make obtuse angles with each other (i.e.,  $\pmb{u}_\ell^\top \pmb{u}_k < 0$  for all  $\ell \neq k$ ). Then the minimizer  $\pmb{f}^*$  of (19) is unique and is given by
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \boldsymbol {u} _ {1} \phi_ {1} \left(\boldsymbol {u} _ {1} ^ {\top} \boldsymbol {y}\right) + \dots + \boldsymbol {u} _ {L} \phi_ {L} \left(\boldsymbol {u} _ {L} ^ {\top} \boldsymbol {y}\right), \tag {20}
$$
where  $\phi_{\ell}:\mathbb{R}\to \mathbb{R}$  has the form of the 1-D minimizer (17)  $f_{1D}^{*}$  with  $x_{n} = c_{n}^{(\ell)}$ $\epsilon_n^{\mathrm{max}} = -\epsilon_n^{\mathrm{min}} = \rho$
Additionally, in the Appendix C.2.1 we show that this min-cost solution is stable with respect to small perturbations of the data. In particular, if the training data is perturbed from the rays, the functional form of the min-cost solution only changes slightly, such that the inner and outer-layer weight vectors align with the line segments connecting consecutive data points.
5.3 Special case: training data forming a simplex
Here, we study the representation cost minimizers for  $N \leq d + 1$  training points that form the vertices of a  $(N - 1)$ -simplex, i.e., a  $(N - 1)$ -dimensional simplex in  $\mathbb{R}^d$  (e.g., a 2-simplex is a triangle, a 3-simplex is a tetrahedron, etc.). As we will show, the angles between vertices of the simplex (e.g., an acute versus obtuse triangle in  $N = 3$ ) influences the functional form of the min-cost solution.
Our first result considers one extreme where the simplex has one vertex that makes an obtuse angle with all other vertices (e.g., an obtuse triangle for  $N = 3$ ).
Proposition 2. Suppose the convex hull of the training points  $\{\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_N\} \subset \mathbb{R}^d$  is a  $(N - 1)$ -simplex such that  $\pmb{x}_1$  forms an obtuse angle with all other vertices, i.e.,  $(\pmb{x}_j - \pmb{x}_1)^\top (\pmb{x}_i - \pmb{x}_1) < 0$
for all  $i \neq j$  with  $i, j > 1$ . Then the minimizer  $\mathbf{f}^*$  of (19) is unique, and is given by
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \boldsymbol {x} _ {1} + \sum_ {n = 2} ^ {N} \boldsymbol {u} _ {n} \phi_ {n} \left(\boldsymbol {u} _ {n} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {x} _ {1}\right)\right) \tag {21}
$$
where  $\begin{array}{r}\pmb {u}_n = \frac{\pmb{x}_n - \pmb{x}_1}{\|\pmb{x}_n - \pmb{x}_1\|},\phi_n(t) = s_n([t - a_n]_+ - [t - b_n]_+), \end{array}$  with  $a_{n} = \rho$ $b_{n} = \| \pmb{x}_{n} - \pmb{x}_{1}\| -\rho ,$  and  $s_n = \| \pmb {x}_n - \pmb {x}_1\| /(b_n - a_n)$  for all  $n = 2,\ldots ,N$
This result is essentially a corollary of Theorem 3, since after translating  $x_{1}$  to be the origin, the vertices of the simplex belong to rays making obtuse angles with each other, where there is exactly one sample per ray. Details of the proof are given in Appendix C.2.
At the opposite extreme, we consider the case where every vertex of the simplex is acute, meaning for all  $n = 1,\ldots ,N$  we have  $(\pmb {x}_i - \pmb {x}_n)^\top (\pmb {x}_j - \pmb {x}_n) > 0$  for all  $i,j\neq n$ . In this case, we make following conjecture: the min-cost solution is instead a sum of  $N$  rank-one piecewise linear interpolators, each aligned orthogonal to a different  $(N - 2)$ -dimensional face of the simplex.
Conjecture 1. Suppose the convex hull of the training points  $\{\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_N\} \subset \mathbb{R}^d$  is a  $(N - 1)$ -simplex where every vertex of the simplex is acute. Then the minimizer  $\pmb{f}^*$  of (19) is unique, and is given by
$$
\boldsymbol {f} ^ {*} (\boldsymbol {y}) = \overline {{\boldsymbol {x}}} + \sum_ {n = 1} ^ {N} \boldsymbol {v} _ {n} \phi_ {n} \left(\boldsymbol {u} _ {n} ^ {\top} \left(\boldsymbol {y} - \boldsymbol {z} _ {n}\right)\right) \tag {22}
$$
where:  $\mathbf{z}_n$  is the projection of  $\mathbf{x}_n$  onto the unique  $(N - 2)$ -dimensional face of the simplex not containing  $\mathbf{x}_n$ ;  $\overline{\mathbf{x}}$  is the weighted geometric median of the vertices specified by
$$
\overline {{\boldsymbol {x}}} = \underset {\boldsymbol {x} \in \mathbb {R} ^ {d}} {\arg \min} \sum_ {n = 1} ^ {N} \frac {\left\| \boldsymbol {x} _ {n} - \boldsymbol {x} \right\|}{\left\| \boldsymbol {x} _ {n} - \boldsymbol {z} _ {n} \right\|};
$$
$\pmb{u}_n = \frac{\pmb{x}_n - \pmb{z}_n}{\|\pmb{x}_n - \pmb{z}_n\|}, \pmb{v}_n = \frac{\pmb{x}_n - \overline{\pmb{x}}}{\|\pmb{x}_n - \overline{x}\|};$  and  $\phi_n(t) = s_n([t - a_n]_+ - [t - b_n]_+)$  with  $a_{n} = \rho$ ,  $b_{n} = \left\| \pmb{x}_{n} - \pmb{z}_{n}\right\| -\rho$ , and  $s_n = \| x_n - \overline{x}\| /(b_n - a_n)$ .
Justification for this conjecture is given in Appendix C.3.2. In particular, we prove that the interpolator  $f^{*}$  given in (86) is a min-cost solution in the special case of three training points whose convex hull is an equilateral triangle. If true in general, this would imply a phase transition behavior in the min-cost solution when the simplex changes from having one obtuse vertex to all acute vertices, such that ReLU boundaries go from being aligned orthogonal to the edges connecting vertices, to being aligned parallel with the simplex faces. Figure 2 illustrates this for  $N = 3$  training points forming a triangle in  $d = 2$  dimensions. Moreover, Figure 2 shows that the empirical minimizer obtained using noisy samples and weight decay regularization agrees well with the form of the exact min-cost solution predicted by Proposition 2 and Conjecture 1.
In general, any given vertex of a simplex may make acute angles with some vertices and obtuse angles with others. This case is not covered by the above results. Currently, we do not have a conjectured form of the min-cost solution in this case, and we leave this as an open problem for future work.
6 Related works
Numerous methods have been proposed for image denoising. In last decade NN-based methods achieve state-of-the-art results [Zhang et al., 2017, 2021]. See [Elad et al., 2023] for a comprehensive review of image denoising. Sonthalia and Nadakuditi [2023] empirically showed a double decent behavior in NN denoisers, and theoretically proved it in a linear model. Similar to a denoiser, an Auto-Encoder (AE) is a NN model whose output dimension equals its input dimension, and is trained to match the output to the input. For AE, the typical goal is to learn an efficient lower-dimensional representation of the samples. Radhakrishnan et al. [2018] proved that a single hidden-layer AE that interpolates the training data (i.e., achieves zero loss), projects the input onto a nonlinear span of the training data. In addition, Radhakrishnan et al. [2018] empirically demonstrated that a multi-layer ReLU AE is locally contractive toward the training samples by iterating the AE and showing that the points converge to one of the training samples. Denoising autoencoders inject noise into the input
data in order to learn a good representation [Alain and Bengio, 2014]. The marginalized denoising autoencoder, proposed by Chen et al. [2014], approximates the marginalized loss over the noise (which is equivalent to observing infinitely many noisy samples) by using a Taylor approximation. Chen et al. [2014] demonstrated that by using the approximate marginalized loss we can achieve a substantial speedup in training and improved representation compared to standard denoising AE.
Many recent works aim to characterize function space properties of interpolating NN with minimal representation cost (i.e., min-cost solutions). Building off of the connection between weight decay and path-norm regularization identified in Neyshabur et al. [2015, 2017], Savarese et al. [2019] showed that the representation cost of a function realizable as a univariate two-layer ReLU network coincides with the  $L^1$ -norm of the second derivative of the function. Extensions to the multivariate setting were studied in Ongie et al. [2020], which identified the representation cost of a multivariate function with its  $R$ -norm, a Banach space semi-norm defined in terms of the Radon transform. Related work has extended the  $R$ -norm to other activation functions Parhi and Nowak [2021], vector-valued networks Shenouda et al. [2023], and deeper architectures Parhi and Nowak [2022]. A separate line of research studies min-cost solutions from a convex duality perspective Ergen and Pilanci [2021], including two-layer CNN denoising AEs Sahiner et al. [2021]. Recent work also studies properties of min-cost solutions in the case of arbitrarily deep NNs with ReLU activation Jacot [2022], Jacot et al. [2022].
Despite these advances in understanding min-cost solutions, there are few results explicitly characterizing their functional form. One exception is Hanin [2021], which gives a complete characterization of min-cost solutions in the case of shallow univariate ReLU networks with unregularized bias. This characterization is possible because the univariate representation cost is defined in terms of the 2nd derivative, which acts locally. Therefore, global minimizers can be found by minimizing the representation cost locally over intervals between data points. An extension of these results to the case of regularized bias is studied in Boursier and Flammarion [2023]. In the multivariate setting, the representation cost involves the Radon transform of the function - a highly non-local operation - that complicates the analysis. Parhi and Nowak [2021] prove a representative theorem showing that there always exists a min-cost solution realizable as a shallow ReLU network with finitely many neurons, and Ergen and Pilanci [2021] give an implicit characterization of min-cost NNs the solution to a convex optimization problem, and give explicit solutions in the case of co-linear training features. However, to the best of our knowledge, there are no results explicitly characterizing min-cost solutions in the case of non-colinear multivariate inputs, even for networks having scalar outputs.
7 Discussions
Conclusions. We have explored the elementary properties of NN solutions for the denoising problem, while focusing on offline training of a one hidden-layer ReLU network. When the noisy clusters of the data samples are well-separated, there are multiple networks with zero loss, even in the case of under-parameterization, while having a different representation cost. In contrast, previous theoretical works focused on the over-parametrized regime. In the univariate case, we have derived a closed-form solution to such global minima with minimum representation cost. We also showed that the univariate NN solution generalizes better than the eMMSE denoiser. In the multivariate case, we showed that the interpolating solution with minimal representation cost is aligned with the edges and/or faces connecting the clean data points in several cases.
Limitations. One limitation of our analysis in the multivariate case is that we assume the denoiser interpolates data on a full  $d$ -dimensional ball centered at each clean training sample, where  $d$  is the input dimension. In practical settings, often the number of noisy samples  $M \ll d$ . A more accurate model would be to assume that denoiser interpolates over an  $(M - 1)$ -dimensional disc centered at each training sample. This model may still be a tractable alternative to assuming interpolation of finitely many noisy samples. Also, our results relate to NN denoisers trained with explicit weight decay regularization, which is not always used in practice. However, recent work shows that stable minimizers of SGD must have low representation cost Mulayoff et al. [2021], Nacson et al. [2023], and so some of our analysis may provide insight for unregularized training, as well. Finally, for mathematical tractability, we focussed on the case of fully-connected ReLU networks with one hidden-layer. Extending our analysis to deeper architectures and convolutional neural networks is an important direction for future work.
We thank Itay Hubara for his technical advice and valuable comments on the manuscript. The research of DS was funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of Schmidt Career Advancement Chair in AI. The research of NW was supported by the Israel Science Foundation (ISF), grant no. 1782/22. GO was supported by the National Science Foundation (NSF) CRII award CCF-2153371.