Title: How do Minimum-Norm Shallow Denoisers Look in Function Space?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Denoiser

Denoising

Neural network

Function space
Abstract: Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers --- in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal $\ell^2$ norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. 
We empirically verify this alignment phenomenon on synthetic data and real images.

How do Minimum-Norm Shallow Denoisers Look in Function Space?

Abstract

Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers - in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal ℓ 2 norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. We empirically verify this alignment phenomenon on synthetic data and real images.

1 Introduction

The ability to reconstruct an image from a noisy observation has been studied extensively in the last decades, as it is useful for many practical applications (e.g., Hasinoff et al. [2010]). In recent years, Neural Network (NN) denoisers commonly replace classical expert-based approaches as they achieve substantially better results than the classical approaches (e.g., Zhang et al. [2017]). Beyond this natural usage, NN denoisers also serve as essential building blocks in a variety of common computer vision tasks, such as solving inverse problems [Zhang et al., 2021] and image generation [Song and Ermon, 2019, Ho et al., 2020]. To better understand the role of NN denoisers in such complex applications, we first wish to theoretically understand the type of solutions they converge to.

In practice, when training denoisers, we sample multiple noisy samples for each clean image and minimize the Mean Squared Error (MSE) loss for recovering the clean image. Since we sample numerous noisy samples per clean sample, the number of training samples is typically larger than the number of parameters in the network. Interestingly, even in such an under-parameterized regime, the loss has multiple global minima corresponding to distinct denoiser functions which achieve zero

loss on the observed data. To characterize these functions, we study, similarly to previous works [Savarese et al., 2019, Ongie et al., 2020], the shallow NN solutions that interpolate the training data with minimal representation cost, i.e., where the ℓ 2 -norm of the weights (without biases and skip connections) is as small as possible. This is because we converge to such min-cost solutions when we minimize the loss with a vanishingly small ℓ 2 regularization on these weights.

We first examine the univariate input case: building on existing results [Hanin, 2021], we characterize the min-cost interpolating solution and its generalization to unseen data. Next, we aim to extend this analysis to the multivariate case. However, this is challenging, since, to the best of our knowledge, there are no results that explicitly characterize these min-cost solutions for general multivariate shallow NNs - except in two basic cases. In the first case, the input data is co-linear [Ergen and Pilanci, 2021]. In the second case, the input samples are identical to their target outputs, so the trivial min-cost solution is the identity function. The NN denoisers' training regime is 'near' the second case: there, the input samples are noisy versions of the clean target outputs. Interestingly, we find that this regime leads to non-trivial min-cost solutions far from identity - even with an infinitesimally small input noise. We analytically investigate these solutions here.

Our Contributions. We study the NN solutions in the setting of interpolation of noisy samples with min-cost, in a practically relevant 'low noise regime' where the noisy samples are well clustered. In the univariate case,

We find a closed-form solution for the minimum representation cost NN denoiser. Then, we prove this solution generalizes better than the empirical minimum MSE (eMMSE) denoiser.

We prove this min-cost NN solution is contractive toward the clean data points, that is, applying the denoiser necessarily reduces the distance of a noisy sample to one of the clean samples.

In the multivariate case,

We derive a closed-form solution for the min-cost NN denoiser in multivariate case under various assumptions on the geometric configuration of the clean training samples. To the best of our knowledge, this is the first set of results to explicitly characterize a min-cost interpolating NN in a non-basic multivariate setting.

We illustrate a general alignment phenomenon of min-cost NN denoisers in the multivariate setting: the optimal NN denoiser decomposes into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting clean training samples.

2 Preliminaries and problem setting

The denoising problem. Let y ∈ R d be a noisy observation of x ∈ R d , such that y = x + ϵ where x and ϵ are statistically independent, and E [ ϵ ] = 0 . Commonly, this noise is Gaussian with covariance matrix σ 2 I . The ultimate goal of a denoiser ˆ x ( y ) is to minimize the MSE loss over the joint probability distribution of the data and the noisy observation ('population distribution'), i.e., to minimize

The well-known optimal solution for (1) is the minimum mean square error (MMSE) denoiser, i.e.,

Since we do not have access to the distribution of the data, and hence not to the posterior distribution, we rely on a finite amount of clean data { x n } N n =1 in order to learn a good approximation for the MMSE estimator. One approach is to assume an empirical data distribution and derive the optimal solution of (1), i.e., the empirical minimum mean square error (eMMSE) denoiser,

If the noise is Gaussian with a covariance of σ 2 I , an explicit solution to the eMMSE is given by

An alternative approach to computing the eMMSE directly is to draw M noisy samples for each clean data point, as y n,m = x n + ϵ n,m , where ϵ n,m ∼ N ( 0 , σ 2 I ) are independent and identically distributed, and to minimize the following loss function

Denoiser model and algorithms. In practice, we approximate the optimal denoiser ˆ x ( y ) using a parametric model h θ ( y ) , typically a NN. We focus on a shallow ReLU network model with a skip connection of the form

where θ = (( θ k ) K k =1 ; c , V ) with θ k = ( b k , a k , w k ) ∈ R × R d × R d and c ∈ R d , V ∈ R d × d . We train the model on a finite set of clean data points { x n } N n =1 . The common practical training method is based on an online approach. First, we sample a random batch (with replacement) from the data B ⊆ { x n } N n =1 . Then, for each clean data point x n ∈ B , we draw a noisy sample y n = x n + ϵ n , where ϵ n ∼ N ( 0 , σ 2 I ) are independent of the clean data points and other noise samples. At each iteration t out of T iterations, we update the model parameters according to a stochastic gradient descent rule, with a vanishingly small regularization term λC ( θ ) , that is,

Another training method [Chen et al., 2014] is based on an offline approach. We sample M noisy sample for each clean data point and minimize (5) plus a regularization term

Similarly to previous works [Savarese et al., 2019, Ongie et al., 2020], we assume an ℓ 2 penalty on the weights, but not on the biases and skip connections, i.e.,

Low noise regime. In this paper, we study the solution of the NN denoiser when the clusters of noisy samples around each clean point are well-separated, a setting which we refer to as the 'low noise regime'. This is a rather relevant regime since denoisers are practically used when the noise level is mild. Indeed, common image-denoising benchmarks test on low (but not negligible) noise levels. For instance, in the commonly used denoising benchmark BSD68 [Roth and Black, 2009], the noise level σ = 0 . 1 is in the low noise regime. 1 Moreover, this setting is important, for example, in diffusion-based image generation, since at the end of the reverse denoising process, new images are sampled by denoising smaller and smaller noise levels. 2

3 Basic properties of neural network denoisers

Offline v.s. online NN solutions. NN denoisers are traditionally trained in an online fashion (7), using a finite amount of T iterations. Consequently, only a finite number of noisy samples are used for each clean data point. We empirically observe that the solutions in the offline and online settings are similar. Specifically, in the univariate case, we show in Figure 1 that denoisers based on offline and

NN-Denoiser-online

4

NN-Denoiser-offline

eMMSEdenoiser

Online trainingdata

2

Prediction

0

-2

-6

-4

2

0

2

4

6

online loss functions converge to indistinguishable solutions. For the multivariate case, we observe (Figure 11 in Appendix D) that the offline and online solutions achieve approximately the same test MSEwhen trained on a subset of the MNIST dataset. The comparison is made using the same number of iterations for both training methods, while using much less noisy samples in the offline setting. Evidently, this lower number of samples does not significantly affect the generalization error. Hence, in the rest of the paper, we focus on offline training (i.e., minimizing the offline loss L offline ,M ), as it defines an explicit loss function with solutions that can be theoretically analyzed, as in [Savarese et al., 2019, Ongie et al., 2020].

The empirical MMSE denoiser. The law of large numbers implies that the denoiser minimizing the offline loss L offline ,M approaches the eMMSE estimator in the limit of infinitely many noisy samples,

Therefore, it may seem that for a reasonable number of noise samples M , a large enough model, and small enough regularization, the denoiser we get by minimizing the offline loss (6) will also be similar to the eMMSE estimator. However, Figure 1 shows that the eMMSE solution and the NN solutions (both online and offline) are quite different. The eMMSE denoiser has a much sharper transition and maps almost all inputs to a value of a clean data point. This is because in the case of low noise the eMMSE denoiser (4) approximates the one nearest-neighbor ( 1 -NN) classifier, i.e.,

In contrast, the NN denoiser maps each noisy sample to its corresponding clean sample only in a limited 'noise ball' around the clean point, and interpolates near-linearly between the 'noise balls'. Hence, we may expect that the smoother NN denoiser typically generalizes better than the eMMSE denoiser. We prove that this is indeed true for the univariate case in Section 4.

Why the NN denoiser does not converge to the eMMSE denoiser? Note that the limit in (10) is not practically relevant for the low-level noise regime, since we need an exponentially large M in order to converge in this limit. For example, in the case of univariate Gaussian noise, we have that P ( | ϵ | > t ) ≤ 2 exp( -t 2 2 σ 2 ) , ∀ t > 0 . Therefore, during training, we effectively observe only noisy samples that are in a bounded interval of size 2 σ √ log M around each clean sample (see Figure 1). In other words, in the low-noise regime and for non-exponential M , there is no way to distinguish if the noise is sampled from some distribution with limited support of from a Gaussian distribution. The denoiser minimizing the loss with respect to a bounded-support distribution can be radically different from the eMMSE denoiser in the regions outside the 'noise balls' surrounding the clean samples, where the denoiser function is not constrained by the loss. This leads to a large difference between the NN denoiser and the MMSE estimator.

Alternatively, one may suggest that the NN denoiser does not converge to the eMMSE denoiser due to an approximation error (i.e., the shallow NN's model capacity is too small to approximate the MMSE denoiser). Nevertheless, we provide empirical evidence indicating it is not the case. Specifically, recall that in the low noise regime, the eMMSE denoiser tends to the nearest-neighbor classifier, and

such a solution does not generalize well to test data. Thus, if the NN denoiser would have converged to the eMMSE solution, then its test error would have increased with the network size, in contrast to what we observe in Figure 12 (Appendix D).

Therefore, in order to approximate the eMMSE with a NN, it seems we must have an exponentially large M . Alternatively, we may converge to the eMMSE if we use a loss function marginalized over the Gaussian noise. This idea was previously suggested by Chen et al. [2014], with the goal of effectively increasing the number of noisy samples and thus improving the training performance of denoising autoencoders. Therein, this improvement was obtained by approximating the marginalized loss function by a Taylor series expansion. However, for shallow denoisers, we may actually obtain an explicit expression for this marginalized loss, without any approximation. Specifically, if we assume for simplicity, that the network does not have a linear unit ( V = 0 ) and its bias terms are zero ( c = 0 , b k = 0 ), then the marginalized loss for Gaussian noise, derived in Appendix A, is given by

where w k = ˆ w k ∥ w k ∥ and H ⪰ 0 , ˜ ϕ ( · ) are defined in Appendix A. NN denoisers trained over this loss function will thus tend to the eMMSE solution as the network size is increased. However, as we explained above, this is not necessarily desirable, so we only mention (12) to show exact marginalization is feasible.

Regularization biases toward specific neural network denoisers. To further explore the converged solution for offline training, we note that the offline loss function L offline ,M ( θ ) allows the network to converge to a zero-loss solution. This is in contrast to online training for which each batch leads to new realizations of noisy samples, and thus the training error is never exactly zero. Specifically, consider the low noise regime (well-separated noisy clusters). Then, the network can perfectly fit all the noisy samples using a finite number of neurons (see Section 4 for a more accurate description in the univariate case). Importantly, there are multiple ways to cluster the noisy data points with such neurons, and so there are multiple global training loss minima that the network can achieve with zero loss, each with a different generalization capability. In contrast to the standard case considered in the literature, this holds even in the under-parameterized case (where NM , the total number of noisy samples, is larger than the number of parameters).

Since there are many minima that perfectly fit the training data, we converge to specific minima which also minimize the ℓ 2 regularization we use (even though we assumed it is vanishing). Specifically, in the limit of vanishing regularization C ( θ ) , the minimizers of L offline ,M ( θ ) also minimize the representation cost.

Definition 1. Let h θ : R d → R d denote a shallow ReLU network of the form (6) . For any function f : R d → R d realizable as a shallow ReLU network, we define its representation cost as

and a minimizer of this cost, i.e. the 'min-cost' solution, as

where the second equality in (13) holds due to the 1 -homogeneity of the ReLU activation function, and since the bias terms are not regularized (see [Savarese et al., 2019, Appendix A]). In the next sections, we examine which function we obtain by minimizing the representation cost R ( f ) in various settings.

4 Closed form solution for the NN denoiser function - univariate data

In this section, we prove that NN denoisers that minimize R ( f ) for univariate data have the specific piecewise linear form observed in Figure 1, and we discuss the properties of this form. We observe N

clean univariate data points { x n } N n =1 , s.t. -∞ < x 1 < x 2 < · · · < x N < ∞ , and M noisy samples (drawn from some known distribution) for each clean data point, such that y n,m = x n + ϵ n,m . We denote by ϵ max n the maximal noise seen for data point x n , and by ϵ min n the minimal noise seen for data point x n , i.e.,

and assume the following,

Assumption 1. Assume the data { x n } N n =1 is well-separated after the addition of noise, i.e.,

So we can state the following,

Proposition 1. For all datasets such that Assumption 1 holds, the unique minimizer of R ( f ) is

The proof (which is based on Theorem 1.2. in [Hanin, 2021]) can be found in Appendix B.1. As can be seen from Figure 1, the empirical simulation matches Proposition 1. 3 Proposition 1 states that (17) is a closed-form solution for (8) with minimal representation cost. Notice that the minimal number of neurons needed to represent f ∗ 1 D using h θ ( y ) is 2 N -2 , which is less than the number of the total training samples NM for M ≥ 2 .

In the case of univariate data, we can prove that the representation cost minimizer f ∗ 1 D (linear interpolation) generalizes better than the optimal estimator over the empirical distribution (eMMSE) for low noise levels.

Theorem 1. Let y = x + ϵ where x ∼ p x ( x ) and ϵ ∼ N ( 0 , σ 2 ) where x and ϵ are statistically independent. Then for all datasets such that Assumption 1 holds, and for all density probability distributions p x ( x ) with bounded second moment such that p x ( x ) > 0 for all x ∈ [min n x n , max n x n ] , the following holds,

See Appendix B.2 for the proof. We may deduce from Theorem 1 that for each density probability distribution p ( x ) there exists a critical noise level for which the the representation cost minimizer f ∗ 1 D has strictly lower MSE than the eMMSE for all smaller noise levels (this is because the MSE is a continuous function of σ ). The critical noise level can change significantly depending on p ( x ) . For example, if p ( x ) has a high 'mass' in between the training points then the critical noise level is large. However, if the density function has a low 'mass' between the training points then the critical noise level is small. In Appendix D we show the MSE vs. the noise level on MNIST denoiser for NN denoiser and eMMSE denoiser (Figure 13). As can be seen there, the critical noise level in this case is not small ( ∼ 5 ).

Intuitively, the difference between the NN denoiser and the eMMSE denoiser is how they operate on inputs that are not close to any of the clean samples (compared to the noise standard deviation). For such a point, the eMMSE denoiser does not take into account that the empirical distribution of the clean samples does not approximate well their true distribution. Thus, for small noise, it insists on 'assigning' it to the closest clean sample point. By contrast, the NN denoiser generalizes better since it takes into account that, far from the clean samples, the data distribution is not well approximated by the empirical sample distribution. Thus, its operation there is near the identity function, with a small contraction toward the clean points, as we discuss next.

Minimal norm leads to contractive solutions on univariate data. Radhakrishnan et al. [2018] have empirically shown that Auto-Encoders (AE, i.e. NN denoisers without input noise), are locally contractive toward the training samples. Specifically, they showed that the clean dataset can be recovered when iterating the AE output multiple times until convergence. Additionally, they showed that, as we increase the width or the depth of the NN, the network becomes more contractive toward the training examples. In addition, Radhakrishnan et al. [2018] proved that 2 -layer AE models are locally contractive under strong assumptions (the weights of the input layer are fixed and the number of neurons goes to infinity). Next, we prove that a univariate shallow NN denoiser is globally contractive toward the clean data points without using the assumptions used by Radhakrishnan et al. [2018] (i.e., the minimizer optimizes over both layers and has a finite number of neurons).

Definition 2. We say that f : R d → R d is contractive toward a set of points { x n } N n =1 on Y ⊆ R d if there exists a real number 0 ≤ α < 1 such that for any y ∈ Y there exists i ∈ [ N ] so that

Lemma 1. f ∗ 1 D ( y ) is contractive toward the clean training points { x n } N n =1 on Y = R \ ∪ n ∈ [ N -1] { x n +1 ϵ max n -x n ϵ min n +1 ϵ max -ϵ min } .

5 Minimal norm leads to alignment phenomenon on multivariate data

In the multivariate case, min-cost solutions of (14) are difficult to explicitly characterize. Even in the setting of fitting scalar-valued shallow ReLU networks, explicitly characterizing min-cost solutions under interpolation constraints remains an open problem, except in some basic cases (e.g., co-linear data [Ergen and Pilanci, 2021]).

As an approximation to (14) that is more mathematically tractable, we assume the functions being fit are constant and equal to x n on a closed ball of radius ρ centered at each x n , i.e., f ( y ) = x n for all ∥ y -x n ∥ ≤ ρ , such that the balls do not overlap. Letting B ( x n , ρ ) denote the ball of radius ρ centered at x n , we can write this constraint more compactly as f ( B ( x n , ρ )) = { x n } . Consider minimizing the representation cost under this constraint:

However, even with this approximation, explicitly describing minimizers of (19) for an arbitrary collection of training samples remains challenging. Instead, to gain intuition, we describe minimizers of (19) assuming the training samples belong to simple geometric structures that yield explicit solutions. Our results reveal a general alignment phenomenon, such that the weights of the representation cost minimizer align themselves with edges and/or faces connecting data points. We also show that approximate solutions of (14) obtained numerically by training a NN denoiser with weight decay match well with the solutions of (19) having exact closed-form expressions.

5.1 Training data on a subspace

In the event that the clean training samples belong to a subspace, we show the representation cost minimizer depends only on the projection of the inputs onto the subspace containing the training data, and its output is also constrained to this subspace.

Theorem 2. Assume the training samples { x n } N n =1 belong to a linear subspace S ⊂ R d , and let P S ∈ R d × d denote the orthogonal projector onto S . Then any minimizer f ∗ of (19) satisfies f ∗ ( y ) = P S f ∗ ( P S y ) for all y ∈ R d .

The proof of this result and all others in this section is given in Appendix C.

Note the assumption that the dataset lies on a subpaces is practically relevant, since, in general, large datasets are (approximately) low rank, i.e., lie on a linear subspace [Udell and Townsend, 2019]. In Appendix D we also validated that common image datasets are (approximately) low rank (Table 1).

Specializing to the case co-linear training data (i.e., training samples belonging to a one-dimensional subspace) the min-cost solution is unique and is described by the following corollary:

f

1

(

y

1

,

y

2

)

f

2

(

y

1

,

y

2

)

ReLU boundaries

y

y

2

predicted

denoiser

trained

denoiser

f

1

(

y

1

,

y

2

)

f

2

(

y

1

,

y

2

)

ReLU boundaries

1

Corollary 1. Assume the training samples { x n } N n =1 are co-linear, i.e., x n = c n u for some scalars c 1 < c 2 < · · · < c n where u ∈ R d is a unit-vector. Then the minimizer f ∗ of (19) is unique and given by f ∗ ( y ) = u ϕ ( u ⊤ y ) where ϕ : R → R has the same form as the 1-D minimizer (17) f ∗ 1 D with x n = c n and ϵ max n = -ϵ min n = ρ .

In other words, the min-cost solution has a particularly simple form in this case: f ∗ ( y ) = u ϕ ( u ⊤ y ) , where ϕ is a monotonic piecewise linear function. We call any function of this form a rank-one piecewise linear interpolator . Below we show that many other min-cost solutions can be expressed as superpositions of rank-one piecewise linear interpolators.

5.2 Training data on rays

As an extension of the previous setting, we now consider training data belonging to a union of one-sided rays sharing a common origin. Assuming the rays are well-separated (in a sense made precise below) we prove that the representation cost minimizer decomposes into a sum of rank-one piecewise linear interpolators aligned with each ray.

̸

Theorem 3. Suppose the training samples X belong to a union of L rays plus a sample at the origin: X = { 0 } ∪ { x (1) n } N 1 n =1 ∪ · · · ∪ { x ( L ) n } N L n =1 where x ( ℓ ) n = c ( ℓ ) n u ℓ for some unit vector u ℓ and constants 0 < c ( ℓ ) 1 < c ( ℓ ) 2 < · · · < c ( ℓ ) N ℓ . Assume that the rays make obtuse angles with each other (i.e., u ⊤ ℓ u k < 0 for all ℓ = k ). Then the minimizer f ∗ of (19) is unique and is given by

where ϕ ℓ : R → R has the form of the 1-D minimizer (17) f ∗ 1 D with x n = c ( ℓ ) n , ϵ max n = -ϵ min n = ρ .

Additionally, in the Appendix C.2.1 we show that this min-cost solution is stable with respect to small perturbations of the data. In particular, if the training data is perturbed from the rays, the functional form of the min-cost solution only changes slightly, such that the inner and outer-layer weight vectors align with the line segments connecting consecutive data points.

5.3 Special case: training data forming a simplex

Here, we study the representation cost minimizers for N ≤ d +1 training points that form the vertices of a ( N -1) -simplex, i.e., a ( N -1) -dimensional simplex in R d (e.g., a 2 -simplex is a triangle, a 3 -simplex is a tetrahedron, etc.). As we will show, the angles between vertices of the simplex (e.g., an acute versus obtuse triangle in N = 3 ) influences the functional form of the min-cost solution.

Our first result considers one extreme where the simplex has one vertex that makes an obtuse angle with all other vertices (e.g., an obtuse triangle for N = 3 ).

Proposition 2. Suppose the convex hull of the training points { x 1 , x 2 , ..., x N } ⊂ R d is a ( N -1) -simplex such that x 1 forms an obtuse angle with all other vertices, i.e., ( x j -x 1 ) ⊤ ( x i -x 1 ) < 0

̸

for all i = j with i, j > 1 . Then the minimizer f ∗ of (19) is unique, and is given by

where u n = x n -x 1 ∥ x n -x 1 ∥ , ϕ n ( t ) = s n ([ t -a n ] + -[ t -b n ] + ) , with a n = ρ , b n = ∥ x n -x 1 ∥ -ρ , and s n = ∥ x n -x 1 ∥ / ( b n -a n ) for all n = 2 , ..., N .

This result is essentially a corollary of Theorem 3, since after translating x 1 to be the origin, the vertices of the simplex belong to rays making obtuse angles with each other, where there is exactly one sample per ray. Details of the proof are given in Appendix C.2.

̸

At the opposite extreme, we consider the case where every vertex of the simplex is acute, meaning for all n = 1 , ..., N we have ( x i -x n ) ⊤ ( x j -x n ) > 0 for all i, j = n . In this case, we make following conjecture: the min-cost solution is instead a sum of N rank-one piecewise linear interpolators, each aligned orthogonal to a different ( N -2) -dimensional face of the simplex.

Conjecture 1. Suppose the convex hull of the training points { x 1 , x 2 , ..., x N } ⊂ R d is a ( N -1) -simplex where every vertex of the simplex is acute. Then the minimizer f ∗ of (19) is unique, and is given by

where: z n is the projection of x n onto the unique ( N -2) -dimensional face of the simplex not containing x n ; x is the weighted geometric median of the vertices specified by

u n = x n -z n ∥ x n -z n ∥ , v n = x n -x ∥ x n -x ∥ ; and ϕ n ( t ) = s n ([ t -a n ] + -[ t -b n ] + ) with a n = ρ , b n = ∥ x n -z n ∥ -ρ , and s n = ∥ x n -x ∥ / ( b n -a n ) .

Justification for this conjecture is given in Appendix C.3.2. In particular, we prove that the interpolator f ∗ given in (86) is a min-cost solution in the special case of three training points whose convex hull is an equilateral triangle. If true in general, this would imply a phase transition behavior in the min-cost solution when the simplex changes from having one obtuse vertex to all acute vertices, such that ReLU boundaries go from being aligned orthogonal to the edges connecting vertices, to being aligned parallel with the simplex faces. Figure 2 illustrates this for N = 3 training points forming a triangle in d = 2 dimensions. Moreover, Figure 2 shows that the empirical minimizer obtained using noisy samples and weight decay regularization agrees well with the form of the exact min-cost solution predicted by Proposition 2 and Conjecture 1.

In general, any given vertex of a simplex may make acute angles with some vertices and obtuse angles with others. This case is not covered by the above results. Currently, we do not have a conjectured form of the min-cost solution in this case, and we leave this as an open problem for future work.

6 Related works

Numerous methods have been proposed for image denoising. In last decade NN-based methods achieve state-of-the-art results [Zhang et al., 2017, 2021]. See [Elad et al., 2023] for a comprehensive review of image denoising. Sonthalia and Nadakuditi [2023] empirically showed a double decent behavior in NN denoisers, and theoretically proved it in a linear model. Similar to a denoiser, an Auto-Encoder (AE) is a NN model whose output dimension equals its input dimension, and is trained to match the output to the input. For AE, the typical goal is to learn an efficient lower-dimensional representation of the samples. Radhakrishnan et al. [2018] proved that a single hidden-layer AE that interpolates the training data (i.e., achieves zero loss), projects the input onto a nonlinear span of the training data. In addition, Radhakrishnan et al. [2018] empirically demonstrated that a multi-layer ReLU AE is locally contractive toward the training samples by iterating the AE and showing that the points converge to one of the training samples. Denoising autoencoders inject noise into the input

data in order to learn a good representation [Alain and Bengio, 2014]. The marginalized denoising autoencoder, proposed by Chen et al. [2014], approximates the marginalized loss over the noise (which is equivalent to observing infinitely many noisy samples) by using a Taylor approximation. Chen et al. [2014] demonstrated that by using the approximate marginalized loss we can achieve a substantial speedup in training and improved representation compared to standard denoising AE.

Many recent works aim to characterize function space properties of interpolating NN with minimal representation cost (i.e., min-cost solutions). Building off of the connection between weight decay and path-norm regularization identified in Neyshabur et al. [2015, 2017], Savarese et al. [2019] showed that the representation cost of a function realizable as a univariate two-layer ReLU network coincides with the L 1 -norm of the second derivative of the function. Extensions to the multivariate setting were studied in Ongie et al. [2020], which identified the representation cost of a multivariate function with its R -norm, a Banach space semi-norm defined in terms of the Radon transform. Related work has extended the R -norm to other activation functions Parhi and Nowak [2021], vector-valued networks Shenouda et al. [2023], and deeper architectures Parhi and Nowak [2022]. A separate line of research studies min-cost solutions from a convex duality perspective Ergen and Pilanci [2021], incuding two-layer CNN denoising AEs Sahiner et al. [2021]. Recent work also studies properties of min-cost solutions in the case of arbitrarily deep NNs with ReLU activation Jacot [2022], Jacot et al. [2022].

Despite these advances in understanding min-cost solutions, there are few results explicitly characterizing their functional form. One exception is Hanin [2021], which gives a complete characterization of min-cost solutions in the case of shallow univariate ReLU networks with unregularized bias. This characterization is possible because the univariate representation cost is defined in terms of the 2nd derivative, which acts locally. Therefore, global minimizers can be found by minimizing the representation cost locally over intervals between data points. An extension of these results to the case of regularized bias is studied in Boursier and Flammarion [2023]. In the multivariate setting, the representation cost involves the Radon transform of the function - a highly non-local operation - that complicates the analysis. Parhi and Nowak [2021] prove a representer theorem showing that there always exists a min-cost solution realizable as a shallow ReLU network with finitely many neurons, and Ergen and Pilanci [2021] give an implicit characterization of min-cost NNs the solution to a convex optimization problem, and give explicit solutions in the case of co-linear training features. However, to the best of our knowledge, there are no results explicitly characterizing min-cost solutions in the case of non-colinear multivariate inputs, even for networks having scalar outputs.

7 Discussions

Conclusions. We have explored the elementary properties of NN solutions for the denoising problem, while focusing on offline training of a one hidden-layer ReLU network. When the noisy clusters of the data samples are well-separated, there are multiple networks with zero loss, even in the case of under-parameterization, while having a different representation cost. In contrast, previous theoretical works focused on the over-parametrized regime. In the univariate case, we have derived a closed-form solution to such global minima with minimum representation cost. We also showed that the univariate NN solution generalizes better than the eMMSE denoiser. In the multivariate case, we showed that the interpolating solution with minimal representation cost is aligned with the edges and/or faces connecting the clean data points in several cases.

Limitations. One limitation of our analysis in the multivariate case is that we assume the denoiser interpolates data on a full d -dimensional ball centered at each clean training sample, where d is the input dimension. In practical settings, often the number of noisy samples M ≪ d . A more accurate model would be to assume that denoiser interpolates over an ( M -1) -dimensional disc centered at each training sample. This model may still be a tractable alternative to assuming interpolation of finitely many noisy samples. Also, our results relate to NN denoisers trained with explicit weight decay regularization, which is not always used in practice. However, recent work shows that stable minimizers of SGD must have low representation cost Mulayoff et al. [2021], Nacson et al. [2023], and so some of our analysis may provide insight for unregularized training, as well. Finally, for mathematical tractability, we focussed on the case of fully-connected ReLU networks with one hidden-layer. Extending our analysis to deeper architectures and convolutional neural networks is an important direction for future work.

Acknowledgments

Wethank Itay Hubara for his technical advice and valuable comments on the manuscript. The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of Schmidt Career Advancement Chair in AI. The research of NW was supported by the Israel Science Foundation (ISF), grant no. 1782/22. GO was supported by the National Science Foundation (NSF) CRII award CCF-2153371.

References

Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research , 15(1):3563-3593, 2014.

Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. arXiv preprint arXiv:2303.01353 , 2023.

Minmin Chen, Kilian Weinberger, Fei Sha, and Yoshua Bengio. Marginalized denoising autoencoders for nonlinear representations. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1476-1484, Bejing, China, 22-24 Jun 2014. PMLR. URL https: //proceedings.mlr.press/v32/cheng14.html .

Michael Elad, Bahjat Kawar, and Gregory Vaksman. Image denoising: The deep learning revolution and beyond-a survey paper. SIAM Journal on Imaging Sciences , 16(3):1594-1654, 2023. doi: 10.1137/23M1545859. URL https://doi.org/10.1137/23M1545859 .

Tolga Ergen and Mert Pilanci. Convex geometry and duality of over-parameterized neural networks. The Journal of Machine Learning Research , 22(1):9646-9708, 2021.

Boris Hanin. Ridgeless interpolation with shallow relu networks in 1 d is nearest neighbor curvature extrapolation and provably generalizes on lipschitz functions. arXiv preprint arXiv:2109.12960 , 2021.

Samuel W Hasinoff, Frédo Durand, and William T Freeman. Noise-optimal capture for high dynamic range photography. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 553-560. IEEE, 2010.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

William C Horrace. Moments of the truncated normal distribution. Journal of Productivity Analysis , 43:133-138, 2015.

Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In International Conference on Learning Representations (ICLR) , 2022.

Arthur Jacot, Eugene Golikov, Clement Hongler, and Franck Gabriel. Feature learning in l_2-regularized dnns: Attraction/repulsion and sparsity. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 6763-6774, 2022.

G. Manjunath and Stefan Wilhelm. Moments calculation for the doubly truncated multivariate normal density. Journal of Behavioral Data Science , 1(1):17-33, May 2021. doi: 10.35566/jbds/ v1n1/p2. URL https://jbds.isdsa.org/index.php/jbds/article/view/9 .

Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems (NeurIPS) , 34: 17749-17761, 2021.

Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability in multivariate shallow relu networks. In International Conference on Learning Representations (ICLR) , 2023.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory (COLT) , pages 1376-1401. PMLR, 2015.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017.

Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=H1lNPxHKDH .

Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research , 22(1):1960-1999, 2021.

Rahul Parhi and Robert D Nowak. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal on Mathematics of Data Science , 4(2):464-489, 2022.

Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Caroline Uhler. Memorization in overparameterized autoencoders. arXiv preprint arXiv:1810.10333 , 2018.

Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. arXiv preprint arXiv:2305.19693 , 2023.

Stefan Roth and Michael J. Black. Fields of experts. International Journal of Computer Vision , 82(2):205-229, 2009. doi: 10.1007/s11263-008-0197-6. URL https://doi.org/10.1007/ s11263-008-0197-6 .

Arda Sahiner, Morteza Mardani, Batu Ozturkler, Mert Pilanci, and John M. Pauly. Convex regularization behind neural reconstruction. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=VErQxgyrbfn .

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pages 2667-2690. PMLR, 2019.

Joseph Shenouda, Rahul Parhi, Kangwook Lee, and Robert D Nowak. Vector-valued variation spaces and width bounds for DNNs: Insights on weight decay regularization. arXiv preprint arXiv:2305.16534 , 2023.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Rishi Sonthalia and Raj Rao Nadakuditi. Training data size induced double descent for denoising feedforward neural networks and the role of training noise. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=FdMWtpVT1I .

Madeleine Udell and Alex Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science , 1(1):144-160, 2019.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7): 3142-3155, 2017.

Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):6360-6376, 2021.

A Marginalized loss

In this section, we derive the marginal loss for the case of a 1-hidden layer ReLU neural network. The loss function is,

We denote by

Note that H ⪰ 0 since H is a covariance matrix. Thus we get,

Lemma 2. In the case of the ReLU activation function and Gaussian noise the following holds,

where φ, Φ are the density and cumulative distribution of standard normal distribution, and

where,

2.45

2.5

2.55

2.6

2.65

0

100

200

300

400

500

600

700

Analytical Expectation

(a)

1.45

1.5

1.55

1.6

0

100

200

300

400

500

600

700

800

900

Analytical Expectation

(b)

and and we obtain,

Let ( z 1 , z 2 ) ⊤ be a Gaussian random vector with 4

Notice that we denote by P ( z 1 > 0 , z 2 > 0; µ ( i,j ) , Σ ( i,j ) ) the probability that z 1 > 0 , z 2 > 0 where ( z 1 , z 2 ) ⊤ ∼ N ( µ ( i,j ) , Σ ( i,j ) ) .

Note that given x > 0 the distribution of x is truncated normal [Horrace, 2015]. Therefore,

Note that given x , y is a Gaussian random vector. Therefore, given x ,

then,

Given z 1 > 0 , z 2 > 0 the distribution of ( z 1 , z 2 ) ⊤ is truncated multivariate normal distribution [Manjunath and Wilhelm, 2021], therefore,

where f ( -µ ; 0 , Σ ) is a density function of of Gaussian random vector with mean vector 0 and covariance matrix Σ at the point -µ , and

We denote by Λ = Σ -1 thus,

Thus,

Similarly, we obtain,

Therefore,

2.8

2.85

2.9

2.95

3

0

100

200

300

400

500

600

700

800

Analytical Expectation

(a)

13.9

13.95

14

14.05

14.1

14.15

0

100

200

300

400

500

600

Analytical Expectation

(b)

(

-

4

17

average. Figure (a) is for

µ

=

,

Σ

=

13

-

9

-

9

8

, the normalized error is

-

E

= 0

.

0093%

.

Next, we present in Figures 3 and 4 a numerical evaluation of (23) and (24). As can be seen, the Monte-Carlo simulations verify the analytical results.

B Proofs of Results in Section 4

B.1 Proof of Proposition 1

Proof. Let

such that Assumption 1 holds. We define a reduced dataset, which only contains the noisy samples with the most extreme noises

We define the ℓ 2 penalty on the weights,

According to Theorem 1.2. in [Hanin, 2021], since we have opposite discrete curvature on the intervals [ x 1 + ϵ max 1 , x 2 + ϵ min 1 ] · · · [ x N -1 + ϵ max 1 , x N + ϵ min 1 ] ,

where

Also note that,

̸

since f ∗ 1 D ( y ) interpolates all the points in D , and if we assume by contradiction that C ∗ > inf θ,K { C ( θ, K ) | ∀ ( y, x ) ∈ D h θ ( y ) = x } then C ∗ = inf θ,K { C ( θ, K ) | ∀ ( y, x ) ∈ ¯ D h θ ( y ) = x } , thus contradicting Theorem 1.2. in [Hanin, 2021]. Note that the minimal number of neurons needed to represent f ∗ 1 D ( y ) is 2 N -2 . So, if K ≥ 2 N -2 then f ∗ 1 D is the minimizer of the representation cost (i.e., f ∗ 1 D ∈ arg min f R ( f ) ).

)

(

)

|

E

¯

E

|

B.2 Proof of Theorem 1

First we prove the following lemmas.

Lemma 3. Let y = x + σϵ where x, ϵ ∈ R , σ > 0 then,

Proof. Notice that the following holds,

In addition,

so we obtain,

Therefore,

Lemma 4. Let y = x + σϵ where x ∈ R , σ > 0 and ϵ ∼ N (0 , 1) then,

Proof.

Note that,

̸

̸

Similarly,

thus ∃ M 0 > 0 ∀ σ > 0

almost surely. Note that,

Therefore, by the Dominated convergence theorem we obtain

Similarly,

Since ˆ x eMMSE ( y ) and ˆ x 1 -NN ( y ) are bounded and E [ x ] < ∞ . Therefore, we get

Next, we prove Theorem 1.

Proof. Assume, without loss of generality, that N = 2 . Let p x ( x ) be a probability density function with bounded second moment such that p x ( x ) > 0 for all x ∈ [ x 1 , x 2 ] . According to Lemma 4

So we need to prove that

For the case of N = 2 , the training set includes two data points { x 1 , x 2 } . So we get,

where max m ∈ [ M ] ϵ 1 ,m = ∆ 1 > 0 , min m ∈ [ M ] ϵ 2 ,m = ∆ 2 < 0 . Note that ˆ x 1 -NN ( y ) = f ∗ 1 D ( y ) for all y ∈ ( -∞ , x 1 +∆ 1 ) ∪ ( x 2 +∆ , ∞ ) and E [ x ] < ∞ , E [ x 2 ] < ∞ . Therefore,

First, we derive (25):

thus by the Dominated convergence theorem we obtain

since p x ( x ) > 0 for all x ∈ [ x 1 , x 2 ] . Next, we derive (26)

Note that,

E x,ϵ [ 1 { x + σϵ ∈ [ x 1 +∆ 1 , x 2 +∆ 2 ] } ( f ∗ 1 D ( x + σϵ ) -x ) 2 ] < E x,ϵ [ ( f ∗ 1 D ( x + σϵ ) -x ) 2 ] <

thus by the Dominated convergence theorem we obtain

since

B.3 Proof of Lemma 1

Proof. First we prove that for all y ∈ ∪ n ∈ [ N -1] { x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 } f ∗ 1 D ( y ) = y . We find the intersection between the line f ∗ 1 D ( y ) = y and the linear part of (17) (the third branch)

which holds according to Assumption 1.

which holds according to Assumption 1.

Next we prove that f ∗ 1 D ( y ) is contractive toward a set of the clean datapoints { x n } N n =1 on R \ ∪ n ∈ [ N -1] { x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 } . In the case where y ∈ ( -∞ , x 1 + ϵ min 1 ] or y ∈ ∪ n ∈ [ N ] [ x n + ϵ min n , x n + ϵ max n ] or y ∈ [ x N + ϵ max N , ∞ ) f ∗ 1 D ( y ) ∈ { x n } N n =1 therefore (18) holds for all 0 ≤ α < 1 . In the case where y ∈ ∪ n ∈ [ N -1] [ x n + ϵ max n , x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 ) we choose i = n ,

Note that

Since,

𝑥

!"#

𝑥

!

𝑥

!$#

𝑓

#%

∗

(𝑦)

𝑦

𝑥

!"#

+𝜖

!"#

'(!

𝑥

!

+𝜖

!

')*

𝑓 𝑦

= y

𝑥

!"#

𝜖

!

')*

-𝑥

!

𝜖

!"#

'(!

𝜖

!

')*

-𝜖

!"#

'(!

There exists 0 < γ 1 < 1 such that

since for y ∈ ∪ n ∈ [ N -1] [ x n + ϵ max n , x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 ) f ∗ 1 D ( y ) is bellow the line f ( y ) = y because f ∗ 1 D ( y ) is an affine function with slope larger than 1 and f ∗ 1 D ( x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 ) = x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 . Therefore there exists 0 < α 1 < 1

since,

In the case where y ∈ ∪ n ∈ [ N -1] ( x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 , x n +1 + ϵ min n +1 ] we choose i = n +1 ,

There exists 0 < γ 2 < 1 such that

since for y ∈ ∪ n ∈ [ N -1] ( x n +1 ϵ max n -x n ϵ min n +1 ϵ max n -ϵ min n +1 , x n +1 + ϵ min n +1 ] f ∗ 1 D ( y ) is above the line f ( y ) = y because max min max min f ∗ 1 D ( y ) is an affine function with slope larger than 1 and f ∗ 1 D ( x n +1 ϵ n -x n ϵ n +1 ϵ max n -ϵ min n +1 ) = x n +1 ϵ n -x n ϵ n +1 ϵ max n -ϵ min n +1 . Therefore there exists 0 < α 2 < 1

since,

Therefore (18) holds for α = max { α 1 , α 2 } .

C Proofs of Results in Section 5

Let f : R d → R d be any function realizable as a shallow ReLU network. Consider any parametrization of f given by f ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ] + + V y + c . We say such a parametrization is a minimal representative of f if ∥ w k ∥ 2 = 1 and a k = 0 for all k = 1 , ..., K , and R ( f ) = ∑ K k =1 ∥ a k ∥ 2 . In particular, the units making up a minimal representative must be distinct in the sense that the hyperplanes describing ReLU boundaries H k = { x ∈ R d : w ⊤ k x + b k = 0 } are distinct, which implies that no units can be cancelled or combined.

̸

We will also make use of the following lemma, which shows that representation costs are invariant to a translation of the training samples, assuming the function is suitably translated.

Lemma 5. Let f : R d → R d be any function realizable as a shallow ReLU net satisfying norm-ball interpolation constraints f ( B ( x n ; ρ )) = { x n } for all n = 1 , ..., N . Let x 0 ∈ R d . Then the function g ( y ) = f ( y -x 0 ) + x 0 is such that R ( g ) = R ( f ) and g ( B ( x n + x 0 ; ρ )) = { x n + x 0 } for all n = 1 , ..., N .

Proof. First we show g ( B ( x n + x 0 ; ρ )) = { x n + x 0 } for all n = 1 , ..., N . Fix any n , and let y ∈ B ( x n + x 0 ; ρ ) . Then y = y ′ + x 0 for some y ′ ∈ B ( x n ; ρ ) , and so g ( y ) = f ( y ′ ) + x 0 = x n + x 0 , as claimed.

To show that R ( g ) = R ( f ) , let f ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ] + + V y + c be any minimal representative of f . Then

with ˜ b k = b k -w ⊤ k x 0 and ˜ c = c + x 0 -V x 0 . And so R ( g ) ≤ ∑ K k =1 ∥ a k ∥ 2 = R ( f ) . A parallel argument with the roles of g and f reversed shows that R ( f ) ≤ R ( g ) , and so R ( f ) = R ( g ) .

In particular, this lemma shows that if f ∗ ( y ) is a norm-ball interpolating representation cost minimizer of the training samples { x n } N n =1 , then g ∗ ( y ) = f ∗ ( y -x 0 )+ x 0 is a norm-ball interpolating min-cost solution of the translated training samples { x n + x 0 } N n =1 .

C.1 Training data belonging to a subspace

The proof of Theorem 2 is a direct consequence of the following two lemmas:

̸

Lemma6. Suppose the clean training samples { x n } N n =1 belong to a r -dimensional subspace S ⊂ R d , and let P S denote the orthogonal projector onto S . Let f ( y ) be any function realizable as a shallow ReLU network satisfying f ( B ( x n , ρ )) = { x n } for all n = 1 , ..., N . Define g ( y ) = P S f ( x ) . Then g ( B ( x n , ρ )) = { x n } for all n = 1 , ..., N , and R ( g ) ≤ R ( f ) , with strict inequality if f = g .

Proof. First, for any y ∈ B ( x n , ρ ) we have g ( y ) = P S f ( y ) = P S x n = x n . Therefore, g ( B ( x n , ρ )) = { x n } for all n = 1 , ..., N .

Now, let f ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ] + + V y + c be a minimal representative of f . Then g ( y ) = ∑ K k =1 Pa k [ w ⊤ k y + b k ] + + P ( V y + c ) . Since ∥ Pa k ∥ ≤ ∥ a k ∥ for all k , we have R ( g ) ≤ ∑ K k =1 ∥ P S a k ∥ ≤ ∑ K k =1 ∥ a k ∥ = R ( f ) .

̸

Now we show f = g implies R ( g ) < R ( f ) . Observe that if any of the outer-layer weight vectors a k ̸∈ S then ∥ P S a k ∥ < ∥ a k ∥ , which implies R ( g ) < R ( f ) . Hence, it suffices to show that f = g implies some a k ̸∈ S .

̸

̸

First, consider the case where P S V = V and P S c = c . Then in this case f = g if only if ∑ K k =1 ( P S a k -a k )[ w ⊤ k y + b k ] + = 0 for all y ∈ R d , which implies P S a k = a k for some k , or equivalently a k ̸∈ S .

̸

̸

̸

̸

Next, assume either P S V = V or P S c = c . Fix any training sample index n , and let A n denote the index set of units active over the ball B ( x n , ρ ) . Then, since f ( y ) is constant for all y ∈ B ( x n , ρ ) , the Jacobian ∂f ( y ) = ∑ k ∈ A n a k w ⊤ k + V = 0 for all y ∈ B ( x n , ρ ) . This gives V = -∑ k ∈ A n a k w ⊤ k . Therefore, if P S V = V then at least one of the a k ̸∈ S . On the other hand, if P S c = c then for all y ∈ B ( x n , ρ ) we have

̸

̸

which implies c = x n -∑ k ∈ A n b k a k , and since x n ∈ S this implies some a k ̸∈ S , proving the claim.

̸

Lemma 7. Suppose the clean training samples { x n } N n =1 belong to an r -dimensional subspace S ⊂ R d , and let P S denote the orthogonal projector onto S . Let f be any network satisfying f ( B ( x n , ρ )) = { x n } . Define h ( y ) = f ( P S y ) . Then h ( B ( x n , ρ )) = { x n } and R ( h ) ≤ R ( f ) , with strict inequality if h = f .

Proof. Define P -1 S ( B ( x n , ρ )) := { y ∈ R d : P S y ∈ B ( x n , ρ ) } , i.e., the set of points in R d whose projection onto S is contained in B ( x n , ρ ) . Then clearly h ( P -1 S ( B ( x n , ρ ))) = { x n } . Also, by properties of norm-balls, we have B ( x n , ρ ) ⊂ P -1 S ( B ( x n , ρ )) . Therefore, h ( B ( x n , ρ )) = { x n } for all n = 1 , ..., N .

Next, let f ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ] + + V y + c be a minimal representative of f . Then h ( y ) = ∑ K k =1 a k [( P S w k ) ⊤ y + b k ] + + V P S y + c . Since ∥ P S w k ∥ ≤ ∥ w k ∥ for all k , we have R ( h ) ≤ ∑ K k =1 ∥ a k ∥∥ P S w k ∥ ≤ ∑ K k =1 ∥ a k ∥∥ w k ∥ = R ( f ) .

̸

Finally, we show that if f = h , then R ( h ) < R ( f ) . Observe that if any of the inner-layer weight vectors w k ̸∈ S then ∥ Pw k ∥ < ∥ w k ∥ , which implies R ( h ) < R ( f ) . Hence, it suffices to show that f = h implies some w k ̸∈ S . First, consider the case V P S = V . Then f = h if and only if P S w k = w k for some k , or equivalently, w k ̸∈ S . Next, consider the case V P S = V . Fix any training sample index n , and let A n denote the index set of units active over B ( x n , ρ ) . Then, since f ( y ) is constant for all y ∈ B ( x n , ρ ) , the Jacobian ∂f ( x ) = ∑ k ∈ A n a k w ⊤ k + V = 0 for all y ∈ B n ( x n , ρ ) . This gives V = -∑ k a k w ⊤ k . Therefore, if V P S = V (i.e., at least one row of V is not contained in S ) then at least one of the w k ̸∈ S , proving the claim.

Finally, we now give the proof Theorem 2 and Corollary 1.

Proof of Theorem 2. Let f ∗ be any min-cost solution. Applying both Lemma 6 and 7 we see it must be the case that f ∗ ( y ) = P S f ∗ ( P S y ) for all y ∈ R d , since otherwise the representation cost of f ∗ could be reduced.

Proof of Corollary 1. Suppose S is one-dimensional, i.e., S = span { u } for some unit vector u ∈ R d . Theorem 2 shows we can express any min-cost solution f ∗ as

where

is such that R ( f ∗ ) = R ( ϕ ) . Therefore, minimizing the representation cost subject to norm-ball interpolation constraints is reduces to a univariate problem:

where c n = u ⊤ x n . The minimizing ϕ ∗ is unique and coincides with the 1-D denoiser f ∗ 1 D in (17) with x n = c n and ϵ max n = -ϵ min n = ρ .

̸

̸

̸

̸

̸

C.2 Data along rays

We begin with a key lemma that is central to the proof of Theorem 3.

̸

Lemma 8. Suppose { u i } n i =1 ⊂ R d is a collection of n > 1 unit vectors such that u ⊤ i u j < 0 for all i = j , and w is a unit vector such that u ⊤ i w > 0 for all i = 1 , ..., n . Let a ∈ R d be any vector. Then,

Before giving the proof of Lemma 8, we first prove an auxiliary result.

̸

Lemma9. Let a ∈ R d be a unit vector, and suppose { u i } n i =1 ⊂ R d is a collection of unit vectors such that u ⊤ i a > 0 for all i and u ⊤ i u j < 0 for all i = j . Let b = ∑ n i =1 u i u ⊤ i a . Then ∥ b -a / 2 ∥ ≤ 1 / 2 with strict inequality if n > 1 .

Proof. It suffices to show b ⊤ a ≥ b ⊤ b , since if this is the case then

which also holds with strict inequality when b ⊤ a > b ⊤ b .

First, if n = 1 , then b = a ⊤ u 1 u ⊤ 1 , and so b ⊤ b = a ⊤ u 1 u ⊤ 1 u 1 u ⊤ 1 a = a ⊤ u 1 u ⊤ 1 a = b ⊤ a . Now assume n > 1 . Then we have

̸

The first and last equalities hold by definition. The second equality holds because each u i is a unit vector. The last inequality holds because for i = j

i j since u ⊤ i u j < 0 and a ⊤ u i , a ⊤ u j > 0 by assumption.

Now we give the proof of Lemma 8.

Proof of Lemma 8. Let v = ∑ n i =1 | u ⊤ i a | u i . Then we have

WLOG, we may assume ∥ a ∥ = 1 , in which case the lemma reduces to showing S = v ⊤ w < 1 . By the Cauchy-Schwartz inequality, it suffices to show ∥ v ∥ < 1 . Toward this end, let us write v = v + + v -where v + = ∑ i : u ⊤ i a > 0 | u ⊤ i a | u i = ∑ i : u ⊤ i a > 0 u i u ⊤ i a and v -= ∑ i : u ⊤ i a < 0 | u ⊤ i a | u i = ∑ i : u ⊤ i a < 0 u i u ⊤ i ( -a ) . By Lemma 9 we have ∥ v + -a / 2 ∥ ≤ 1 / 2 and ∥ v -+ a / 2 ∥ ≤ 1 / 2 , and so

Also, if either of the index sets I + := { i : u ⊤ i a > 0 } or I -:= { i : u ⊤ i a < 0 } has cardinality greater than one then the lemma above guarantees ∥ v + -a / 2 ∥ < 1 / 2 or ∥ v + -a / 2 ∥ < 1 / 2 , and so ∥ v ∥ < 1 , which gives S < 1 .

It remains to show S < 1 when I + or I -has cardinality less than or equal to one, i.e., | I + | ≤ 1 or | I -| ≤ 1 . We consider the various possibilities:

Case 1: | I + | = 0 & | I -| = 0 . Then v = 0 and so S = v ⊤ w = 0 < 1 .

̸

Case 2: | I + | = 1 & | I -| = 0 or | I + | = 0 & | I -| = 1 . Then v = | u ⊤ i a | u i for some i , and | u ⊤ j a | = 0 for all j = i . By way of contradiction, assume that S = v ⊤ w = | u ⊤ i a | u ⊤ i w = 1 . Since u i , a , and w are all unit vectors, the only way this is possible is if | u ⊤ i a | = 1 and u ⊤ i w = 1 , which implies a = ± w and u i = w , and so a = ± u i . However, this shows that u ⊤ j u i = 0 for all j = i , contradicting our assumption that u ⊤ j u i < 0 for all i = j . Therefore, S < 1 in this case as well.

̸

Case 3: | I + | = 1 & | I -| = 1 . Then v = | u ⊤ i a | u i + | u ⊤ j a | u j for some i ∈ I + and j ∈ I -, and so

where the last inequality follows by the fact that ∥ a ∥ = 1 . Finally, since the matrix u i u ⊤ i -u j u ⊤ j is symmetric, its operator norm ∥ u i u ⊤ i -u j u ⊤ j ∥ coincides with the maximum eigenvalue (in absolute value). Eigenvectors in the span of u i and u j have the form v = c i u i + c j u j where c i , c j satisfy the equation

where λ ∈ R is the corresponding eigenvalue. Expanding and equating coefficients gives the system

which has non-trivial solutions iff

and so | λ | < 1 . Therefore, ∥ v ∥ ≤ ∥ u i u ⊤ i -u j u ⊤ j ∥ < 1 as claimed. And so S ≤ ∥ v ∥ < 1 .

R

1

R

2

w

R

1

R

2

P

1

w

P

2

w

Proof of Theorem 3. Let f ∗ be any representation cost minimizer. We prove the claim by showing that if f ∗ fails to have the form specified in (20) then it is possible to construct a norm-ball interpolant h whose units are aligned with the rays that has strictly smaller representation cost than f ∗ .

̸

First, let f ∗ ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ]+ V x + c be any minimal representative of f ∗ . By properties of minimal representatives, none of the ReLU boundary sets { y ∈ R d : w ⊤ k y + b k = 0 } can intersect any of the norm-balls centered at the training samples, since otherwise f ∗ would be non-constant on one of the norm-balls. Also, we may alter this parameterization in such a way that the active set of every unit avoids the ball centered at the origin B 0 := B ( 0 , ρ ) without changing the representation cost. In particular, suppose the active set of the k th unit contains B 0 . Then, using the identity [ t ] + = t -[ -t ] + for all t ∈ R , we have

for all y ∈ R d . The active set of the reversed unit -a k [ -w ⊤ k y -b k ] + does not intersect B 0 . Therefore, after applying this transformation to all units whose active sets contain B 0 , we can write f ∗ = g ∗ + ℓ where g ∗ is a sum of ReLU units whose active sets do not contain B 0 and ℓ is an affine function that combines the original affine part V y + c with a sum of linear units of the form a k ( w ⊤ k y + b k a k ) arising from the transformation above. However, because of the interpolation constraint f ( B 0 ) = { 0 } , and by the fact that no units in g ∗ are active over B 0 , for all y ∈ B 0 we have

which implies ℓ ≡ 0 , i.e., f ∗ ( y ) = g ∗ ( y ) for all y . Finally, note that the inner- and outer-layer weight vectors on the reversed units change only in sign. Therefore, this re-parameterization does not change the representation cost, and so is also a minimal representative.

Now we construct a new norm-ball interpolant h as follows. Let f ℓ be the function defined as the sum of all units making up the re-parametrized f ∗ that are active on at least one norm-ball centered on the ℓ th ray. Also, let P ℓ = u ℓ u ⊤ ℓ ∈ R d × d denote the orthogonal projector onto the ℓ th ray. Define h = ∑ L ℓ =1 h ℓ where h ℓ ( y ) := P ℓ f ℓ ( P ℓ y ) . Put in words, h is constructed by 'splitting' any units active over multiple rays into a sum of multiple units aligned with each ray; see Figure 6 for an illustration.

̸

First, we prove that h satisfies norm-ball interpolation constraints. Let a [ w ⊤ y + b ] + denote any unit belonging to f ℓ . Since this unit is active on a norm-ball centered on ray ℓ , this implies that w ⊤ u ℓ > 0 , and since the unit is inactive on B 0 we must have b < 0 . Also, because we assume u ⊤ ℓ u m < 0 for any m = ℓ , we see that the projected unit P ℓ a [ w ⊤ P ℓ y + b ] + = P ℓ a [( w ⊤ u ℓ ) u ⊤ ℓ y + b ] + is active on norm-balls centered on ray ℓ and inactive on norm-balls centered on any other ray. In particular, if y belongs to a norm-ball centered on ray ℓ , then h m ( y ) = 0 for all m = ℓ . Therefore, if y ∈ B ( x ( ℓ ) n , ρ ) where x ( ℓ ) n denotes a training sample along ray ℓ , then

̸

which shows that h satisfies norm-ball interpolation constraints, as claimed.

̸

Next, we show that if h = f ∗ , then h has strictly smaller representation cost. Let u ( y ) = a [ w ⊤ y + b ] + denote any ReLU unit belonging to f ∗ . Since we assume ∥ w ∥ = 1 , the contribution of unit u to the representation cost of f ∗ is ∥ a ∥ . Let R⊂{ 1 , 2 , ..., L } index the subset of rays ℓ for which unit u is active over at least one norm-ball centered on that ray. In constructing the interpolant h , the unit u get mapped to a sum of |R| units:

whose contribution to the representation cost of h is bounded above by

(a) Training samples along rays

(b) Perturbed training samples

If |R| > 1 , then by Lemma 8 guarantees C < ∥ a ∥ . And if |R| = 1 then C = | u ⊤ ℓ a || u ⊤ ℓ w | ≤ ∥ a ∥ , with strict inequality unless w , a belong to the span of u ℓ . This implies that any min-cost solution must have all its units aligned with the rays, i.e., f ∗ must have the form f ∗ ( y ) = ∑ L ℓ =1 u ℓ ϕ ℓ ( u ⊤ ℓ y ) , where each ϕ ℓ is a univariate function. The representation cost of any function f ∗ in this form is the sum of the (1-D) representation costs of the ϕ ℓ . Therefore, ϕ ℓ must also be the 1-D min-cost solution of the norm-ball constraints projected onto ray ℓ , and so must have the form specified by (17).

C.2.1 Perturbations of samples along rays

As an extension of the above setting, suppose the training samples along rays x ( ℓ ) n have been slightly perturbed, i.e., we consider training samples ˜ x ( ℓ ) n = x ( ℓ ) n + ϵ ( ℓ ) n for some vectors ϵ ( ℓ ) n with ∥ ϵ ( ℓ ) n ∥ < δ for some sufficiently small δ > 0 . In particular, we make the following two assumptions:

̸

(A1) Let ∆ ℓ := { ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 } N ℓ n =1 be the collection of difference vectors between successive perturbed points along the ℓ th ray (with ˜ x ( ℓ ) 0 := 0 ). Assume that for all v ℓ ∈ ∆ ℓ and for all v k ∈ ∆ k with k = ℓ , we have v ⊤ ℓ v k < 0 .

(A2) If H is any halfspace not containing the origin that contains the ball B (˜ x ( ℓ ) n ; ρ ) , then H also contains all successive balls along that ray, i.e., H contains B (˜ x ( ℓ ) m ; ρ ) for m ≥ n .

Note that if the original points x ( ℓ ) n belong to rays that satisfy the conditions of Theorem 3, then for sufficiently small δ the perturbed samples ˜ x ( ℓ ) n will satisfy assumptions (A1)&(A2) above.

We show in this case the norm-ball interpolating representation cost minimizer is a perturbed version of the min-cost solution for the unperturbed samples as identified in Theorem 3. In particular, the ReLU boundaries align with the line segments connecting successive points along the rays. See Figure 7 for an illustration in the case of L = 2 rays.

Proposition 3. Suppose the training samples X are a perturbation of data belong to a union of L rays plus a sample at the origin, i.e., X = { 0 }∪{ ˜ x (1) n } N 1 n =1 ∪·· ·∪{ ˜ x ( L ) n } N L n =1 where ˜ x ( ℓ ) n = c ( ℓ ) n u ℓ + ϵ ( ℓ ) n for some unit vector u ℓ , constants 0 < c ( ℓ ) 1 < c ( ℓ ) 2 < · · · < c ( ℓ ) N ℓ , and vectors ϵ ( ℓ ) n . Assume (A1)&(A2) above hold. Then the minimizer f ∗ of (19) is unique and is given by

Proof. Let f ∗ be any min-cost solution. Following the same steps as in the proof of Theorem 3, there is a minimal representative of f ∗ having the form f ∗ ( y ) = ∑ k a k [ w ⊤ k y + b k ] + where the active sets of all units in this representation have empty intersection with the ball B ( 0 , ρ ) .

Let f ( ℓ ) n denote the sum of all units in f ∗ whose active set boundary { y ∈ R d : w ⊤ k y + b k = 0 } intersects the line segment connecting the training samples ˜ x ( ℓ ) n -1 and ˜ x ( ℓ ) n . By assumption (A2), this is equivalent to the sum of units that are active over balls B ( ˜ x ( ℓ ) m , ρ ) for m ≥ n , and inactive for the balls with m<n . In particular, for y ∈ B ( ˜ x ( ℓ ) n , ρ ) , we have f ∗ ( y ) = ∑ m ≤ n f ( ℓ ) m ( y ) . This implies f ( ℓ ) 1 ( y ) = ˜ x 1 for all y ∈ B ( ˜ x ( ℓ ) 1 , ρ ) . Likewise, f ( ℓ ) 2 ( y ) = ˜ x ( ℓ ) 2 -˜ x ( ℓ ) 1 for all y ∈ B ( ˜ x ( ℓ ) 2 ; ρ ) , and so on, such that for all n = 1 , ..., N ℓ we have f ( ℓ ) n ( y ) = ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 for all y ∈ B ( ˜ x ( ℓ ) n ; ρ ) .

Now we show how to construct a new interpolating function h having representation cost less than or equal to f ∗ . Let u ( ℓ ) n = ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 ∥ ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 ∥ and define P ( ℓ ) n = u ( ℓ ) n ( u ( ℓ ) n ) ⊤ , the orthogonal projector onto the span of the difference vector ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 . Note that the map y ↦→ P ( ℓ ) n y + ˜ x ( ℓ ) n -1 is projection onto the affine line connecting samples ˜ x ( ℓ ) n -1 and ˜ x ( ℓ ) n . Consider the function

where

Put in words, h is constructed by aligning all inner-layer and outer-layer weights of units in f ∗ with the line segment connecting successive data points over which that unit first activates. See Figure 8 for an illustration.

x

0

= 0

f

1

f

2

S

1

S

2

h

1

(

y

) =

P

1

f

1

(

P

1

y

)

x

0

= 0

S

1

S

2

B

1

B

2

B

1

h

2

(

y

)

=

P

2

f

2

(

P

2

y

+

x

1

)

h

=

h

1

+

h

2

B

2

f

1

(

y

) =

x

1

f

2

(

y

) = 0

f

=

f

1

+

f

2

f

1

(

x

) =

x

1

f

2

(

x

) =

x

2

-

x

1

Align units

h

1

(

y

) =

x

1

h

2

(

y

) = 0

h

1

(

y

) =

x

1

h

2

(

y

) =

x

2

-

x

1

Now we show that h satisfies norm-ball interpolation constraints. By assumption (A1), the function h ( ℓ ) n is non-zero only on the norm-balls B ( ˜ x ( ℓ ) m , ρ ) for m ≥ n . This implies that for all y ∈ B ( ˜ x ( ℓ ) n ; ρ ) we have h ( y ) = ∑ m ≤ n h ( ℓ ) n ( y ) .

Observe that h ( B ( 0 , ρ )) = { 0 } since all functions h ( ℓ ) n are zero on B ( 0 , ρ ) . Also, for all y ∈ B ( ˜ x ( ℓ ) 1 ; ρ ) we have P ( ℓ ) 1 y ∈ B ( ˜ x ( ℓ ) 1 ; ρ ) , and so

Similarly, the only terms in h active for y ∈ B ( ˜ x ( ℓ ) 2 ; ρ ) are h ( ℓ ) 1 and h ( ℓ ) 2 , for which h ( ℓ ) 1 ( y ) = x ( ℓ ) 1 , and h ( ℓ ) 2 ( y ) = P ( ℓ ) 2 f ( ℓ ) 2 ( P ( ℓ ) 2 y + ˜ x ( ℓ ) 1 ) = P ( ℓ ) 2 ( ˜ x ( ℓ ) 2 -˜ x ( ℓ ) 1 ) = ˜ x ( ℓ ) 2 -˜ x ( ℓ ) 1 . This gives

Iterating this procedure for all n = 1 , ..., N ℓ we see that if y ∈ B ( ˜ x ( ℓ ) n , ρ ) then h ( ℓ ) n ( y ) = ˜ x ( ℓ ) n -˜ x ( ℓ ) n -1 , and so h ( y ) = ∑ m ≤ n h ( ℓ ) m ( y ) = x ( ℓ ) n , as claimed.

̸

Following steps similar to the proof of Theorem 3, we can show that R ( h ) ≤ R ( f ∗ ) and with strict inequality if h = f ∗ . First, if any unit u ( y ) = a [ w ⊤ y + b ] + in f ∗ is active over balls centered on multiple rays R ⊂ { 1 , 2 , ..., L } , then in the construction of h the unit u gets mapped to a sum of multiple units:

̸

for some n ℓ with 1 ≤ n ℓ ≤ N ℓ . The contribution of the sum of these units to the representation cost of h is less than or equal to C = ∑ ℓ ∈R | ( u ( ℓ ) n ℓ ) ⊤ a || ( u ( ℓ ) n ℓ ) ⊤ w | . By assumption (A1), we know ( u ( ℓ ) m ) ⊤ u ( p ) n < 0 for all p = ℓ and for all m,n . Therefore, by Lemma 8 we know C < ∥ a ∥ . This shows f ∗ cannot have any units active over balls centered on different rays, since otherwise h has strictly smaller representation cost. Therefore, we have shown f ∗ as f ∗ = ∑ L ℓ =1 ∑ N ℓ n =1 f ( ℓ ) n . Additionally, we see that h ( ℓ ) n = f ( ℓ ) n , i.e., all inner- and outer-layer weight vectors of units in f ( ℓ ) n are aligned with u ( ℓ ) n , since otherwise the representation cost of f ∗ could be reduced. Therefore, each f ( ℓ ) n must have the form f ( ℓ ) n ( y ) = u ( ℓ ) n ψ ( ℓ ) n (( u ( ℓ ) n ) ⊤ y ) . Minimizing the ψ ( ℓ ) n separately under interpolation constraints, we arrive at the unique solution given in (79).

C.3 Simplex Data

C.3.1 Simplex with one obtuse vertex

Proof of Proposition 2. Consider training samples x 1 , x 2 , ..., x N ∈ R d whose convex hull is a ( N -1) -simplex such that x 1 makes an obtuse angle with all other vertices, i.e., ( x n -x 1 ) ⊤ x 1 < 0 for all n = 2 , ..., N . By Lemma 5, f ∗ is a norm-ball interpolating min-cost solution if and only if g ∗ ( y ) = f ∗ ( y -x 1 ) + x 1 is a norm-ball interpolating min-cost solution for the translated points 0 , x 2 -x 1 , ..., x N -x 1 ∈ R d . The latter configuration satisfies the hypotheses of Theorem 3 with a single training sample per ray. Therefore, the min-cost solution g ∗ of the translated points has units whose inner- and outer-layer weight vetors are aligned with x n -x 1 , n = 2 , ..., N , and likewise for f ∗ , since it is translation of g ∗ .

C.3.2 Simplex with all acute vertices

Justification of Conjecture 1. For concreteness we focus on the case of three points x 1 , x 2 , x 3 ∈ R d whose convex hull is an acute triangle, and let B 1 , B 2 , B 3 be open balls of radius ρ centered at x 1 , x 2 , x 3 (respectively).

Let f be any norm-ball interpolating min-cost solution, and let f ( y ) = ∑ k a k [ w ⊤ k y + b k ] + V x + c be any minimal representative of f .

First, by properties of minimal representatives, none of the ReLU boundary sets H k = { y ∈ R d : w ⊤ k y + b k = 0 } intersect any of the balls centered at the training samples. Also, the active set of each unit in f must contain either one or two norm-balls, since otherwise the unit is either inactive over all balls or active over all balls, in which cases the unit can be removed or absorbed into unregularized linear part while strictly reducing the representation cost. By 'reversing units', as in the proof of Theorem 3, we may transform the parameterization in such a way that the active set of every unit contains exactly one ball, and do this without changing the representation cost.

After this transformation, we may write

where f i is a sum of units active only on the ball B i and no others, and where ℓ ( y ) = Ay + b is an affine function. Then we have

Finding the min-cost solution then amounts to minimizing of the representation cost of f 1 , f 2 , f 3 subject to the above constraints. This is made challenging by the fact that the constraints are coupled together by the parameters A and b of the affine part. However, under the assumption A = 0 , we prove below that the min-cost solution must have the conjectured form f ∗ as given in (86). However, since we cannot a priori assume A = 0 , this does not constitute a full proof.

Before proceeding, we give a lemma that will be used to lower bound R ( f ) (assuming A = 0 ).

Lemma 10. Suppose g is a sum of ReLU units such that g = 0 on a closed convex region C ⊂ R d and g is constant and equal to c on a closed ball B ⊂ R d , and g has a minimal representative where all its units are active on B . Then

where dist ( B,C ) = min y ∈ B, x ∈ C ∥ y -x ∥ .

Proof. Let g ( y ) = ∑ k a k [ w ⊤ k y + b k ] + be a minimal representative where each unit is active on B . For g to be constant on B it must be the case that ∑ k a k w ⊤ k = 0 . Let y = y 0 be any value for which ∥ ∂g ( y 0 ) ∥ , the operator norm of the Jacobian of g , is maximized. Since g is piecewise linear, we see that its Lipschitz constant Lip ( g ) = ∥ ∂g ( y 0 ) ∥ . Let I 0 be the set of indices of active units at y 0 so that ∂g ( y 0 ) = ∑ k ∈ I 0 a k w ⊤ k , and let I 1 be the complementary index set. Then because ∑ k a k w ⊤ k = 0 , we have ∑ k ∈ I 0 a k w ⊤ k = -∑ k ∈ I 1 a k w ⊤ k , and so ∥ ∑ k ∈ I 0 a k w ⊤ k ∥ = ∥ ∑ k ∈ I 1 a k w ⊤ k ∥ . Therefore,

Finally,

Combining this with the previous inequality gives the claim.

̸

Now, returning to the proof of the main claim, for n = 1 , 2 , 3 , let C n be the closed convex region given by the intersection of all closed half-planes containing the balls B j and B k , j = n , and k = n , j = k . By assumption, f n vanishes on C n and f n is constant and equal to x n -b on the closed ball B n . So, by Lemma 10, we see that

where δ n = dist ( B n , C n ) . Therefore, for all b ∈ R d we have

and so,

̸

̸

Let x ∈ R d be the minimizer of which is independent of b .

Finally, simple calculations show that the conjectured min-cost solution f ∗ specified in (86) has representation cost achieving this lower bound. Therefore, under the assumption A = 0 , f ∗ is a min-cost solution.

Now we prove that, in the special case where the convex hull of the training points is an equilateral triangle, the norm-ball interpolator f ∗ identified in Conjecture 1 is a min-cost solution (though not necessarily the unique min-cost solution). In this case, we may also give f ∗ a more explicit form, as detailed below.

Proposition 4. Suppose the convex hull of the training points x 1 , x 2 , x 3 ∈ R d is an equilateral triangle. Assume the norm-balls B n := B ( x n , ρ ) centered at each training point have radius ρ < ∥ x n -x 0 ∥ / 2 , n = 1 , 2 , 3 , where x 0 = 1 3 ( x 1 + x 2 + x 3 ) is the centroid of the triangle. Then a minimizer f ∗ of (19) is given by

Proof. By translation and scale invariance of min-cost solutions, without loss of generality we may assume x 1 , x 2 , x 3 ∈ R d are unit-norm vectors and mean-zero (i.e., the triangle centroid x 0 = 0 ). In this case, the assumption on ρ translates to ρ < 1 / 2 . Additionally, it suffices to prove the claim in the case d = 2 . This is because, if x 1 , x 2 , x 3 ∈ R d are the vertices of an equilateral triangle whose centroid is at the origin, then these points are contained in a two-dimensional subspace S ⊂ R d . And if we let P ∈ R d × 2 be a matrix whose columns are an orthonormal basis for S , then by Theorem 2, f is a min-cost solution if and only if f ( y ) = Pf 0 ( P ⊤ y ) , where f 0 : R 2 → R 2 is a min-cost solution under the constraints f 0 ( B ( P ⊤ x n , ρ )) = { P ⊤ x n } for all n = 1 , 2 , 3 . Therefore, the problem reduces to finding a min-cost solution of the projected points P ⊤ x 1 , P ⊤ x 2 , P ⊤ x 3 ∈ R 2 whose convex hull is an equilateral triangle in R 2 .

̸

So now let f : R 2 → R 2 be any min-cost solution under the assumption x 1 , x 2 , x 3 ∈ R 2 are unit-norm, have zero mean, and ρ < 1 / 2 . By reversing units as necessary, there exists a minimal representative of f that can be put in the form f ( y ) = f 1 ( y ) + f 2 ( y ) + f 3 ( y ) + Ay + c , such that f n ( y ) is a sum of ReLU units, all of which are active on B n , and all of which are inactive on B j for j = n .

̸

Let Q be the reflection matrix Q = 2 x 1 x ⊤ 1 -I , which reflects points across the line spanned by x 1 . In particular, y ∈ B 1 then Qy ∈ B 1 while if y ∈ B 2 then Qy ∈ B 3 (and vice versa). Also, Qx 1 = x 1 , Qx 2 = x 3 , and Qx 3 = x 2 . Define g ( y ) = 1 2 ( f ( y ) + Q -1 f ( Qy ) ) . Then it is easy to check that g satisfies interpolation constraints, and since Q is unitary, we have R ( g ) ≤ 1 2 R ( f )+ 1 2 R ( Q ◦ f ◦ Q -1 ) = 1 2 R ( f )+ 1 2 R ( f ) = R ( f ) . Furthermore, since f is a min-cost solution, we must have R ( g ) = R ( f ) . Additionally, since none of the units making up f are active over more than one ball, neither are the units making up g , which implies no pair of units belonging to g combine to form an affine function. Therefore, we may write g ( y ) = g 1 ( y ) + g 2 ( y ) + g 3 ( y ) + By + v , where B = A + Q -1 AQ , such that each g n is a sum of ReLU units, all of which are active on B n , and all of which are inactive on B j for j = n .

Let U be a rotation matrix by 120 degrees such that x 2 = Ux 1 , x 3 = Ux 2 , and x 1 = Ux 3 . Consider the symmetrized version of g given by h ( y ) := 1 3 ( g ( y ) + U -1 g ( Uy ) + U -2 g ( U 2 y ) ).

Then plugging in b = x above, we have the lower bound

̸

Since for all y ∈ B n we have Uy ∈ B n +1 (with indices understood modulo 3 ), it is easy to verify that h satisfies interpolation constraints. Also, since U is unitary, we have R ( h ) ≤ R ( g ) ≤ R ( f ) , which implies R ( h ) = R ( f ) since f is a min-cost solution. Again, since none of the units making up g are active over more than one ball, neither are the units making up h , which implies no pair of units belonging to h combine to form an affine function. And so, we may write h ( y ) = h 1 ( y ) + h 2 ( y ) + h 3 ( y ) + C + u , such that each h n is a sum of ReLU units, all of which are active on B n , and all of which are inactive on B j for j = n .

Observe that u = 1 3 ( I + U -1 + U -2 ) v = 0 . Also, we have C = 1 3 ( B + U -1 BU + U -2 BU 2 ) . This implies C commutes with U , because it is easy to see that C = U -1 CU . Additionally, C commutes with Q , because it is easy to see B = Q -1 BQ , and by properties of rotations/reflections we have UQ = QU -1 , which implies

which shows C commutes with Q . Now we show that any matrix C that commutes with both U and Q is a scaled identity matrix. Let x ⊥ 1 denote a unit-vector perpendicular to x 1 , such that { x 1 , x ⊥ 1 } form an orthonormal basis for R 2 . First, we know Q has eigenvectors x 1 and x ⊥ 1 with eigenvalues 1 and -1 , respectively. And so QCx 1 = CQx 1 = Cx 1 , while QCx ⊥ 1 = CQx ⊥ 1 = -Cx 1 , which implies Cx 1 = a x 1 and Cx ⊥ 1 = b x ⊥ 1 for some a, b ∈ R , i.e., x 1 and x ⊥ 1 are eigenvectors of C with eigenvalues a and b , respectively. Furthermore, we have CUx 1 = UCx 1 = a Ux 1 and CUx ⊥ 1 = UCx ⊥ 1 = b Ux 1 . This implies Ux 1 and Ux ⊥ 1 are also eigenvectors of C with eigenvalues a and b , respectively. But since x 1 and Ux 1 are linearly independent, and likewise so are x ⊥ 1 and Ux ⊥ 1 , the only way this could be true is if a = b , i.e., every vector in R 2 is an eigenvector of C for the same eigenvalue λ ∈ R , which implies C = λ I for some λ ∈ R .

Therefore, we have shown that every min-cost solution f maps to a min-cost solution h of the form

̸

and h ⊥ n ( t ) = -λt if | t | ≤ ρ .

Claim 1: For all n = 1 , 2 , 3 , R 0 ( h n ) ≥ R 0 ( h ∥ n ) + R 0 ( h ⊥ n ) . Proof: Let h n ( y ) = ∑ k a k [ w ⊤ k y + b k ] + + c be any realization of h n , whose representation cost is C = ∑ k 1 2 ( ∥ a k ∥ 2 + ∥ w k ∥ 2 ) . Then realizations of h ∥ n and h ⊥ n are given by

whose representation costs C ∥ and C ⊥ , respectively, are given by

and by the Pythagorean Theorem we see that C = C ∥ + C ⊥ . Therefore, C ≥ R 0 ( h ∥ n ) + R 0 ( h ⊥ n ) and finally minimizing over all realizations of h n gives the claim.

Proof: By results in Savarese et al. [2019], R 0 ( h ∥ n ) ≥ R 0 ( p ) where p ( t ) is the function satisfying the same constraints as h ∥ n given in (87) while linearly interpolating over the interval t ∈ [ α, β ] . From the formula R 0 ( p ) = max { ∫ | p ′′ ( t ) | dt, | p ′ ( ∞ ) + p ′ ( -∞ ) |} as established in Savarese et al. [2019], we can show directly that R 0 ( p ) = ∣ ∣ ∣ 1 -λβ β -α ∣ ∣ ∣ + ∣ ∣ ∣ 1 -λα β -α ∣ ∣ ∣ , which gives the claimed bound.

Claim 3: For all n = 1 , 2 , 3 , R 0 ( h ⊥ n ) ≥ | λ | . Proof : The function q ( t ) with minimal R 0 -cost satisfying the constraint q ( t ) = -λt for | t | ≤ ρ is a single ReLU unit plus a constant: q ( t ) = -λ [ t + ρ ] + + λρ , which has R 0 -cost | λ | .

̸

for some λ ∈ R , where each h n is a sum of ReLU units, all of which are active on B n and all of which are inactive on B j , j = n . In particular, h n ( y ) = x n -λ y for all y ∈ B n and h n ( y ) = 0 for all y ∈ C n , where C n is the intersection of all half-planes containing the balls B j , j = n .

This means we can write h ( y ) = ∑ K k =1 a k [ w ⊤ k y + b k ] + + λ I and h n ( y ) = ∑ k ∈A n a k [ w ⊤ k y + b k ] + for some index sets A 1 , A 2 , A 3 partitioning { 1 , ..., K } so that R ( h ) = ∑ K k =1 ∥ a k ∥ = ∑ 3 n =1 ∑ k ∈A n ∥ a k ∥ . Let R 0 ( · ) denote the representation cost of a function computed without an unregularized linear part, i.e., define R 0 analogously to R except where the model class h θ in (6) is constrained to have V = 0 . Then, since the realizations of the h n functions considered above do not have a linear part, we see that ∑ k ∈A n ∥ a k ∥ ≥ R 0 ( h n ) and so R ( h ) = ∑ 3 n =1 ∑ k ∈A n ∥ a k ∥ ≥ R 0 ( h 1 ) + R 0 ( h 2 ) + R 0 ( h 3 ) .

Nowweshow how lower bound R 0 ( h n ) for all n = 1 , 2 , 3 . Let x ⊥ n denote a unit-vector perpendicular to x n , such that { x n , x ⊥ n } form an orthonormal basis for R 2 . Consider the univariate functions h ∥ n ( t ) := x ⊤ n h n ( x n t ) and h ⊥ n ( t ) := ( x ⊥ n ) ⊤ h n ( x ⊥ n t + x n ) . Here h ∥ n is the projection of h onto the line spanned by x n , and h ⊥ n is a projection onto the line perpendicular to x n passing through the point x n . In particular, by the constraints on h n , we see that h ∥ n and h ⊥ n satisfy the constraints

Putting the above claims together, we see that

where in the last inequality we used the fact that -2 α β -α > 0 since α = -1 / 2 + ρ < 0 and β -α = 3 / 2 -2 ρ > 0 .

Therefore, R ( f ) = R ( h ) ≥ R 0 ( h 1 ) + R 0 ( h 2 ) + R 0 ( h 3 ) ≥ 6 β -α . Also, the function f ∗ given by ∗ ∗ ∗ ∗

where

satisfies norm-ball interpolation constraints and R ( f ∗ ) = 6 β -α . Hence, f ∗ is a min-cost solution.

D Additional simulations

We train a one-hidden-layer ReLU network without a skip connection using the setting we used in Figure 1. As can be seen from Figure 10 we get a similar result for the NN denoiser with and without the skip connection. Therefore in Appendix D.1 we use one-hidden-layer ReLU network without a skip connection.

NN-Denoiser-online

4

NN-Denoiser-offline

eMMSEdenoiser

Onlinetrainingdata

2

Prediction

0

-6

-4

-2

0

2

4

6

D.1 MNIST

We use the MNIST dataset to verify various properties. First, the offline and online solutions achieve approximately the same test MSE when trained on a subset of the MNIST dataset (Figure 11). Second, to show that the fact that NN denoiser does not converge to the eMMSE denoiser is not due to approximation error (Figure 12). Lastly, to present the critical noise level in which representation cost minimizer f ∗ 1D has strictly lower MSE than the eMMSE, for all smaller noise levels (Figure 13).

Online

0.8

Offline with M=100

Offline with M=10

0.7

E

0.6

MSE

Test

0.5

T

0.4

0.3

0.2

0

2000

4000

6000

8000

10000

Iterations

0.0070

0.0069

loss

0.0068

Test

0.0067

0.0066

0.0065

1000020000300004000050000600007000080000

Layerwidth

0

1

2

3

4

5

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

MSE

eMMSE

NN Denoiser

Noise std (

)

D.2 Three non-colinear training samples

We show in Figures 14 and 15 that for N = 3 training points from the MNIST dataset that forming a triangle in d = 2 dimensions the empirical minimizer obtained using noisy samples and weight decay regularization agrees well with the form of the exact representation cost minimizer predicted by Proposition 2 and Conjecture 4.

D.3 Empirical validation of the subspace assumption

We validated that the following image datasets are (approximately) low rank:

CIFAR10

CINIC10

Tiny ImageNet (a lower resolution version of ImageNet, enabling us to use SVD)

BSD (a denoising benchmark composed from 128 X 1600 patches of size 40 X 40 cropped from 400 images [Zhang et al., 2017])

Results-Direction 0

Theory- Direction 0

3

1

-2

-3

Direction1

Direction 1

3

C.8

1.6

-1

6

As can be seen from Table 1 all the datasets that we used are (approximately) low rank.

Results- Direction 0

Theory-Direction 0

2.4

D

Direction1

Direction 1

2

0.8

1.6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the shape and properties of one layer networks on a denoising problem. Specifically, the authors compute a closed-form solution for a NN trained offline with regularization and show that its ability to generalize is better than the eMMSE estimator -- which acts as a piece-wise constant function -- in a low-noise regime and in 1d. Then they provide results on multivariate cases where the training data points are contained in a lower dimensional subspace, showing that the image of the network is also contained in the subspace, and deriving closed-forms for specific cases where the data are aligned.\n\nSTRENGTHS:\nRegarding the presentation, the paper is clearly written and each theoretical result is well-explained, making the reading easier. Regarding the content, the theorems seem new and provide interesting insights on the behavior of simple networks on the denoising problem. The authors provide a complete study of the univariate case when the noise level is reasonable, and give insights on what happens in higher dimensions. Numerical illustrations support the theoretical claims.\n\nWEAKNESSES:\nThe study is limited to low noise levels (noisy samples supports can't intersect), and it would have been interesting to have the authors opinion on whether removing assumption 1 change or not the behavior shown in figure 1 (theoretical development or even empirical illustrations). In particular, theorem 1 is valid in the limit when $\\sigma \\to 0$, while we would have preferred a \"threshold\" (as the authors mention, there exists a threshold for the strict inequality but it is potentially very small).\n\nQUESTIONS:\n* Could the authors illustrate what happens in practice (in 1d or in higher dimension) when the noise level increases ? In particular, do we still have a gain using a NN rather than the eMMSE ?\n\n* Is it possible to give bounds on $\\sigma$ for which theorem 1 is valid ? Otherwise, could the authors provide MSE curves highlighting the difference between NN and eMMSE depending on $\\sigma$ (in the case where the noisy samples support are disjoint and in the general case) ?\n\nLIMITATIONS:\nThe limitations are addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how two-layer ReLU denoising networks look like when minimizing common losses such as the empirical minimum mean square error, an empirical alternative that draws finitely many noisy samples, as well as representation costs that find the data interpolating function with minimal norm of the weights. It is proven that - for univariate data - representation cost minimizers generalize better than empirical minimum mean square error minimizers in the low noise regime. Moreover, the solution to the representation costs is explicitly stated in the univariate case as well as in the multivariate case for specific geometric configurations, including one conjecture. Small numerical experiments illustrate that theory is correct and the conjecture is justified.\n\nSTRENGTHS:\n- The paper is very well written and easy to follow despite its theoretical nature\n- It characterizes several interesting and important properties of shallow denoising networks in the low-noise regime for univariate data.\n- It gives inspiring insights on the behavior in the multivariate case (under particular assumptions). \n- It encourages further research in the investigated direction by posting one (partially open) conjecture, as well as by accurately stating several limitations (tight to very interesting directions of future research). \n- It contributes to our understanding of denoisers in function space\n\nWEAKNESSES:\nI am not familiar enough with the explicit characterization of networks that minimize representation costs to judge the novelty and impact of the contribution. I was a little surprised to read \"Hanin [2021] gave a full characterization of univariate representation cost minimizers subject to data interpolation constraints\", and that there are follow-up extensions, which seem very relevant to the paper at hand. \n\nBeyond this, except for the limitations that have already been stated by the authors themselves, I do not see any major weaknesses. In practical terms, I am now curious to what extent larger (deeper + more sophisticated, e.g. convolutional or normalization including) models inherit some of the shown properties, but can understand that this goes beyond the scope of the paper. Also, is the M<<d problem discussed in line 333 a reason why adversarial examples exist? The strict interpolation on a ball would prevent them.\n\nQUESTIONS:\n- Please clarify in what way your work extends previous characterizations of minimal representation cost solutions.\n- And just out of curiosity - if \"Savarese et al. [2019] showed that the representation cost of a function realizable as a univariate two-layer ReLU network coincides with the L1-norm of the second derivative of the function\" (and assuming that the second derivative of a ReLU network has some meaning like the total variation of the (step-function-like) first derivative), doesn't proposition 1 follow from this quite directly?\n\nLIMITATIONS:\nThe authors have addressed the limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe elementary properties of NN solutions for the denoising problem have been explored, with a focus on offline training of a one hidden-layer ReLU network. When the noisy clusters of the data samples are well-separated, there exist multiple networks with zero loss, even in the case of under-parameterization, while having a different representation cost.  In the univariate case, a closed-form solution to such global minima with minimum representation cost has been derived. It has also been demonstrated that the univariate NN solution generalizes better than the eMMSE denoiser. In the multivariate case, it has been shown that the interpolating solution with minimal representation cost is aligned with the edges and/or faces connecting the clean data points in several basic cases.\n\nSTRENGTHS:\nThe NN solutions is studied in the setting of interpolation of noisy samples with minimal representation cost, in a practically relevant “low noise regime” where the noisy samples are well clustered. \n\nIn the univariate case, a closed-form solution for the minimal representation cost NN denoiser is derived and shown to have better generalizition behavior than the empirical minimum MSE  denoiser. \n\nIn the multivariate case, a closed-form solution for the minimal representation cost NN denoiser in multivariate case under various assumptions on the geometric configuration of the clean training samples. Moreover,  a general alignment phenomenon of minimal representation cost NN denoisers is illustrated in the multivariate setting.\n\nWEAKNESSES:\nWeakness 1: Empirical effectiveness.  This paper proposes a shallow denoiser and analyizes its theoretical performance. \n\nW1.1 Despite the interesting theoretical properties of the shallow denoiser, it is not clear whether the proposed model performs well in real image denoising applications due to the lack of sufficient empirical evaluation and analysis. \n\nW1.2 In the multi-variate settings, the authors consider training data on a subspace. However, it is not clarified in which typical settings this subspace assumption holds. Thus, the empirical reasonability of the subspace-based analysis is not well addressed. \n\nWeakness 2: The theoretical findings may be not sufficiently new. For example, the NN solution is shown contractive towards the clean data points, which has already been empirically observed in Autoencoders by Radhakrishnan et al.  [2018].\n\nQUESTIONS:\nPlease see and address the weaknesses.\n\nLIMITATIONS:\nI think the authors have well discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at the denoising problem and compares the neural network denoiser to what the paper calls the eMMSE denoiser. The eMMSE denoiser is the optimal denoiser (in function space) given finite data and known noise distribution. In contrast, the Neural Network denoiser has access to a finite number of copies of each of the finite data points and minimizes the empirical noise. \n\nThe paper shows that even in simple settings (univariate data), the minimum norm neural network denoiser has better generalization than the eMMSE denoiser, as it interpolates in between the data points whereas the eMMSE denoiser acts as a nearest neighbor denoiser. The paper further explores some other settings such as low rank and data on rays.\n\nSTRENGTHS:\nThe main strength of the paper is finding the explicit form of the minimum norm two-layer denoiser. This is interesting as it identifies the best point in function space that can be represented as a two-layer network. Hence gives the global minimizer for the neural network which is traditionally not easy. Hence this is important. \n\nOther strengths of the paper include that they show these denoisers are contracting and generalize better than some \"optimal\" denoisers.\n\nWEAKNESSES:\nThe are a few weaknesses of the paper. \n\n1) I think the presentation of the paper can be improved significantly. In many cases, the paper switches between functional representations of the function and the parametric representation of the function. While both are interesting, it would be nice to have to a clear distinction between the two and a way of translating from one representation to the other. \n\n2) While the results of the paper are interesting, there are limitations in the types of data that they consider. Specifically, univariate data, data on a line, and data on the union of lines are the cases in which the paper can identify the min norm solutions. While these cases are interesting and I understand that more general cases are challenging. Hence I don't think it is needed to solve more general cases. However, if the paper could extract some insights from these theoretical cases that might apply to more general cases, this would help strengthen the paper. \n\n3) I think the related works section is missing various theoretical works on denoising. [A-D] look at denoising via factorizations which fit in nicely with the low-rank structure that the paper looks at, and [E] looks at denoising regression case. \n\n\n[A] Raj R. Nadakuditi. OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage. IEEE Transactions on Information Theory, 2014\n\n[B] Marc Lelarge and Léo Miolane. Fundamental Limits of Symmetric Low-Rank Matrix Estimation. In Proceedings of the 2017 Conference on Learning Theory, 2017 \n\n[C] Antoine Maillard, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Perturbative Construction of Mean-Field Equations in Extensive-Rank Matrix Factorization and Denoising. Journal of Statistical Mechanics: Theory and Experiment, 2022 \n\n[D] Emanuele Troiani, Vittorio Erba, Florent Krzakala, Antoine Maillard, and Lenka Zdeborov’a. Optimal Denoising of Rotationally Invariant Rectangular Matrices. ArXiv, abs/2203.07752, 2022\n\n[E] Rishi Sonthalia and Raj Rao Nadakuditi. Training data size induced double descent for denoising feedforward neural networks and the role of training noise. Transactions on Machine Learning Research, 2023\n\nQUESTIONS:\n1. The paper presents the network with a skip connection, but as far as I can tell, the skip connection version is not studied in the paper. As $R(f)$ (Definition 1, Equation 19) is without the skip connection. \n\n2. The denoiser in Corollary 1, what would be the neural network representation of that function with ReLU activation?\n\n3. The fact that the online denoiser and the offline are equivalent is interesting. Do you have any intuition as to why the offline captures the online phenomena despite having much fewer samples?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}