SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chen Zeno
Electrical and Computer Engineering Technion
Greg Ongie
Department Mathematical and Statistical Sciences Marquette University
Yaniv Blumenfeld, Nir Weinberger, Daniel Soudry
Technion
{chenzeno,yanivbl}@campus.technion.ac.il, gregory.ongie@marquette.edu  
nirwein@technion.ac.il, daniel.soudry@gmail.com
Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers — in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal  $\ell^2$  norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. We empirically verify this alignment phenomenon on synthetic data and real images.
The ability to reconstruct an image from a noisy observation has been studied extensively in the last decades, as it is useful for many practical applications (e.g., Hasinoff et al. [2010]). In recent years, Neural Network (NN) denoisers commonly replace classical expert-based approaches as they achieve substantially better results than the classical approaches (e.g., Zhang et al. [2017]). Beyond this natural usage, NN denoisers also serve as essential building blocks in a variety of common computer vision tasks, such as solving inverse problems [Zhang et al., 2021] and image generation [Song and Ermon, 2019, Ho et al., 2020]. To better understand the role of NN denoisers in such complex applications, we first wish to theoretically understand the type of solutions they converge to.
In practice, when training denoisers, we sample multiple noisy samples for each clean image and minimize the Mean Squared Error (MSE) loss for recovering the clean image. Since we sample numerous noisy samples per clean sample, the number of training samples is typically larger than the number of parameters in the network. Interestingly, even in such an under-parameterized regime, the loss has multiple global minima corresponding to distinct denoiser functions which achieve zero
loss on the observed data. To characterize these functions, we study, similarly to previous works [Savarese et al., 2019, Ongie et al., 2020], the shallow NN solutions that interpolate the training data with minimal representation cost, i.e., where the  $\ell^2$ -norm of the weights (without biases and skip connections) is as small as possible. This is because we converge to such min-cost solutions when we minimize the loss with a vanishingly small  $\ell^2$  regularization on these weights.
We first examine the univariate input case: building on existing results [Hanin, 2021], we characterize the min-cost interpolating solution and its generalization to unseen data. Next, we aim to extend this analysis to the multivariate case. However, this is challenging, since, to the best of our knowledge, there are no results that explicitly characterize these min-cost solutions for general multivariate shallow NNs — except in two basic cases. In the first case, the input data is co-linear [Ergen and Pilanci, 2021]. In the second case, the input samples are identical to their target outputs, so the trivial min-cost solution is the identity function. The NN denoisers' training regime is 'near' the second case: there, the input samples are noisy versions of the clean target outputs. Interestingly, we find that this regime leads to non-trivial min-cost solutions far from identity — even with an infinitesimally small input noise. We analytically investigate these solutions here.
Our Contributions. We study the NN solutions in the setting of interpolation of noisy samples with min-cost, in a practically relevant "low noise regime" where the noisy samples are well clustered. In the univariate case,
- We find a closed-form solution for the minimum representation cost NN denoiser. Then, we prove this solution generalizes better than the empirical minimum MSE (eMMSE) denoiser.
- We prove this min-cost NN solution is contractive toward the clean data points, that is, applying the denoiser necessarily reduces the distance of a noisy sample to one of the clean samples.
In the multivariate case,
- We derive a closed-form solution for the min-cost NN denoiser in multivariate case under various assumptions on the geometric configuration of the clean training samples. To the best of our knowledge, this is the first set of results to explicitly characterize a min-cost interpolating NN in a non-basic multivariate setting.
- We illustrate a general alignment phenomenon of min-cost NN denoisers in the multivariate setting: the optimal NN denoiser decomposes into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting clean training samples.
2 Preliminaries and problem setting
The denoising problem. Let  $\pmb{y} \in \mathbb{R}^d$  be a noisy observation of  $\pmb{x} \in \mathbb{R}^d$ , such that  $\pmb{y} = \pmb{x} + \pmb{\epsilon}$  where  $\pmb{x}$  and  $\pmb{\epsilon}$  are statistically independent, and  $\mathbb{E}[\pmb{\epsilon}] = 0$ . Commonly, this noise is Gaussian with covariance matrix  $\sigma^2\mathrm{I}$ . The ultimate goal of a denoiser  $\hat{\pmb{x}}(\pmb{y})$  is to minimize the MSE loss over the joint probability distribution of the data and the noisy observation ("population distribution"), i.e., to minimize
$$
\mathcal {L} (\hat {\boldsymbol {x}}) = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| \hat {\boldsymbol {x}} (\boldsymbol {y}) - \boldsymbol {x} \| ^ {2}. \tag {1}
$$
The well-known optimal solution for (1) is the minimum mean square error (MMSE) denoiser, i.e.,
$$
\hat {\boldsymbol {x}} ^ {*} (\boldsymbol {y}) = \mathbb {E} _ {\boldsymbol {x} | \boldsymbol {y}} [ \boldsymbol {x} \mid \boldsymbol {y} ] \in \arg \min  _ {\hat {\boldsymbol {x}} (\boldsymbol {y})} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| \boldsymbol {x} - \hat {\boldsymbol {x}} (\boldsymbol {y}) \| ^ {2}. \tag {2}
$$
Since we do not have access to the distribution of the data, and hence not to the posterior distribution, we rely on a finite amount of clean data  $\{\pmb{x}_n\}_{n=1}^N$  in order to learn a good approximation for the MMSE estimator. One approach is to assume an empirical data distribution and derive the optimal solution of (1), i.e., the empirical minimum mean square error (eMMSE) denoiser,
$$
\hat {\boldsymbol {x}} ^ {\mathrm {e M M S E}} (\boldsymbol {y}) \in \underset {\hat {\boldsymbol {x}} (\boldsymbol {y})} {\arg \min } \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\boldsymbol {y} \mid \boldsymbol {x} _ {n}} \| \hat {\boldsymbol {x}} (\boldsymbol {y}) - \boldsymbol {x} _ {n} \| ^ {2}. \tag {3}
$$
If the noise is Gaussian with a covariance of  $\sigma^2\mathrm{I}$ , an explicit solution to the eMMSE is given by
$$
\hat {\boldsymbol {x}} ^ {\mathrm {e M M S E}} (\boldsymbol {y}) = \frac {\sum_ {n = 1} ^ {N} \boldsymbol {x} _ {n} \exp \left(- \frac {\| \boldsymbol {y} - \boldsymbol {x} _ {n} \| ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {n = 1} ^ {N} \exp \left(- \frac {\| \boldsymbol {y} - \boldsymbol {x} _ {n} \| ^ {2}}{2 \sigma^ {2}}\right)}. \tag {4}
$$
An alternative approach to computing the eMMSE directly is to draw  $M$  noisy samples for each clean data point, as  $\pmb{y}_{n,m} = \pmb{x}_n + \pmb{\epsilon}_{n,m}$ , where  $\pmb{\epsilon}_{n,m} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$  are independent and identically distributed, and to minimize the following loss function
$$
\mathcal {L} _ {\text {o f f l i n e}, M} (\hat {\boldsymbol {x}}) = \frac {1}{M N} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {N} \| \hat {\boldsymbol {x}} \left(\boldsymbol {y} _ {n, m}\right) - \boldsymbol {x} _ {n} \| ^ {2}. \tag {5}
$$
Denoiser model and algorithms. In practice, we approximate the optimal denoiser  $\hat{\pmb{x}} (\pmb {y})$  using a parametric model  $h_\theta (y)$ , typically a NN. We focus on a shallow ReLU network model with a skip connection of the form
$$
\boldsymbol {h} _ {\theta} (\boldsymbol {y}) = \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} \left[ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} + b _ {k} \right] _ {+} + \boldsymbol {V} \boldsymbol {y} + \boldsymbol {c} \tag {6}
$$
where  $\theta = ((\theta_k)_{k=1}^K; c, V)$  with  $\theta_k = (b_k, a_k, w_k) \in \mathbb{R} \times \mathbb{R}^d \times \mathbb{R}^d$  and  $c \in \mathbb{R}^d$ ,  $V \in \mathbb{R}^{d \times d}$ . We train the model on a finite set of clean data points  $\{\pmb{x}_n\}_{n=1}^N$ . The common practical training method is based on an online approach. First, we sample a random batch (with replacement) from the data  $\mathcal{B} \subseteq \{\pmb{x}_n\}_{n=1}^N$ . Then, for each clean data point  $\pmb{x}_n \in \mathcal{B}$ , we draw a noisy sample  $\pmb{y}_n = \pmb{x}_n + \pmb{\epsilon}_n$  where  $\pmb{\epsilon}_n \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathrm{I})$  are independent of the clean data points and other noise samples. At each iteration  $t$  out of  $T$  iterations, we update the model parameters according to a stochastic gradient descent rule, with a vanishingly small regularization term  $\lambda C(\pmb{\theta})$ , that is,
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} - \eta \nabla_ {\boldsymbol {\theta} _ {t}} \frac {1}{| \mathcal {B} |} \sum_ {n \in \mathcal {B}} \left\| \boldsymbol {h} _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {y} _ {n}\right) - \boldsymbol {x} _ {n} \right\| ^ {2} - \eta \lambda \nabla_ {\boldsymbol {\theta} _ {t}} C \left(\boldsymbol {\theta} _ {t}\right). \tag {7}
$$
Another training method [Chen et al., 2014] is based on an offline approach. We sample  $M$  noisy sample for each clean data point and minimize (5) plus a regularization term
$$
\mathcal {L} _ {\text {o f f l i n e}, M} (\boldsymbol {\theta}) = \frac {1}{M N} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {N} \left\| \boldsymbol {h} _ {\boldsymbol {\theta}} \left(\boldsymbol {y} _ {n, m}\right) - \boldsymbol {x} _ {n} \right\| ^ {2} + \lambda C (\boldsymbol {\theta}). \tag {8}
$$
Similarly to previous works [Savarese et al., 2019, Ongie et al., 2020], we assume an  $\ell^2$  penalty on the weights, but not on the biases and skip connections, i.e.,
$$
C (\theta) = \frac {1}{2} \sum_ {k = 1} ^ {K} \left(\left\| \boldsymbol {a} _ {k} \right\| ^ {2} + \left\| \boldsymbol {w} _ {k} \right\| ^ {2}\right). \tag {9}
$$
Low noise regime. In this paper, we study the solution of the NN denoiser when the clusters of noisy samples around each clean point are well-separated, a setting which we refer to as the "low noise regime". This is a rather relevant regime since denoisers are practically used when the noise level is mild. Indeed, common image-denoising benchmarks test on low (but not negligible) noise levels. For instance, in the commonly used denoising benchmark BSD68 [Roth and Black, 2009], the noise level  $\sigma = 0.1$  is in the low noise regime. $^{1}$  Moreover, this setting is important, for example, in diffusion-based image generation, since at the end of the reverse denoising process, new images are sampled by denoising smaller and smaller noise levels. $^{2}$
3 Basic properties of neural network denoisers
Offline vs. online NN solutions. NN denoisers are traditionally trained in an online fashion (7), using a finite amount of  $T$  iterations. Consequently, only a finite number of noisy samples are used for each clean data point. We empirically observe that the solutions in the offline and online settings are similar. Specifically, in the univariate case, we show in Figure 1 that denoisers based on offline and
<sup>1</sup>The minimum distance between two images in BSD68 is about 97 while the image resolution is  $d = 481 \times 321$ . Also, the norm of the noise concentrates around the value of  $\sqrt{d}\sigma \approx \sqrt{481 \cdot 321} \cdot 0.1 \approx 40 < 97$ . Therefore, the clusters of noisy samples around each clean point are generally well-separated.
2Interestingly, it was suggested that the "useful" part of the diffusion dynamics happens only below some critical noise level [Raya and Ambrogioni, 2023].
Figure 1: NN denoiser vs eMMSE denoiser. We trained a one-hidden-layer ReLU network with a skip connection on a denoising task. The clean dataset has four points equally spaced in the interval  $[-5,5]$ , and the noisy samples are generated by adding zero-mean Gaussian noise with  $\sigma = 1.5$ . We use  $\lambda = 10^{-5}$  in both setting. The Figure shows the denoiser output as a function of its input for: (1) NN denoiser trained online using (7) for  $100K$  iterations, (2) NN denoiser trained offline using (8) with  $M = 9000$  and  $20K$  epochs, and (3) the eMMSE denoiser (4).
online loss functions converge to indistinguishable solutions. For the multivariate case, we observe (Figure 11 in Appendix D) that the offline and online solutions achieve approximately the same test MSE when trained on a subset of the MNIST dataset. The comparison is made using the same number of iterations for both training methods, while using much less noisy samples in the offline setting. Evidently, this lower number of samples does not significantly affect the generalization error. Hence, in the rest of the paper, we focus on offline training (i.e., minimizing the offline loss  $\mathcal{L}_{\mathrm{offline},M}$ ), as it defines an explicit loss function with solutions that can be theoretically analyzed, as in [Savarese et al., 2019, Ongie et al., 2020].
The empirical MMSE denoiser. The law of large numbers implies that the denoiser minimizing the offline loss  $\mathcal{L}_{\mathrm{offline},M}$  approaches the eMMSE estimator in the limit of infinitely many noisy samples,
$$
\hat {\boldsymbol {x}} ^ {\mathrm {e M M S E}} (\boldsymbol {y}) \in \arg \min  _ {\hat {\boldsymbol {x}}} \lim  _ {M \rightarrow \infty} \mathcal {L} _ {\text {o f f l i n e}, M} (\hat {\boldsymbol {x}}). \tag {10}
$$
Therefore, it may seem that for a reasonable number of noise samples  $M$ , a large enough model, and small enough regularization, the denoiser we get by minimizing the offline loss (6) will also be similar to the eMMSE estimator. However, Figure 1 shows that the eMMSE solution and the NN solutions (both online and offline) are quite different. The eMMSE denoiser has a much sharper transition and maps almost all inputs to a value of a clean data point. This is because in the case of low noise the eMMSE denoiser (4) approximates the one nearest-neighbor (1-NN) classifier, i.e.,
$$
\lim  _ {\sigma \rightarrow 0 ^ {+}} \hat {\boldsymbol {x}} ^ {\mathrm {e M M S E}} (\boldsymbol {y}) = \underset {\boldsymbol {x} \in \left\{\boldsymbol {x} _ {i} \right\} _ {i = 1} ^ {N}} {\arg \min } \| \boldsymbol {y} - \boldsymbol {x} \| . \tag {11}
$$
In contrast, the NN denoiser maps each noisy sample to its corresponding clean sample only in a limited "noise ball" around the clean point, and interpolates near-linearly between the "noise balls". Hence, we may expect that the smoother NN denoiser typically generalizes better than the eMMSE denoiser. We prove that this is indeed true for the univariate case in Section 4.
Why the NN denoiser does not converge to the eMMSE denoiser? Note that the limit in (10) is not practically relevant for the low-level noise regime, since we need an exponentially large  $M$  in order to converge in this limit. For example, in the case of univariate Gaussian noise, we have that  $P(|\epsilon| > t) \leq 2\exp\left(-\frac{t^2}{2\sigma^2}\right)$ ,  $\forall t > 0$ . Therefore, during training, we effectively observe only noisy samples that are in a bounded interval of size  $2\sigma \sqrt{\log M}$  around each clean sample (see Figure 1). In other words, in the low-noise regime and for non-exponential  $M$ , there is no way to distinguish if the noise is sampled from some distribution with limited support of from a Gaussian distribution. The denoiser minimizing the loss with respect to a bounded-support distribution can be radically different from the eMMSE denoiser in the regions outside the "noise balls" surrounding the clean samples, where the denoiser function is not constrained by the loss. This leads to a large difference between the NN denoiser and the MMSE estimator.
Alternatively, one may suggest that the NN denoiser does not converge to the eMMSE denoiser due to an approximation error (i.e., the shallow NN's model capacity is too small to approximate the MMSE denoiser). Nevertheless, we provide empirical evidence indicating it is not the case. Specifically, recall that in the low noise regime, the eMMSE denoiser tends to the nearest-neighbor classifier, and
such a solution does not generalize well to test data. Thus, if the NN denoiser would have converged to the eMMSE solution, then its test error would have increased with the network size, in contrast to what we observe in Figure 12 (Appendix D).
Therefore, in order to approximate the eMMSE with a NN, it seems we must have an exponentially large  $M$ . Alternatively, we may converge to the eMMSE if we use a loss function marginalized over the Gaussian noise. This idea was previously suggested by Chen et al. [2014], with the goal of effectively increasing the number of noisy samples and thus improving the training performance of denoising autoencoders. Therein, this improvement was obtained by approximating the marginalized loss function by a Taylor series expansion. However, for shallow denoisers, we may actually obtain an explicit expression for this marginalized loss, without any approximation. Specifically, if we assume for simplicity, that the network does not have a linear unit  $(V = 0)$  and its bias terms are zero  $(c = 0, b_k = 0)$ , then the marginalized loss for Gaussian noise, derived in Appendix A, is given by
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {\theta}, \sigma) = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| \boldsymbol {h} _ {\theta} (\boldsymbol {y}) - \boldsymbol {x} \| ^ {2} = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left\| \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} [ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} ] _ {+} - \boldsymbol {x} \right\| ^ {2} = \mathbb {E} _ {\boldsymbol {x}} \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left\| \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {i} [ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} ] _ {+} - \boldsymbol {x} \right\| ^ {2} \\ = \mathbb {E} _ {\boldsymbol {x}} \left\| \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} \tilde {\phi} \left(\hat {\boldsymbol {w}} _ {k} ^ {\top} \boldsymbol {x}, \| \boldsymbol {w} _ {k} \|, \sigma\right) - \boldsymbol {x} \right\| ^ {2} + \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {K} \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {a} _ {j} \mathbf {H} _ {i j} \left(\boldsymbol {w} _ {i}, \boldsymbol {w} _ {j}, \sigma^ {2}\right), \tag {12} \\ \end{array}
$$
where  $\pmb{w}_k = \hat{\pmb{w}}_k\|\pmb{w}_k\|$  and  $\mathbf{H} \succeq 0$ ,  $\tilde{\phi}(\cdot)$  are defined in Appendix A. NN denoisers trained over this loss function will thus tend to the eMMSE solution as the network size is increased. However, as we explained above, this is not necessarily desirable, so we only mention (12) to show exact marginalization is feasible.
Regularization biases toward specific neural network denoisers. To further explore the converged solution for offline training, we note that the offline loss function  $\mathcal{L}_{\mathrm{offline},M}(\theta)$  allows the network to converge to a zero-loss solution. This is in contrast to online training for which each batch leads to new realizations of noisy samples, and thus the training error is never exactly zero. Specifically, consider the low noise regime (well-separated noisy clusters). Then, the network can perfectly fit all the noisy samples using a finite number of neurons (see Section 4 for a more accurate description in the univariate case). Importantly, there are multiple ways to cluster the noisy data points with such neurons, and so there are multiple global training loss minima that the network can achieve with zero loss, each with a different generalization capability. In contrast to the standard case considered in the literature, this holds even in the under-parameterized case (where  $NM$ , the total number of noisy samples, is larger than the number of parameters).
Since there are many minima that perfectly fit the training data, we converge to specific minima which also minimize the  $\ell^2$  regularization we use (even though we assumed it is vanishing). Specifically, in the limit of vanishing regularization  $C(\theta)$ , the minimizers of  $\mathcal{L}_{\mathrm{offline},M}(\pmb{\theta})$  also minimize the representation cost.
Definition 1. Let  $h_\theta : \mathbb{R}^d \to \mathbb{R}^d$  denote a shallow ReLU network of the form (6). For any function  $f : \mathbb{R}^d \to \mathbb{R}^d$  realizable as a shallow ReLU network, we define its representation cost as
$$
R (\boldsymbol {f}) = \inf  _ {\theta : \boldsymbol {f} = \boldsymbol {h} _ {\theta}} C (\theta) = \inf  _ {\theta} \sum_ {k = 1} ^ {K} \| \boldsymbol {a} _ {k} \| \text {s . t .} \| \boldsymbol {w} _ {k} \| = 1 \forall k, \boldsymbol {f} = \boldsymbol {h} _ {\theta}, \tag {13}
$$
and a minimizer of this cost, i.e. the 'min-cost' solution, as
$$
\boldsymbol {f} ^ {*} \in \underset {\boldsymbol {f}} {\operatorname {a r g m i n}} R (\boldsymbol {f}) \text {s . t .} \boldsymbol {f} \left(\boldsymbol {y} _ {n, m}\right) = \boldsymbol {x} _ {n} \forall n, m, \tag {14}
$$
where the second equality in (13) holds due to the 1-homogeneity of the ReLU activation function, and since the bias terms are not regularized (see [Savarese et al., 2019, Appendix A]). In the next sections, we examine which function we obtain by minimizing the representation cost  $R(\pmb{f})$  in various settings.
4 Closed form solution for the NN denoiser function — univariate data
In this section, we prove that NN denoisers that minimize  $R(f)$  for univariate data have the specific piecewise linear form observed in Figure 1, and we discuss the properties of this form. We observe  $N$
clean univariate data points  $\{x_{n}\}_{n = 1}^{N}$ , s.t.  $-\infty < x_{1} < x_{2} < \dots < x_{N} < \infty$ , and  $M$  noisy samples (drawn from some known distribution) for each clean data point, such that  $y_{n,m} = x_{n} + \epsilon_{n,m}$ . We denote by  $\epsilon_{n}^{\max}$  the maximal noise seen for data point  $x_{n}$ , and by  $\epsilon_{n}^{\min}$  the minimal noise seen for data point  $x_{n}$ , i.e.,
$$
\epsilon_ {n} ^ {\max } \equiv \max  _ {m} \epsilon_ {n, m}, \quad \epsilon_ {n} ^ {\min } \equiv \min  _ {m} \epsilon_ {n, m}, \tag {15}
$$
and assume the following,
Assumption 1. Assume the data  $\{x_{n}\}_{n = 1}^{N}$  is well-separated after the addition of noise, i.e.,
$$
\forall n \in [ N - 1 ]: x _ {n} + \epsilon_ {n} ^ {\max } <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min }, \tag {16}
$$
and  $\epsilon_n^{\mathrm{max}} > 0, \epsilon_n^{\mathrm{min}} < 0$ .
So we can state the following,
Proposition 1. For all datasets such that Assumption 1 holds, the unique minimizer of  $R(f)$  is
$$
f _ {1 D} ^ {*} (y) = \left\{ \begin{array}{l l} x _ {1}, & y <   x _ {1} + \epsilon_ {1} ^ {\min } \\ x _ {n}, & x _ {n} + \epsilon_ {n} ^ {\min } \leq y \leq x _ {n} + \epsilon_ {n} ^ {\max } \\ \frac {x _ {n + 1} - x _ {n}}{x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)} \left(y - \left(x _ {n} + \epsilon_ {n} ^ {\max }\right)\right) + x _ {n}, & x _ {n} + \epsilon_ {n} ^ {\max } <   y <   x _ {n + 1} + \epsilon_ {n + 1} ^ {\min } \\ x _ {N}, & y > x _ {N} + \epsilon_ {N} ^ {\max } \end{array} . \right. \tag {17}
$$