SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. arXiv preprint arXiv:2303.01353, 2023.
Minmin Chen, Kilian Weinberger, Fei Sha, and Yoshua Bengio. Marginalized denoising autoencoders for nonlinear representations. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1476-1484, Beijing, China, 22-24 Jun 2014. PMLR. URL https://proceedings.mlr.press/v32/cheng14.html.
Michael Elad, Bahjat Kawar, and Gregory Vaksman. Image denoising: The deep learning revolution and beyond—a survey paper. SIAM Journal on Imaging Sciences, 16(3):1594-1654, 2023. doi: 10.1137/23M1545859. URL https://doi.org/10.1137/23M1545859.
Tolga Ergen and Mert Pilanci. Convex geometry and duality of over-parameterized neural networks. The Journal of Machine Learning Research, 22(1):9646-9708, 2021.
Boris Hanin. Ridgeless interpolation with shallow relu networks in  $1d$  is nearest neighbor curvature extrapolation and provably generalizes on lipschitz functions. arXiv preprint arXiv:2109.12960, 2021.
Samuel W Hasinoff, Frédo Durand, and William T Freeman. Noise-optimal capture for high dynamic range photography. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 553-560. IEEE, 2010.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
William C Horrace. Moments of the truncated normal distribution. Journal of Productivity Analysis, 43:133-138, 2015.
Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In International Conference on Learning Representations (ICLR), 2022.
Arthur Jacot, Eugene Golikov, Clement Hongler, and Franck Gabriel. Feature learning in 1_2-regularized dnns: Attraction/repulsion and sparsity. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 6763-6774, 2022.
B. G. Manjunath and Stefan Wilhelm. Moments calculation for the doubly truncated multivariate normal density. Journal of Behavioral Data Science, 1(1):17-33, May 2021. doi: 10.35566/jbds/v1n1/p2. URL https://jbds.isDSA.org/index.php/jbds/article/view/9.
Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems (NeurIPS), 34: 17749-17761, 2021.
A Marginalized loss
In this section, we derive the marginal loss for the case of a 1-hidden layer ReLU neural network. The loss function is,
$$
\mathbf {H} _ {i j} \left(\boldsymbol {w} _ {i}, \boldsymbol {w} _ {j}, \sigma^ {2}\right) = \mathbb {E} _ {\boldsymbol {x}} \left[ \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left[ \left[ \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {y} \right] _ {+} \left[ \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {y} \right] _ {+} \right] - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left[ \left[ \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {y} \right] _ {+} \right] \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left[ \left[ \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {y} \right] _ {+} \right] \right].
$$
Note that  $\mathbf{H} \succeq 0$  since  $\mathbf{H}$  is a covariance matrix. Thus we get,
$$
\mathcal {L} (\boldsymbol {\theta}, \sigma) = \mathbb {E} _ {\boldsymbol {x}} \left\| \sum_ {k = 1} ^ {K} \boldsymbol {a} _ {k} \tilde {\phi} \left(\hat {\boldsymbol {w}} _ {k} ^ {\top} \boldsymbol {x}, \| \boldsymbol {w} _ {k} \|, \sigma\right) - \boldsymbol {x} \right\| ^ {2} + \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {K} \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {a} _ {j} \mathbf {H} _ {i j} \left(\boldsymbol {w} _ {i}, \boldsymbol {w} _ {j}, \sigma^ {2}\right).
$$
Lemma 2. In the case of the ReLU activation function and Gaussian noise the following holds,
$$
\tilde {\phi} \left(\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x}, \| \boldsymbol {w} _ {i} \|, \sigma\right) = \| \boldsymbol {w} _ {i} \| \left(\left(1 - \Phi \left(- \frac {\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x}}{\sigma}\right)\right) \hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x} + \varphi \left(- \frac {\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x}}{\sigma}\right)\right)
$$
where  $\varphi, \Phi$  are the density and cumulative distribution of standard normal distribution, and
$$
\mathbf {H} _ {i j} = \mathbb {E} _ {\boldsymbol {x}} \left[ \Psi (\boldsymbol {x}, \boldsymbol {w} _ {i}, \boldsymbol {w} _ {j}, \sigma^ {2}) - \tilde {\phi} (\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x}, \| \boldsymbol {w} _ {i} \|, \sigma) \tilde {\phi} (\hat {\boldsymbol {w}} _ {j} ^ {\top} \boldsymbol {x}, \| \boldsymbol {w} _ {j} \|, \sigma) \right]
$$
$$
\begin{array}{l} \Psi \left(\boldsymbol {x}, \boldsymbol {w} _ {i}, \boldsymbol {w} _ {j}, \sigma^ {2}\right) = P \left(z _ {1} > 0, z _ {2} > 0; \boldsymbol {\mu} ^ {(i, j)}, \boldsymbol {\Sigma} ^ {(i, j)}\right) \left(\sigma_ {1 2} + \det  (\boldsymbol {\Sigma}) \frac {f \left(- \boldsymbol {\mu} ^ {(i , j)} ; \mathbf {0} , \boldsymbol {\Sigma} ^ {(i , j)}\right)}{P \left(z _ {1} > - \mu_ {1} , z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma} ^ {(i , j)}\right)} \right. \\ \left. + \sigma_ {1 1} F _ {1} (- \mu_ {1}) \mu_ {2} + \mu_ {1} \sigma_ {2 2} F _ {2} (- \mu_ {2}) + \mu_ {1} \mu_ {2}\right) \\ \end{array}
$$
$$
F _ {2} \left(- \mu_ {2}\right) = \frac {\varphi \left(- \frac {\mu_ {2}}{\sqrt {\sigma_ {2 2}}}\right) P \left(z > - \mu_ {1} ; \frac {\Lambda_ {1 2} \mu_ {2}}{\Lambda_ {1 1}} , \frac {1}{\Lambda_ {1 1}}\right)}{P \left(z _ {1} > - \mu_ {1} , z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)}
$$
$$
F _ {1} \left(- \mu_ {1}\right) = \frac {\varphi \left(- \frac {\mu_ {1}}{\sqrt {\sigma_ {1 1}}}\right) P \left(z > - \mu_ {2} ; \frac {\Lambda_ {2 1} \mu_ {1}}{\Lambda_ {2 2}} , \frac {1}{\Lambda_ {2 2}}\right)}{P \left(z _ {1} > - \mu_ {1} , z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)}
$$
(b)
Figure 3: Numerical evaluation of (23) Histogram of the sample average of  $\mathrm{ReLU}(x)$  for 10000 Monte-Carlo samples. We denote by  $E$  the analytical expectation and by  $\bar{E}$  the sample average. Figure (a) is for  $\mu = 1, \sigma = 5$ , the normalized error is  $\frac{|E - \bar{E}|}{E} = 0.0032\%$ . Figure (b) is for  $\mu = -1, \sigma = 5$ , the normalized error is  $\frac{|E - \bar{E}|}{E} = 0.0059\%$ .
$$
\boldsymbol {\mu} ^ {(i, j)} = \left( \begin{array}{c} \mu_ {1} \\ \mu_ {2} \end{array} \right) = \left( \begin{array}{c} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} \\ \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {x} \end{array} \right)
$$
$$
\boldsymbol {\Sigma} ^ {(i, j)} = \left( \begin{array}{c c} \sigma_ {1 1} & \sigma_ {1 2} \\ \sigma_ {1 2} & \sigma_ {2 2} \end{array} \right) = \left( \begin{array}{c c} \sigma^ {2}   \| \boldsymbol {w} _ {i} \| ^ {2} & \sigma^ {2} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {w} _ {j} \\ \sigma^ {2} \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {w} _ {i} & \sigma^ {2}   \| \boldsymbol {w} _ {j} \| ^ {2} \end{array} \right)
$$
$$
\boldsymbol {\Lambda} = \boldsymbol {\Sigma} ^ {- 1}.
$$
Notice that we denote by  $P\left(z_{1} > 0,z_{2} > 0;\pmb{\mu}^{(i,j)},\pmb{\Sigma}^{(i,j)}\right)$  the probability that  $z_{1} > 0,z_{2} > 0$  where  $(z_{1},z_{2})^{\top}\sim \mathcal{N}\left(\pmb{\mu}^{(i,j)},\pmb{\Sigma}^{(i,j)}\right)$ .
Proof. Let  $x \sim \mathcal{N}(\mu, \sigma^2)$ , then
$$
\begin{array}{l} E \left[ [ x ] _ {+} \right] = E \left[ [ x ] _ {+} | x > 0 \right] P (x > 0) + E \left[ [ x ] _ {+} | x <   0 \right] P (x > 0) \\ = E [ x | x > 0 ] P (x > 0) \\ = E [ x | x > 0 ] \left(1 - \Phi \left(- \frac {\mu}{\sigma}\right)\right) \\ \end{array}
$$
Note that given  $x > 0$  the distribution of  $x$  is truncated normal [Horrace, 2015]. Therefore,
$$
\mathbb {E} \left[ x | x > 0 \right] = \mu + \sigma \frac {\varphi \left(- \frac {\mu}{\sigma}\right)}{1 - \Phi \left(- \frac {\mu}{\sigma}\right)}
$$
$$
\mathbb {E} \left[ [ x ] _ {+} \right] = \left(1 - \Phi \left(- \frac {\mu}{\sigma}\right)\right) \mu + \sigma \varphi \left(- \frac {\mu}{\sigma}\right). \tag {23}
$$
Note that given  $\pmb{x}$ ,  $\pmb{y}$  is a Gaussian random vector. Therefore, given  $\pmb{x}$
$$
\boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} \sim \mathcal {N} \left(\boldsymbol {w} _ {k} ^ {\top} \boldsymbol {x}, \sigma^ {2} \left\| \boldsymbol {w} _ {k} \right\| ^ {2}\right)
$$
and we obtain,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {y} \mid \boldsymbol {x}} \left[ \left[ \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {y} \right] _ {+} \right] = \| \boldsymbol {w} _ {k} \| \mathbb {E} _ {\boldsymbol {y} \mid \boldsymbol {x}} \left[ \left[ \hat {\boldsymbol {w}} _ {k} ^ {\top} \boldsymbol {y} \right] _ {+} \right] \\ = \| \boldsymbol {w} _ {k} \| \left(1 - \Phi \left(- \frac {\hat {\boldsymbol {w}} _ {k} ^ {\top} \boldsymbol {x}}{\sigma}\right)\right) \hat {\boldsymbol {w}} _ {k} ^ {\top} \boldsymbol {x} + \sigma \varphi \left(- \frac {\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x}}{\sigma}\right). \\ \end{array}
$$
Let  $(z_{1},z_{2})^{\top}$  be a Gaussian random vector with
$$
\boldsymbol {\mu} = \left( \begin{array}{c} \mu_ {1} \\ \mu_ {2} \end{array} \right)
$$
$$
\boldsymbol {\Sigma} = \left( \begin{array}{c c} \sigma_ {1 1} & \sigma_ {1 2} \\ \sigma_ {1 2} & \sigma_ {2 2} \end{array} \right)
$$
4 Note that  $\sigma_{ii} = \sigma_i^2$
$$
\begin{array}{l} \mathbb {E} \left[ \left[ z _ {1} \right] _ {+} \left[ z _ {2} \right] _ {+} \right] = \mathbb {E} \left[ \left[ z _ {1} \right] _ {+} \left[ z _ {2} \right] _ {+} \mid z _ {1} > 0, z _ {2} > 0 \right] P \left(z _ {1} > 0, z _ {2} > 0\right) \\ = \mathbb {E} \left[ z _ {1} z _ {2} \mid z _ {1} > 0, z _ {2} > 0 \right] P \left(z _ {1} > 0, z _ {2} > 0\right). \\ \end{array}
$$
Given  $z_{1} > 0, z_{2} > 0$  the distribution of  $(z_{1},z_{2})^{\top}$  is truncated multivariate normal distribution [Manjunath and Wilhelm, 2021], therefore,
$$
\begin{array}{l} \mathbb {E} \left[ z _ {1} z _ {2} | z _ {1} > 0, z _ {2} > 0 \right] = \sigma_ {1 2} + \sigma_ {2 1} (- \mu_ {1}) F _ {1} (- \mu_ {1}) + \sigma_ {1 2} (- \mu_ {2}) F _ {2} (- \mu_ {2}) \\ + \left(\sigma_ {1 1} \sigma_ {2 2} - \sigma_ {1 2} \sigma_ {2 1}\right) \frac {f (- \boldsymbol {\mu} ; \mathbf {0} , \boldsymbol {\Sigma})}{p \left(z _ {1} > - \mu_ {1} , z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)} \\ - \left(\sigma_ {1 1} F _ {1} (- \mu_ {1}) + \sigma_ {1 2} F _ {2} (- \mu_ {2})\right) \left(\sigma_ {2 1} F _ {1} (- \mu_ {1}) + \sigma_ {2 2} F _ {2} (- \mu_ {2})\right) \\ + \left(\sigma_ {1 1} F _ {1} (- \mu_ {1}) + \sigma_ {1 2} F _ {2} (- \mu_ {2}) + \mu_ {1}\right) \left(\sigma_ {2 1} F _ {1} (- \mu_ {1}) + \sigma_ {2 2} F _ {2} (- \mu_ {2}) + \mu_ {2}\right) \\ = \sigma_ {1 2} + \det  (\boldsymbol {\Sigma}) \frac {f (- \boldsymbol {\mu} ; \mathbf {0} , \boldsymbol {\Sigma})}{p (z _ {1} > - \mu_ {1} , z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma})} \\ + \sigma_ {1 1} F _ {1} (- \mu_ {1}) \mu_ {2} + \mu_ {1} \sigma_ {2 2} F _ {2} (- \mu_ {2}) + \mu_ {1} \mu_ {2} \tag {24} \\ \end{array}
$$
where  $f(-\mu; \mathbf{0}, \Sigma)$  is a density function of Gaussian random vector with mean vector  $\mathbf{0}$  and covariance matrix  $\Sigma$  at the point  $-\mu$ , and
$$
F _ {2} \left(z _ {2}\right) = \frac {\int_ {- \mu_ {1}} ^ {\infty} f \left(z _ {1} , z _ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right) d z _ {1}}{P \left(Z _ {1} > - \mu_ {1} , Z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)}
$$
$$
F _ {1} \left(z _ {1}\right) = \frac {\int_ {- \mu_ {2}} ^ {\infty} f \left(z _ {1} , z _ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right) d z _ {2}}{P \left(Z _ {1} > - \mu_ {1} , Z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)}.
$$
We denote by  $\Lambda = \Sigma^{-1}$  thus,
$$
\int_ {- \mu_ {1}} ^ {\infty} \exp \left(- \frac {1}{2} \left(\Lambda_ {1 1} z _ {1} ^ {2} + 2 \Lambda_ {1 2} z _ {1} z _ {2} + \Lambda_ {2 2} z _ {2} ^ {2}\right)\right) d z _ {1} =
$$
$$
\exp \left(- \frac {1}{2} \Lambda_ {2 2} z _ {2} ^ {2}\right) \int_ {- \mu_ {1}} ^ {\infty} \exp \left(- \frac {1}{2} \Lambda_ {1 1} z _ {1} ^ {2} - \Lambda_ {1 2} z _ {1} z _ {2}\right) d z _ {1} =
$$
$$
\exp \left(- \frac {1}{2} \Lambda_ {2 2} z _ {2} ^ {2}\right) \int_ {- \mu_ {1}} ^ {\infty} \exp \left(- \frac {1}{2 \frac {1}{\Lambda_ {1 1}}} \left(z _ {1} + \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}}\right) ^ {2} + \frac {\Lambda_ {1 2} ^ {2} z _ {2} ^ {2}}{2 \Lambda_ {1 1}}\right) d z _ {1} =
$$
$$
\exp \left(- \frac {1}{2} \Lambda_ {2 2} z _ {2} ^ {2} + \frac {\Lambda_ {1 2} ^ {2} z _ {2} ^ {2}}{2 \Lambda_ {1 1}}\right) \int_ {- \mu_ {1}} ^ {\infty} \exp \left(- \frac {1}{2 \frac {1}{\Lambda_ {1 1}}} \left(z _ {1} + \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}}\right) ^ {2}\right) d z _ {1} =
$$
$$
\exp \left(- \frac {1}{2} \Lambda_ {2 2} z _ {2} ^ {2} + \frac {\Lambda_ {1 2} ^ {2} z _ {2} ^ {2}}{2 \Lambda_ {1 1}}\right) \sqrt {\frac {2 \pi}{\Lambda_ {1 1}}} P \left(z > - \mu_ {1}; - \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}}, \frac {1}{\Lambda_ {1 1}}\right).
$$
$$
\begin{array}{l} \int_ {- \mu_ {1}} ^ {\infty} f (z _ {1}, z _ {2}; 0, \boldsymbol {\Sigma}) d z _ {1} = \frac {1}{2 \pi \sqrt {\det (\Sigma)}} \exp \left(- \frac {1}{2} \Lambda_ {2 2} z _ {2} ^ {2} + \frac {\Lambda_ {1 2} ^ {2} z _ {2} ^ {2}}{2 \Lambda_ {1 1}}\right) \sqrt {\frac {2 \pi}{\Lambda_ {1 1}}} P (z > - \mu_ {1}; - \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}}, \frac {1}{\Lambda_ {1 1}}) \\ = \frac {1}{\sqrt {2 \pi \sigma_ {2 2}}} \exp \left(- \frac {1}{2 \sigma_ {2 2}} z _ {2} ^ {2}\right) P \left(z > - \mu_ {1}; - \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}}, \frac {1}{\Lambda_ {1 1}}\right). \\ \end{array}
$$
Similarly, we obtain,
$$
\int_ {- \mu_ {2}} ^ {\infty} f (z _ {1}, z _ {2}; 0, \boldsymbol {\Sigma}) d z _ {2} = \frac {1}{\sqrt {2 \pi \sigma_ {1 1}}} \exp \left(- \frac {1}{2 \sigma_ {1 1}} z _ {1} ^ {2}\right) P (z > - \mu_ {2}; - \frac {\Lambda_ {2 1} z _ {1}}{\Lambda_ {2 2}}, \frac {1}{\Lambda_ {2 2}}).
$$
$$
F _ {2} \left(z _ {2}\right) = \frac {\varphi \left(\frac {z _ {2}}{\sqrt {\sigma_ {2 2}}}\right) P \left(z > - \mu_ {1} ; - \frac {\Lambda_ {1 2} z _ {2}}{\Lambda_ {1 1}} , \frac {1}{\Lambda_ {1 1}}\right)}{P \left(Z _ {1} > - \mu_ {1} , Z _ {2} > - \mu_ {2} ; \mathbf {0} , \boldsymbol {\Sigma}\right)}
$$
$$
F _ {1} \left(z _ {1}\right) = \frac {\varphi \left(\frac {z _ {1}}{\sqrt {\sigma_ {1 1}}}\right) P \left(z > - \mu_ {2} ; - \frac {\Lambda_ {2 1} z _ {1}}{\Lambda_ {2 2}} , \frac {1}{\Lambda_ {2 2}}\right)}{P (Z _ {1} > - \mu_ {1} , Z _ {2} > - \mu_ {2} ; \mathbf {0} , \mathbf {\Sigma})}.
$$
(b)
Figure 4: Numerical evaluation of (24) Histogram of the sample average of  $\mathrm{ReLU}(z_1)\mathrm{ReLU}(z_2)$  for 10000 Monte-Carlo samples. We denote by  $E$  the analytical expectation and by  $\bar{E}$  the sample average. Figure (a) is for  $\pmb{\mu} = \begin{pmatrix} -4 \\ 17 \end{pmatrix}$ ,  $\pmb{\Sigma} = \begin{pmatrix} 13 & -9 \\ -9 & 8 \end{pmatrix}$ , the normalized error is  $\frac{|E - \bar{E}|}{E} = 0.0093\%$ . Figure (b) is for  $\pmb{\mu} = \begin{pmatrix} 6 \\ 2 \end{pmatrix}$ ,  $\pmb{\Sigma} = \begin{pmatrix} 10 & 2 \\ 2 & 1 \end{pmatrix}$ , the normalized error is  $\frac{|E - \bar{E}|}{E} = 0.0044\%$ .
Next, we present in Figures 3 and 4 a numerical evaluation of (23) and (24). As can be seen, the Monte-Carlo simulations verify the analytical results.
B Proofs of Results in Section 4
$$
\mathcal {D} = \left\{\left(y _ {n} = x _ {n} + \epsilon_ {n, m}, x _ {n}\right) \right\}, n \in [ N ], m \in [ M ], - \infty <   x _ {1} <   x _ {2} <   \dots <   n _ {N} <   \infty
$$
such that Assumption 1 holds. We define a reduced dataset, which only contains the noisy samples with the most extreme noises
$$
\bar {\mathcal {D}} = \left\{\left(x _ {n} + \epsilon_ {n} ^ {\mathrm {m i n}}, x _ {n}\right), \left(x _ {n} + \epsilon_ {n} ^ {\mathrm {m a x}}, x _ {n}\right) \right\}, n \in [ N ].
$$
We define the  $\ell^2$  penalty on the weights,
$$
C (\theta , K) = \sum_ {k = 1} ^ {K} \left(\| \boldsymbol {a} _ {k} \| ^ {2} + \| \boldsymbol {w} _ {k} \| ^ {2}\right).
$$
According to Theorem 1.2. in [Hanin, 2021], since we have opposite discrete curvature on the intervals  $[x_1 + \epsilon_1^{\max}, x_2 + \epsilon_1^{\min}] \cdots [x_{N-1} + \epsilon_1^{\max}, x_N + \epsilon_1^{\min}]$ ,
$$
\left\{h _ {\theta} (y) \mid h _ {\theta} (y) = x \forall (y, x) \in \bar {\mathcal {D}}, C (\theta , K) = C _ {*} \right\} = \left\{f _ {1 D} ^ {*} (y) \right\},
$$
$$
C _ {*} = \inf  _ {\theta , K} \left\{C \left(\theta , K\right) \mid \forall (y, x) \in \bar {\mathcal {D}} h _ {\theta} (y) = x \right\}.
$$
Also note that,
$$
\left\{h _ {\theta} (y) \mid h _ {\theta} (y) = x \forall (y, x) \in \mathcal {D}, C (\theta , K) = C _ {*} \right\} = \left\{f _ {1 D} ^ {*} (y) \right\}
$$
since  $f_{1D}^{*}(y)$  interpolates all the points in  $\mathcal{D}$ , and if we assume by contradiction that  $C_* > \inf_{\theta, K} \{C(\theta, K) \mid \forall (y, x) \in \mathcal{D} h_\theta(y) = x\}$  then  $C_* \neq \inf_{\theta, K} \{C(\theta, K) \mid \forall (y, x) \in \bar{\mathcal{D}} h_\theta(y) = x\}$ , thus contradicting Theorem 1.2. in [Hanin, 2021]. Note that the minimal number of neurons needed to represent  $f_{1D}^{*}(y)$  is  $2N - 2$ . So, if  $K \geq 2N - 2$  then  $f_{1D}^{*}$  is the minimizer of the representation cost (i.e.,  $f_{1D}^{*} \in \arg \min_f R(f)$ ).
First we prove the following lemmas.
Lemma 3. Let  $y = x + \sigma \epsilon$  where  $x, \epsilon \in \mathbb{R}, \sigma > 0$  then,
$$
\lim _ {\sigma \rightarrow 0 ^ {+}} \hat {x} ^ {\mathrm {e M M S E}} (y) = \hat {x} ^ {1 - \mathrm {N N}} (y)
$$
Proof. Notice that the following holds,
$$
\begin{array}{l} \frac {\exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} = \frac {\exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right) \exp \left(\frac {\min _ {n} | y - x _ {n} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(\frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right) \exp \left(\frac {\min _ {n} | y - x _ {n} | ^ {2}}{2 \sigma^ {2}}\right)} \\ = \frac {\exp \left(- \frac {\left| y - x _ {i} \right| ^ {2} - \operatorname* {m i n} _ {n} \left| y - x _ {n} \right| ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {\left| y - x _ {i} \right| ^ {2} - \operatorname* {m i n} _ {n} \left| y - x _ {n} \right| ^ {2}}{2 \sigma^ {2}}\right)}. \\ \end{array}
$$
$$
\lim  _ {\sigma \to 0 ^ {+}} \exp \left(- \frac {\left| y - x _ {i} \right| ^ {2} - \min  _ {n} \left| y - x _ {n} \right| ^ {2}}{2 \sigma^ {2}}\right) = \left\{ \begin{array}{l l} 1 & \left| y - x _ {i} \right| ^ {2} = \min  _ {n} \left| y - x _ {n} \right| ^ {2} \\ 0 & \left| y - x _ {i} \right| ^ {2} \neq \min  _ {n} \left| y - x _ {n} \right| ^ {2} \end{array} \right.
$$
so we obtain,
$$
\lim  _ {\sigma \to 0 ^ {+}} \frac {\exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} = \left\{ \begin{array}{l l} 1 & | y - x _ {i} | ^ {2} = \min  _ {n} | y - x _ {n} | ^ {2} \\ 0 & | y - x _ {i} | ^ {2} \neq \min  _ {n} | y - x _ {n} | ^ {2} \end{array} \right..
$$
$$
\begin{array}{l} \lim  _ {\sigma \rightarrow 0 ^ {+}} \hat {x} ^ {\mathrm {e M M S E}} (y) = \lim  _ {\sigma \rightarrow 0 ^ {+}} \frac {\sum_ {i = 1} ^ {N} x _ {i} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} = \arg \min  _ {n} | y - x _ {n} | ^ {2} \\ = \arg \min _ {n} | y - x _ {n} | = \hat {x} ^ {1 - \mathrm {N N}} (y). \\ \end{array}
$$
Lemma 4. Let  $y = x + \sigma \epsilon$  where  $x\in \mathbb{R},\sigma >0$  and  $\epsilon \sim \mathcal{N}(0,1)$  then,
$$
\lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(\hat {x} ^ {\mathrm {e M M S E}} (y)\right) = \lim  _ {\sigma \rightarrow 0 ^ {+}} \operatorname {M S E} \left(\hat {x} ^ {1 - \mathrm {N N}} (y)\right)
$$
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {x} ^ {\mathrm {e M M S E}} (y)\right) = \mathbb {E} \left[ \left(\hat {x} ^ {\mathrm {e M M S E}} (y) - x\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\hat {x} ^ {\mathrm {e M M S E}} (y) - \hat {x} ^ {1 - \mathrm {N N}} (y) + \hat {x} ^ {1 - \mathrm {N N}} (y) - x\right) ^ {2} \right] \\ = \operatorname {M S E} \left(\hat {x} ^ {1 - \mathrm {N N}} (y)\right) + \mathbb {E} \left[ \left(\hat {x} ^ {\mathrm {e M M S E}} (y) - \hat {x} ^ {1 - \mathrm {N N}} (y)\right) ^ {2} \right] \\ + 2 \mathbb {E} \left[ \left(\hat {x} ^ {1 - \mathrm {N N}} (y) - x\right) \left(\hat {x} ^ {\mathrm {e M M S E}} (y) - \hat {x} ^ {1 - \mathrm {N N}} (y)\right) \right] \\ \end{array}
$$
Note that,
$$
\begin{array}{l} \hat {x} ^ {\mathrm {e M M S E}} (y) = \frac {\sum_ {i = 1} ^ {N} x _ {i} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} \leq \frac {\sum_ {i = 1} ^ {N} \max _ {\jmath} \{x _ {\jmath} \} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} \\ = \max _ {j} \{x _ {j} \} \frac {\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} = \max _ {j} \{x _ {j} \}, \\ \end{array}
$$
$$
\begin{array}{l} \hat {x} ^ {\mathrm {e M M S E}} (y) = \frac {\sum_ {i = 1} ^ {N} x _ {i} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} \geq \frac {\sum_ {i = 1} ^ {N} \min _ {j} \{x _ {j} \} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} \\ = \min  _ {j} \{x _ {j} \} \frac {\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)}{\sum_ {i = 1} ^ {N} \exp \left(- \frac {| y - x _ {i} | ^ {2}}{2 \sigma^ {2}}\right)} = \min  _ {j} \{x _ {j} \}. \\ \end{array}
$$
$$
\hat {x} ^ {1 - \mathrm {N N}} (y) = \arg \min  _ {i} | y - x _ {i} | \leq \max  _ {i} \{x _ {i} \}
$$
$$
\hat {x} ^ {1 - \mathrm {N N}} (y) = \arg \min  _ {i} | y - x _ {i} | \geq \min  _ {i} \left\{x _ {i} \right\}
$$
thus  $\exists M_0 > 0\forall \sigma >0$
$$
\left| \hat {x} ^ {\mathrm {e M M S E}} (y) - \hat {x} ^ {1 - \mathrm {N N}} (y) \right| <   M _ {0}.
$$
According to Lemma 3
$$
\lim _ {\sigma \to 0 ^ {+}} \hat {x} ^ {\mathrm {e M M S E}} (y) = \hat {x} ^ {1 - \mathrm {N N}} (y)
$$
almost surely. Note that,
$$
\mathbb {E} _ {x, y} \left[ \left(\hat {x} ^ {\mathrm {e M M S E}} (y) - \hat {x} ^ {1 - \mathrm {N N}} (y)\right) ^ {2} \right] = \mathbb {E} _ {x, \epsilon} \left[ \left(\hat {x} ^ {\mathrm {e M M S E}} (x + \sigma \epsilon) - \hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon)\right) ^ {2} \right]
$$
Therefore, by the Dominated convergence theorem we obtain
$$
\lim _ {\sigma \rightarrow 0 ^ {+}} \mathbb {E} _ {x, \epsilon} \left[\left(\hat {x} ^ {\mathrm {e M M S E}} (x + \sigma \epsilon) - \hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon)\right) ^ {2} \right] =
$$
$$
\mathbb {E} _ {x, \epsilon} \left[ \lim  _ {\sigma \rightarrow 0 ^ {+}} \left(\hat {x} ^ {\mathrm {e M M S E}} (x + \sigma \epsilon) - \hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon)\right) ^ {2} \right] = 0
$$
$$
\lim _ {\sigma \rightarrow 0 ^ {+}} \mathbb {E} _ {x, \epsilon} \left[\left(\hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon) - x\right)\left(\hat {x} ^ {\mathrm {e M M S E}} (x + \sigma \epsilon) - \hat {x} ^ {1 - \mathrm {N N}} (x + \sigma \epsilon)\right)\right] =
$$