SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minimum norm interpolation by perceptr: Explicit regularization and implicit bias
Jiyoung Park
wldyddl5510@tamu.edu
Ian Pelakh
ispelakh@iastate.edu
Stephan Wojtowytsch
s.woj@pitt.edu
We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to a minimum norm interpolant as the number of data points and parameters tends to infinity when a weight decay regularizer is penalized with a coefficient which vanishes at a precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.
Modern neural networks mostly operate in an overparametrized regime, i.e. they possess more tunable parameters than the number of data points contributing to the loss function. Safran and Shamir [2018a], E et al. [2019b], Du et al. [2018], Chizat and Bach [2018] associate overparametrization with better training properties, and Belkin et al. [2019, 2020] find it to enhance statistical generalization (see also [Loog et al., 2020] for historical context). For many architectures, overparametrization leads to the ability to fit any values  $y_{i}$  at a given set of data points  $\{x_{1},\ldots ,x_{n}\}$ , and Cooper [2018] shows that generically, the set of weights for which the neural network interpolates prescribed values is a submanifold of high dimension and co-dimension in parameter space. Which solution on the manifold is dynamically chosen by an optimization algorithm, and which solutions have favorable generalization properties, is an active area of research in theoretical machine learning.
Current practice is to estimate a model's generalization to previously unseen data by assessing its performance on a hold-out set (a posteriori error estimate) or by using uniform estimates on the generalization of all elements of a function class (a priori estimate if the function class is encoded ahead of time by explicit regularization, a posteriori if membership is determined after optimization). Neither approach yields information on what a neural network does outside the support of the data distribution, a topic of great interest for the study of distributional shift and adversarial stability.
The main contribution of this work is split between two complementary lines of investigation:
1. We prove that neural network minimizers of regularized empirical risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit (Section 2). Our result improves previous works to more general settings (See Remark 2.2).
2. Training a neural network generally corresponds to solving a non-convex minimization problem. While we provide convergence guarantees for empirical risk minimizers, in general there is no guarantee that a training algorithm finds a global minimizer of an empirical risk functional. Even if convergence holds, it is unclear which minimizer is selected (in the overparametrized regime, where the set of minimizers is a high-dimensional manifold). In settings where the minimum norm interpolant is known (Section 3), we compare numerical solutions to theoretical predictions to better understand (1) the predictive
power of theoretically studying empirical risk minimizers and (2) the implicit bias of different optimization algorithms (Sections 4 and 5). We believe this to be a useful benchmark problem for a better understanding of explicit regularization and implicit bias in optimization.
Minimum norm interpolants are the regression analogue to maximum margin classifiers. They are associated with favorable generalization properties and relative stability even against adversarial perturbations.
Minimum norm interpolation. In classes of linear functions, minimum norm interpolation has a long history as ridge regression (minimal  $\ell^2$ -norm) or as the least absolute shrinkage selection and operator (LASSO, minimal  $\ell^1$ -norm). To the best of our knowledge, minimum norm interpolation by neural networks has only been studied for shallow neural networks in one dimension by Hanin [2021], Debarre et al. [2022], Boursier and Flammarion [2023] and in odd dimension for certain radially symmetric data by Wojtowitsch [2022]. For classification problems, minimum norm/maximum margin classifiers were considered by E and Wojtowitsch [2022]. For finite datasets, the set of minimum norm interpolants was characterized by Parhi and Nowak [2021]. A parametrization of the same function class by neural networks with multiple linear layers and a single ReLU layer induces different concepts of minimum norm interpolation as studied by Ongie and Willett [2022].
Implicit bias. The implicit bias of parameter optimization algorithms has been studied for gradient flows with infinitely wide, but shallow ReLU networks by Chizat et al. [2019], Jin and Montúfar [2020] and for stochastic gradient descent and diagonal linear networks by Pesme et al. [2021]. Chizat and Bach [2020] prove convergence to a maximum margin classifier for infinitely wide ReLU networks with one hidden layer if the parameters follow the gradient flow of an (unregularized) logistic loss risk functional. Many authors, including Damian et al. [2021], Li et al. [2021], Wu et al. [2022] and Wojtowytsch [2020], study the bias of SGD towards solutions at which the loss landscape is 'flat' in the parameter space. Hochreiter and Schmidhuber [1997] conjectured such minimizers to have favorable generalization properties. In many cases, minimizers tend to be flatter if the parameters associated to them are not excessively large. Yang et al. [2021] describe several phase-transitions in parameter space for the relation between flatness in parameter space and generalization. Zhou et al. [2020] compare the implicit bias of SGD and ADAM. Smith et al. [2021], Barrett and Dherin [2020] argue that gradient descent resembles a gradient flow in a modified (regularized) loss landscape more closely than the gradient flow of the original loss function. Hajjar and Chizat [2022] investigate the role of symmetries in optimization.
Barron space. The Barron class is adapted to ReLU networks with a single hidden layer and weights of bounded average magnitude. Slightly different versions of the same function space have been studied by Bach [2017], E et al. [2019a,c], Ongie et al. [2019], E and Wojtowytsch [2020, 2021], Caragea et al. [2020], Parhi and Nowak [2021], Wojtowytsch [2022], Siegel and Xu [2022, 2023] under various names such as  $\mathcal{F}_1$ , Radon-BV or the variation space of the ReLU dictionary.
Conventions.  $\mu$  always denotes a  $\sigma^2$ -sub-Gaussian probability measure on the data domain  $\mathbb{R}^d$ , i.e.
$$
\exists C, \sigma > 0 \quad \text {s . t .} \quad \mathbb {E} _ {X \sim \mu} \left[ \exp \left(\lambda \left\{\| X \| - \mathbb {E} \| X \| \right\}\right) \right] \leq C \exp \left(\frac {\lambda^ {2} \sigma^ {2}}{2}\right) \quad \forall   \lambda > 0.
$$
All norms are Frobenius norms ( $\ell^2$ -norm for vectors). For  $n \in \mathbb{N}$ ,  $S_n = \{x_{n,1}, \ldots, x_{n,n}\}$  is a set of  $n$  iid samples from the distribution  $\mu$ , independent of  $S_{n'}$  for  $n' \neq n$ . When unambiguous, we denote  $x_{n,i} = x_i$ . We take  $\ell(f, y) = |f - y|^2$  as the mean squared error/ $\ell^2$ -loss function, but we remark that the theoretical analysis remains valid if  $\ell_{MSE}$  is replaced by  $\ell^1$ -loss or a Huber or pseudo-Huber loss
$$
\ell_ {H u b} (f, y) = \left\{ \begin{array}{l l} | f - y | ^ {2} & \text {i f} | y - f | <   1 \\ 2 | y - f | - 1 & \text {i f} | y - f | \geq 1 \end{array} \right., \qquad \ell_ {p H} (y, h) = \sqrt {1 + | y - f | ^ {2}} - 1.
$$
For  $m\in \mathbb{N}$  and  $(a,W,b)\in \mathbb{R}^m\times \mathbb{R}^{m\times d}\times \mathbb{R}^{m + 1}$ , let
$$
f _ {(a, W, b)}: \mathbb {R} ^ {d} \to \mathbb {R}, \quad f _ {(a, W, b)} (x) = b _ {0} + \sum_ {i = 1} ^ {m} a _ {i} \sigma (w _ {i} \cdot x + b _ {i}), \quad \sigma (z) = \operatorname {R e L U} (z) = \max  \{z, 0 \},
$$
i.e.  $f_{(a,W,b)}$  is a ReLU network with a single hidden layer and weights  $a, W$  and biases  $b$ . The vector  $w_i \in \mathbb{R}^d$  is the  $i$ -th row of the matrix  $W$ . For  $m, n \in \mathbb{N}$  and  $\lambda \geq 0$ , we denote the regularized empirical risk functional as  $\widehat{\mathcal{R}}_{n,m,\lambda}: \mathbb{R}^m \times \mathbb{R}^{m \times d} \times \mathbb{R}^{m+1} \to [0, \infty)$ :
$$
\widehat {\mathcal {R}} _ {n, m, \lambda} (a, W, b) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \ell \left(f _ {(a, W, b)} (x _ {i}), y _ {i}\right) + \frac {\lambda}{2} \left(\| a \| _ {2} ^ {2} + \| W \| _ {F r o b} ^ {2}\right).
$$
Concepts. We introduce what we dub 'homogeneous Barron space'  $\mathcal{B}$  heuristically here, and in greater detail in Appendix C. As a measure of magnitude of the function, we consider the weight decay (or Tikhonov) regularizer  $\frac{1}{2}\left(\|a\|_2^2 + \|W\|_{Frob}^2\right)$  of the parametrized function  $f_{(a,W,b)}$  which does not control the magnitude of the bias vector.
Note that the function class  $f_{(a,W,b)}$  and its complexity do not change when we consider representations of the form  $f_{m}(x) = \frac{1}{m}\sum_{i=1}^{m}a_{i}\sigma(w_{i}\cdot x + b_{i})$  with a regularizer  $\frac{1}{2m}\left(\|a\|_{2}^{2} + \|W\|_{Frob}^{2}\right) = \frac{1}{2m}\sum_{i=1}^{m}\left(a_{i}^{2} + \|w_{i}\|_{2}^{2}\right)$ . A continuum analogue to these functions represented as an 'empirical average' over individual neurons is a general expectation representation  $f_{\pi}(x) = \mathbb{E}_{(a,w,b)\sim\pi}\left[a\sigma(w^{T}x+b)\right]$  for some probability distribution  $\pi$  on parameter space and a regularizer  $\frac{1}{2}\mathbb{E}_{(a,w,b)\sim\pi}[a^{2} + \|w\|_{2}^{2}]$ . As the parametrization of a function by a neural network is generally non-unique, we define the Barron semi-norm  $[f]_{\mathcal{B}} = \inf_{\{\pi:f_{\pi}\equiv f\}}\mathbb{E}_{(a,w,b)\sim\pi}[a^{2} + \|w\|_{2}^{2}]$  as the lowest value attained by the regularizer over all possible parametrizations. For a more comprehensive understanding, interested readers may refer to Appendix C.
2 General convergence result
We first state a general convergence result to a minimum norm interpolant of given data generated by functions in the homogeneous Barron class  $\mathcal{B}$ .
Theorem 2.1. Take  $\mu, S_n, \widehat{\mathcal{R}}_{n,m,\lambda}$  as in Section 1.2,  $f^* \in \mathcal{B}$ , and let  $y_i = f^*(x_i)$  for  $i = 1, \ldots, n$ . Assume that  $m, \lambda$  are parameters which scale with  $n$  as  $m_n, \lambda_n$  such that
$$
\lim  _ {n \rightarrow \infty} \left(\lambda_ {n} + \frac {1}{m _ {n}}\right) = 0, \quad \lim  _ {n \rightarrow \infty} \left(\frac {1}{\lambda_ {n} m _ {n}} + \frac {\log n}{\lambda_ {n} \sqrt {n}}\right) = 0. \tag {1}
$$
Then almost surely over the random selection of data points in  $S_{n}$ , the following holds: If  $(a,W,b)_{n} \in \operatorname{argmin} \widehat{\mathcal{R}}_{n,m_{n},\lambda_{n}}$  for all  $n \in \mathbb{N}$ , then every subsequence of  $f_{n} := f_{(a,W,b)_{n}}$  has a further subsequence which converges to some limit  $\hat{f}^{*} \in \mathcal{B}$  with  $\hat{f}^{*} = f^{*}$ $\mu$ -almost everywhere and  $[ \hat{f}^{*} ]_{\mathcal{B}} \leq [f^{*}]_{\mathcal{B}}$ . Convergence holds in  $L^p (\mu)$  for all  $p < \infty$  and uniformly on compact subsets of  $\mathbb{R}^d$ . If  $\mathbb{E}_{\mu} \| x \| + \sigma^2 \geq 1$ , then for all  $n \geq 2$ , the following explicit bound holds up to higher order terms in  $n, m = m_{n}, \lambda = \lambda_{n}$  with probability at least  $1 - 1 / n^2$ :
$$
\left\| f _ {(a, W, b) _ {n}} - f ^ {*} \right\| _ {L ^ {2} (\mu)} ^ {2} \leq C \left(\frac {\left[ f ^ {*} \right] _ {\mathcal {B}} ^ {2}}{m} \mathbb {E} _ {\mu} \left[ \| x \| ^ {2} \right] + \left[ f ^ {*} \right] _ {\mathcal {B}} ^ {2} \left(\mathbb {E} _ {\mu} \| x \| + \sigma^ {2}\right) \frac {\log n}{\sqrt {n}} + \lambda \left[ f ^ {*} \right] _ {\mathcal {B}}\right). \tag {2}
$$
Remark 2.2. Theorem 2.1 extends previous results of  $E$  et al. [2019a] in several ways:
1. We allow for general sub-Gaussian rather than compactly supported data distributions.
2. We do not control the magnitude of the bias variables.
3. Our results apply to  $\ell^2$ -loss, which is neither globally Lipschitz-continuous nor bounded.
4. In a limiting regime, we characterize how the empirical risk minimizers interpolate in the region where no data is given by proving uniform convergence to a minimum norm interpolant.
In sum, the first three points necessitate a more careful technical analysis than in the settings of prior work, and the unboundedness of data and loss introduces additional logarithmic terms not present for  $E$  et al. [2019a]. To the best of our knowledge, the last point is novel and our work is the first to use the notion of  $\Gamma$ -convergence in this context.
If the data-distribution  $\mu$  is supported on the entire space  $\mathbb{R}^d$  (e.g. a non-degenerate Gaussian), then  $\hat{f}^{*}\equiv f^{*}$ . In many cases, however,  $\mu$  is supported on a small, potentially compact and generally low-dimensional subset  $M$  of the data space. In this case, the function  $f^{*}$  is only known on the closed set  $M\subsetneq \mathbb{R}^d$ . As a result, there are many  $f\in \mathcal{B}$  such that  $f\equiv f^{*}$  on  $M$  while  $f\not\equiv f^{*}$  on  $\mathbb{R}^d$  in general. The subsequential limit is one of these functions which has a minimal semi-norm  $[f]_{\mathcal{B}}$ .
Thus, beyond knowing that  $f_{n}$  asymptotically fits the function  $f^{*}$  perfectly at known data, Theorem 2.1 provides information about how it may interpolate at points where  $\mu$  has no information. Such knowledge is of interest when a population may naturally evolve in time (distributional shift) of if  $f_{n}$  is applied to a new problem with similar features but distinct geometry (transfer learning).
The proof of Theorem 2.1 is given in Appendix E. In the proof, we combine a direct approximation theorem to construct risk competitors with Rademacher complexity-based generalization bounds. Concentration inequalities are used to bound tail quantities. The scaling conditions that  $\frac{\log n}{\sqrt{n}} \ll \lambda$  and  $\frac{1}{m} \ll \lambda$  are used to ensure that the risk functional has sufficiently strong regularization. They can be weakened if the data measure  $\mu$  is supported on a finite set of points or the function  $f^*$  can be represented by a neural network with a finite number of neurons respectively. For instance, an analogous statement holds with a simpler proof for a fixed data-set  $S$  of  $n$  data points if  $\lambda$  is coupled to  $m$  such that  $\frac{1}{m\lambda_m} \to 0$ . In this case, we take the empirical distribution  $\mu = \frac{1}{n}\sum_{x \in S}\delta_x$  as the population and do not need to bound the generalization gap. This model can be considered appropriate when  $n \ll m$  (heavy overparametrization). A precise statement is given in Appendix F.
The proof remains valid if we only assume that
$$
\limsup_{n\to \infty}\frac{1}{\lambda_{n}}\widehat{\mathcal{R}}_{n,m_{n},\lambda_{n}}(a_{n},W_{n},b_{n}) = \inf \left\{\left[f\right]_{\mathcal{B}}:f\equiv f^{*},\quad \mu -\text{almost everywhere}\right\} ,
$$
i.e. if  $(a, W, b)_n$  parametrizes a function of low excess risk. In the proof, we obtain a more precise version of (2). Using an a priori Lipschitz-bound, it is possible to obtain a rate of convergence in  $L^p(\mu)$  for  $p < \infty$  by interpolation. For uniform convergence on compact sets outside the support of  $\mu$ , we do not obtain a rate in this work.
The proof of uniform convergence on compact sets utilizes the notion of  $\Gamma$ -convergence from the calculus of variations, a very stable notion of convergence which implies that minimizers of approximating functionals converge to the minimizer of a limiting functional. This notion has recently made inroads into machine learning applications and was used e.g. by Neumayer et al. [2023].
All results can easily be generalized to any more general function class which admits the three key ingredients: A bound on its Rademacher complexity, a compact embedding theorem, and a direct approximation theorem.
3 Minimum norm interpolants
3.1 One-dimensional example
In one dimension, [Wojtowitsch, 2022, Proposition 2.5] shows that  $[f]_{\mathcal{B}} = \int_{-\infty}^{\infty}|f''(x)|\mathrm{d}x$  for any smooth function  $f:\mathbb{R}\to \mathbb{R}$  which satisfies  $f^{\prime}(x) = 0$  at some point  $x\in \mathbb{R}$  - see also previous work by Li et al. [2020], E and Wojtowitsch [2020]. This one-dimensional case is, in fact, the simplest case of a general characterization of Barron functions in any dimension by Ongie et al. [2019].
Consider the task of minimizing  $[f]_{\mathcal{B}}$  under the condition that  $f(x) = |x|$  if  $|x| \geq 1$ . Then the minimum is attained for any smooth convex function  $f$  which satisfies the constraint since
$$
2 = f ^ {\prime} (1) - f ^ {\prime} (- 1) = \int_ {- 1} ^ {1} f ^ {\prime \prime} (x) \mathrm {d} x \leq \int_ {- 1} ^ {1} | f ^ {\prime \prime} (x) | \mathrm {d} x = [ f ] _ {\mathcal {B}}
$$
with equality if and only if  $f'' \geq 0$ . The same estimate holds for non-smooth Barron functions if the second derivative is interpreted as a Radon measure. For piecewise linear functions, this corresponds to summing  $|f'(x_i^+) - f'(x_i^-)|$  over the non-smooth points  $x_i$ . More generally, the set of minimum norm interpolants of one-dimensional convex data was characterized by Savarese et al. [2019].
Proposition 3.1. Let  $x_0 < \dots < x_n$  and  $y_i = f^*(x_i)$  for a convex function  $f^*$  and  $i = 0, \ldots, n$ . If  $y_1 < y_0$  and  $y_n > y_{n-1}$ , then  $f$  is a minimum Barron norm interpolant of the dataset  $\{(x_i, y_i)\}_{i=0}^n$ .
if and only if  $f$  is convex,  $f(x_{i}) = y_{i}$  for all  $i = 0, \dots, n$  and
$$
f ^ {\prime} (x) = \frac {y _ {1} - y _ {0}}{x _ {1} - x _ {0}} \quad f o r x <   x _ {1} \quad a n d \quad f ^ {\prime} (x) = \frac {y _ {n} - y _ {n - 1}}{x _ {n} - x _ {n - 1}} \quad f o r x > x _ {n - 1}.
$$
The two given slopes are the largest values that are required for derivatives at any point. We give a proof of Proposition 3.1 in Appendix G. In full generality, minimum norm solutions have been characterized by Hanin [2021] using matching convexities to achieve minimal total curvature.
In a recent article, Boursier and Flammarion [2023] show that if the full Barron norm is controlled, i.e. if the magnitude of biases is included in the regularizer, a specific convex function is selected. This corresponds to minimizing a functional
$$
\int_ {- 1} ^ {1} | f ^ {\prime \prime} (x) | \sqrt {1 + x ^ {2}}   \mathrm {d} x \qquad \text {r a t h e r} \quad \int_ {- 1} ^ {1} | f ^ {\prime \prime} (x) |   \mathrm {d} x.
$$
The first functional prefers  $f^{\prime \prime}$  to be large close to the origin, if it has to be large anywhere. All break points occur as close to the origin as possible, in particular: Either at the origin or at data points. Unlike the Barron semi-norm penalty studied in this work, which does not select a specific minimum norm interpolant, the Barron norm penalty thus induces a sparse neural network interpolant.
3.2 Radially symmetric bump function
Another setting where we have explicit minimum norm interpolant is when we fit a bump function for radially symmetric data. Recall a result of Wojtowytsch [2022] on minimum norm fitting of certain radially symmetric data.
Proposition 3.2. [Wojtowytsch, 2022, Theorem 3.1] Let  $d \geq 3$  be an odd integer and
$$
\mathcal {F} = \left\{f \in C _ {c} \left(\mathbb {R} ^ {d}\right): f (0) = 1 a n d f (x) = 0 i f \| x \| \geq 1 \right\}.
$$
Then there exists a unique radially symmetric function  $f_{d}^{*}:\mathbb{R}^{d}\to \mathbb{R}$  such that  $f_{d}^{*}\in \mathrm{argmin}_{f\in \mathcal{F}}[f]_{\mathcal{B}}$ . The norm of minimizers grows as  $\lim_{d\to \infty}[f_d^* ]_{\mathcal{B}} / d\approx 3.7$
We note that the existence of minimum norm interpolants which are not radially symmetric is not excluded, but if  $\hat{f} \in \operatorname{argmin}_{f \in \mathcal{F}}[f]_{\mathcal{B}}$  is any other minimum norm interpolant, then its radial average  $\operatorname{Av}\hat{f}$  coincides with  $f_d^*$ :  $\operatorname{Av}\hat{f} \equiv f_d^*$ . Here
$$
\operatorname {A v} f (x) := \int_ {S O (d)} f (O x) \mathrm {d} H _ {O} = \int_ {S ^ {d - 1}} f \left(| x | \cdot \nu\right) \mathrm {d} \pi_ {\nu} ^ {0} \tag {3}
$$
where  $H$  is the uniform distribution (Haar measure) on the group of rotations and  $\pi^0$  is the uniform distribution on the  $d - 1$ -dimensional sphere in  $\mathbb{R}^d$ . In [Wojtowytsch, 2022, Section 6], an algorithm is given to find the minimum norm interpolant  $f_{d}^{*}$  by numerically solving a one-dimensional polynomial approximation problem and a linear system of moment conditions.
The uniqueness statement allows us to strengthen the result of Theorem 2.1 in this case. A natural setting is to use a sub-Gaussian data distribution  $\mu$  which gives positive mass to the origin, but has no mass elsewhere in the unit ball. It should have mass everywhere outside the unit ball. Under this natural setting, we have a stronger result of Theorem 2.1.
Corollary 3.3. Take  $\mu, S_n, \widehat{\mathcal{R}}_{n,m,\lambda}$  as in Section 1.2, and assume in addition that
1.  $\mu (\{0\}) > 0$  (positive mass at the origin).
2.  $\mu (B_{1}(0)\setminus \{0\}) = 0$  (no mass elsewhere in the unit ball).
3.  $\mu (U) > 0$  for any open set  $U\subseteq \mathbb{R}^d\setminus \overline{B_1(0)}$  (mass everywhere outside the unit ball).
Assume that  $m, \lambda$  scale with  $n$  as in (1). Almost surely over the random selection of data points in  $S_{n}$ , the following holds: If  $(a, W, b)_{n} \in \operatorname{argmin} \widehat{\mathcal{R}}_{n, m_{n}, \lambda_{n}}$  for all  $n \in \mathbb{N}$ , then sequence of radial averages  $\operatorname{Av} f_{n}$  of  $f_{n} \coloneqq f_{(a, W, b)_{n}}$  converges to  $f_{d}^{*}$  as in Proposition 3.2. Convergence holds in  $L^{2}(\mu)$  for MSE loss (with an explicit rate) and uniformly on compact subsets of  $\mathbb{R}^d$  (without a rate in this work).
In Corollary 3.3, we guarantee convergence to the unique radial minimum norm interpolant  $f_{d}^{*}$  (at least for the radial average), while in Theorem 2.1 we may have different subsequences that converge to different minimum norm interpolants. The proof is given in Appendix E.
4 Relating Interpolation, Optimization and Generalization
For a given bounded set  $K \subseteq \mathbb{R}^d$ , Theorem 2.1 states that for a large number of neurons  $m \in \mathbb{N}$ , a large number of data points  $n \in \mathbb{N}$ , and a small penalty  $\lambda > 0$ , minimizers of the empirical risk functional  $\widehat{\mathcal{R}}_{n,m,\lambda}$  resemble a minimum norm interpolant everywhere in  $K$ . As the Barron semi-norm controls the generalization gap (see Appendix D) and a minimum norm interpolant has minimal Barron norm by definition, this suggests that minimum norm interpolants are optimal in terms of generalization, at least when arguing from this upper bound.
Many authors, including Safran and Shamir [2018b], Venturi et al. [2018], demonstrate that neural network training is a non-convex optimization problem. As such, it is not guaranteed that numerical optimizers (1) converge to interpolants at all, and (2) select minimum norm interpolants out of the large set of different neural networks which interpolate given data, even when a regularizer is included in the training loss functional.