SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, there are settings where an optimization algorithm selects a minimum norm solution even without explicit regularization. This is easily proved for gradient descent on the overparametrized least squares regression problem  $\mathcal{R}_n(a) = \frac{1}{n}\sum_{i=1}^n |a^T x_i - y_i|^2$  with initial condition  $a = 0$  and  $n < m = d$ . Using entirely different methods, Chizat and Bach [2020] prove a similar result for binary classification by shallow neural networks with logistic loss. For regression problems using neural networks, analogous results are not available to the best of our knowledge. This in part motivates the following numerical investigation. Namely, we are interested in exploring the effects of explicit regularization and the implicit bias of optimization algorithms toward minimum norm interpolants. Knowing the analytically optimal solution in between given data provides us the opportunity to compare optimizers on a deeper level than merely testing their performance on unseen data generated from the same distribution.
As seen in Figure 1, the radial profile  $r \mapsto f_d^*(re_1)$  of Wojtowitsch [2022]'s minimum norm interpolant  $f_d^*$  is so close to 0 on  $[r_d, \infty)$  as to be virtually indistinguishable from zero numerically for some  $r_d < 1$  which decreases in  $d$ . Indeed, the first  $(d - 1)/2$  derivatives of vanish at  $r = 1$  due to [Wojtowitsch, 2022, Lemma 4.1] and  $0 \leq f_d^*(x) \leq C d^{3/2}((1 - \| x\|^2) / \| x \|)^{(d - 3)/2}$  due to [Wojtowitsch, 2022, Appendix D.1] for a universal constant  $C > 0$ . In particular, if  $d$  is large,  $\| f_d^*\|_{L^\infty(\mathbb{R}^d \setminus B_{r_d}(0))}$  is negligible compared to the approximation error  $d^2/m$  for any reasonable dimension  $d$  and network width  $m$ . Consequently, the rescaled function  $h_d^*(x) \coloneqq f_d^*(r_d x)$  meets the constraint  $h_d^* \equiv 0$  outside  $B_1(0)$  almost exactly and has the smaller Barron semi-norm  $[h_d^*]_{\mathcal{B}} = r_d[f_d^*]_{\mathcal{B}}$ . For this reason, we compare numerical solutions to the interpolation problem in Corollary 3.3 to rescaled versions of  $f_d^*$  rather than  $f_d^*$  itself, at least in high dimension. For  $r$  below the threshold value  $r_d$ , there is no noticeable trade-off between rescaling  $f_d^*(rx)$  and data-fitting. For larger values of  $r$ , the Barron semi-norm is reduced more significantly, but the data fit becomes appreciably worse.
Our main goal in this section is to gain a more precise understanding of different optimization algorithms by comparing numerical solutions to a known minimum norm interpolant. We consider the two settings in which minimum norm interpolation by Barron functions is best understood: One-dimensional and radially symmetric functions. As a benefit, we can easily visualize the numerical results in both settings. We focus on three questions of interest.
1. Explicit regularization. If  $\lambda > 0$  is moderately small and  $m, n$  are large, then a global minimizer of  $\widehat{\mathcal{R}}_{n,m,\lambda}$  resembles a minimum norm interpolant between known data points due to Theorem 2.1. Is the minimizer which we find numerically close to a minimum norm interpolant, or does it merely fit the function at known data points?
2. Implicit bias. If  $m, n$  are large, does a training algorithm select a minimum norm interpolant out of potentially many possible solutions without explicit regularization (i.e. for  $\lambda = 0$ )?
Figure 1: Left: In Dimension  $d = 31$ , the minimum Barron norm solution  $f_{d}^{*}$  satisfies  $f_{d}^{*} \equiv 0$  on  $\mathbb{R}^d \setminus B_{r_d}(0)$  for  $r_d = 0.4$  to high precision, albeit not exactly. The rescaled function  $f_{d}^{*}(r_{d}x)$  is a suitable candidate for a minimum norm almost-interpolant to high accuracy. Right: For later use, we consider more aggressively rescaled functions  $f_{d}^{*}(r_{d}x)$  for  $d = 15, d = 31$  with lower semi-norm, but worse data fitting properties. We note that the rescalings of these functions which have essentially the same slope as  $f_{3}^{*}$  at  $r = 0$  appear to coincide. We conjecture that this statement allows for a more rigorous formulation.
3. Learning symmetries: The optimal minimum norm interpolant  $f_{d}^{*}$  described in Proposition 3.2 is radially symmetric and satisfies  $0 \leq f_{d}^{*} \leq 1$ . Proposition 3.2 does not rule out the existence of other minimum norm interpolants which are not radially symmetric. Does an optimization algorithm generally find solutions which are (approximately) radially symmetric and confined to the interval [0, 1]? A similar consideration applies in a one-dimensional investigation with reflection symmetry.
The third question is of particular interest for algorithms like ADAM, which operate coordinate-by-coordinate and do not respect Euclidean isometries. By comparison, we expect that SGD, initialized at a radially symmetric configuration, preserves Euclidean isometries. More experiments in similar settings can be found in Appendix A.
5.1 One-dimensional experiments
We consider the classical interpolation problem of numerical analysis: Fit values  $f^{*}(x_{i}) \in \mathbb{R}$  at points  $x_{i} \in \mathbb{R}$  for  $i \in \{1, \dots, n\}$ . In contrast to classical numerical analysis, we consider overparametrized ReLU-networks with a single hidden layer as our model class. As in Section 3, we select  $f^{*}(x) = |x|$  for simplicity.
In Figure 2, a ReLU network with a single hidden layer of width  $m = 200$  was trained to fit  $f^{*}(x) = |x|$  at a symmetric set containing 15 equi-spaced points in (1, 2). Optimizers included SGD (with learning rate  $\eta = 5 \cdot 10^{-5}$  and momentum  $\mu = 0.99$ ), SGD  $(\eta = 10^{-2}, \mu = 0)$ , ADAM  $(\eta = 5 \cdot 10^{-5}$  and default parameters) and the quasi-Newton L-BFGS method. Deterministic gradients based on the  $n = 30$  sample points were used. The final training loss was below  $10^{-4}$  on average. The network weights were initialized by a scaled uniform Xavier initialization, i.e. uniformly in a symmetric interval of length  $2\alpha \cdot \sqrt{6 / (n_{in} + n_{out})}$  where  $n_{in}$  and  $n_{out}$  denote the number of input- and output-units to a layer respectively. The 'gain' factor was selected as  $\alpha \in \{0.5, 1, 5\}$ . Without weight decay and for small gain, the optimizers find a solution close to the smallest possible minimum norm interpolant  $f(x) = |x|$ . The larger the parameters for initialization gain and weight decay penalty, the closer numerical solutions are to the largest possible minimum norm interpolant  $f(x) = \max \{|x|, 1\}$ .
We observe that a higher gain factor  $\alpha$  corresponds to faster initial training, but a high gain like  $\alpha = 5$  produces interpolants which are non-convex without regularization, while a lower gain factor produces convex interpolants in longer time. This observation agrees with the findings of Chizat et al. [2019], who dub the large  $\alpha$  setting the 'lazy training' regime and associate it with worse generalization performance. As Pesme et al. [2021] eloquently put it: "there is a tension between generalisation and optimisation: a longer training time might improve generalisation but comes at the cost of... a longer training time."
If  $m$  is large and  $\alpha$  is not too big, the variation of solutions produced by a training algorithm varies less over different stochastic realizations - see Appendix A for experiments for  $m = 1,000$ . The
Figure 2: We compare numerical approximations of a target function for Momentum-GD (red), GD (magenta), ADAM (purple) and L-BFGS (brown). The target function is drawn in blue and the natural cubic spline in green. For each algorithm, we plot one representative solution to study symmetry selection properties. Vertical grey lines indicate known training data points. The initialization has gain  $\alpha \in \{0.5,1.0,5.0\}$  in the left, middle and right column. The weight decay penalty is  $\lambda \in \{0,0.002,0.005\}$  (top, middle, bottom row). For all optimization algorithms, the final loss is approximately  $0,2 \cdot 10^{-4}$  and  $10^{-3}$  respectively.
For small gain, the optimizer selects a minimum norm interpolant/convex function. For large gain, the interpolant is often non-convex for any one of the optimization algorithms, unless the weight decay is given a positive value. A specific type of minimum norm interpolant in the large set of possible solutions seems to be selected by specifying optimization algorithm, weight decay and initialization. Evidently, initialization and weight decay have far greater influence than the choice of the optimizer. Small gain seems to preference the sparsest solution  $f(x) = |x|$  has predicted with bias penalty by Boursier and Flammarion [2023]. With larger weight decay, the solutions become more convex and more symmetric, but the accuracy of data fitting decreases. Visually, the second order L-BFGS method appears to be the least affected by different choices in initialization.
dynamics are close to those of a limiting 'mean field' model studied by Chizat and Bach [2018], Rotskoff and Vanden-Eijnden [2018], Mei et al. [2018], Sirignano and Spiliopoulos [2020a] and Wojtowytsch [2020]. In these works, the limiting model is typically derived with a factor  $1 / m$  outside the function definition, which is implicit in the initialization here since  $n_{in} + n_{out} \approx m$  for both layers and the ReLU activation is positively one-homogeneous. Global convergence to a minimizer (but not necessarily a minimum norm solution) is guaranteed (up to certain technical assumptions) by Chizat and Bach [2018] and Wojtowytsch [2020].
For comparison, we also present the natural cubic spline interpolant, i.e. the function  $f$  which minimizes the stronger curvature energy  $\int_{-2}^{2} |f''(x)|^2 \, \mathrm{d}x$  under the condition that  $f(x_i) = |x_i|$  for all  $i = 1, \dots, n$ . Unlike the minimum Barron norm interpolants, the natural cubic spline may not be convex (and in fact, it is not if  $f^*$  is replaced by  $h^*(x) = |x - 0.5|$ ).
5.2 Radially symmetric data
We explore the performance of numerical optimization algorithms in the setting of Corollary 3.3 with and without explicit regularization  $\lambda \in \{0,10^{-5}\}$  in dimensions  $d = 3$ ,  $d = 15$  and  $d = 31$ . The numerical solution is then compared to (a rescaled version of) the analytic minimum norm interpolant  $f_{d}^{*}$  described in Proposition 3.2, which we compute by the algorithm described in [Wojtowytsch, 2022, Section 6]. The rescaling factor  $r_d$  is chosen heuristically for an accurate match.
Figure 3: A neural network with a single hidden layer of width  $m = 12,000$  was trained by gradient descent with learning rate  $\eta = 10^{-3}$  and momentum  $\mu = 0.99$  in the setting of Section 5.2. The radial average is sketched by a solid red line. One radial standard deviation around the average, computed over 500 random directions, is shaded.z Top row: Experiment in dimension 3 (left) and dimension 15 (right). The numerical solutions are compared to  $f_{d}^{*}(r_{d}x)$  with  $r_3 = 1 / 1.05$  and  $r_{15} = 1 / 2.55$ . In both cases, the 'minimum norm interpolant' shape is attained to high accuracy. Both solutions are approximately symmetric, more so in low dimension. Bottom row: Numerical approximations to  $f_{15}^{*}(r_{15}\cdot)$  for neural networks of constant width  $m = 12,000$ , trained on data sets of different size (but for an identical number of 200,000 training steps with stochastic estimates computed over a batch of 50 data points). The shape of the radial average is comparable across different dataset sizes, but the fit of the radial average with data is improved and the radial variance reduced for larger datasets. Note that the first two simulations are set in the overparametrized regime, whereas the last experiment on the largest dataset is underparametrized.
Data is generated from a distribution  $\mu = \mu_{1} + \mu_{2} + \mu_{3}$  where  $\mu_{1}$  is a point mass of magnitude  $m_{1}$  at the origin,  $\mu_{2}$  is a uniform measure on the unit sphere  $S^{d - 1}$  with mass  $m_{2}$  and  $\mu_{3}$  is the radially symmetric measure of mass  $1 - m_{1} - m_{2}$  such that  $\| x\| _2$  is distributed uniformly in [1, 7]. We numerically explored various values for  $m_{1}\in [0.1,0.4]$  and  $m_2\in [0.0,0.4]$  and found simulations to be relatively stable under a number of choices.
Results are presented in Figures 3, 4 and Appendix A. We find that all algorithms find a solution with radial average similar to  $f_{d}^{*}(r_{d}x)$ , albeit for rescaling factors  $r_{d}$  which depend on dimension  $d$  and (to a lesser extent) the optimizer. In high dimension, solutions are not perfectly radially symmetric, but the larger amount of variation over a sphere of fixed radius is observed in the domain where  $f_{d}^{*} \approx 0$  rather than in the transition area  $(0, r_{d})$ . Larger datasets improve the compliance with the optimal interface and reduce the radial standard deviation. Solutions do not remain non-negative and drop below zero before leveling off as the radius increases. The drop becomes more noticeable as the dimension increases and less pronounced for wider networks.
The results are essentially identical for normal Xavier initialization and (not radially symmetric) uniform Xavier initialization. In accordance with our expectations, the radial standard deviation is higher for Adam compared to optimizers based in Euclidean geometry. While the neural network function found by Adam resembles a minimum norm interpolant, the weight decay regularizer takes significantly higher values compared to other optimization algorithms.
Shallow ReLU networks converge to minimum norm interpolants of given data: Provably if explicit regularization is included and empirically if it is not. We conclude with a summary of our empirical insight into the implicit bias of neural network optimizers.
Figure 4: Top row: We compare different optimizers (SGD = red, Adam = green and Momentum-SGD = blue) in the setting of Section 5.2 in dimension 31 with  $n = 10^4$  data points. All minimizers attain the minimum norm interpolant shape (left column, rescaled optimal solution in pink) – curiously for Adam, the correct shape is attained despite the fact that at  $\approx 1$ , 600 the weight decay regularizer is an order of magnitude higher than for the other optimizers (right column). The high regularizer value goes hand in hand with a higher radial standard deviation (middle column). The initialization is uniform Xavier (in particular, not radially symmetric). Essentially identical results are observed for the radially symmetric normal Xavier initialization with the same degree of radial symmetry in Figure 8. Bottom row: If we include an explicit weight-decay regularizer with weight  $\lambda = 10^{-5}$ , solutions resemble an optimal interpolant for a smaller rescaling factor  $r_{31,\lambda} = 1/3.8$  compared to  $r_{31} = 1/3.5$  without regularization. This is expected since the norm is weighted more heavily compared to data compliance. Notably, the radial standard deviation does not decrease, even for the (heavily affected) ADAM optimizer. It remains noticeable at 0.1 for  $r = 5$  and Adam.
1. With reasonable (not too large) initialization, all algorithms studied here are biased towards minimum norm interpolant profiles.
2. At least in the case of Adam, this bias is visible on the function level, but not on the parameter level, as the weight decay regularizer increases rapidly to large magnitude. Despite this, ADAM solutions often appear 'flatter' in high dimension with a lower rescaling factor  $r_d$ .
3. Explicit regularization stabilizes towards a minimum norm interpolant shape, but at the cost of a decreased fit with the target values. Its impact is most significant for poorly chosen initial conditions.
4. When the minimum norm interpolant is non-unique, different types of minimum norm interpolants are found depending on the choice of initialization scheme and optimization algorithm. The impact of initialization scale appears more significant.
5. Optimization algorithms which are rooted in Euclidean geometry (such as SGD and momentum-SGD) more successfully preserve Euclidean symmetries compared to the 'coordinate-wise' Adam algorithm.
The last observation is not surprising for radially symmetric initialization laws as radially symmetric parameter distributions induce radially symmetric functions. It is, however, observed also for a uniform initialization scheme which only obeys coordinate symmetries.
We believe minimum norm interpolation to be a useful testbed to study the implicit bias of optimizers and the impact of initialization and regularization. While minimum norm interpolation by deeper networks has not been characterized yet, we anticipate no obstructions to implementing a similar program there in the future.
The research of Jiyoung Park is supported by NSF DMS-2210689.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820-1852, 2020b.
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021.
Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in two-layer neural network optimization landscapes. arXiv preprint arXiv:1802.06384, 2018.
Stephan Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv preprint arXiv:2005.13530, 2020.
Stephan Wojtowytsch. Optimal bump functions for shallow relu networks: Weight decay, depth separation and the curse of dimensionality. arXiv preprint arXiv:2209.01173, 2022.
Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems, 35:4680-4693, 2022.
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 34:18722-18733, 2021.
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285-21296, 2020.
A Numerical Experiments 16
B  $\Gamma$  -convergence 24
C Homogeneous Barron spaces 26
D Rademacher complexity of homogeneous Barron space 29
E Proofs of the convergence theorems 34
F Theorem 2.1 for finite data sets 40
G Minimum norm interpolation in one dimension 40
H Sub-Gaussian random variables 41
A Numerical Experiments
A.1 Hyperparameter settings and computation effort
In all experiments in Dimensions 3 and 15, the following hyperparameter settings were used unless otherwise indicated:
1. Normal Xavier initialization with gain  $\alpha = \sqrt{2}$
2. SGD: Learning rate  $= 10^{-2}$  (Dimension 15),  $10^{-3}$  (Dimension 3).
3. Momentum-SGD: Learning rate  $= 10^{-3}$  and momentum  $\mu = 0.99$
4. ADAM: Learning rate  $= 10^{-3}$  and PyTorch default hyperparameters for  $\beta_{1} = 0.9, \beta_{2} = 0.999, \varepsilon = 10^{-8}$ .
For experiments in Dimension 31, we drop the learning rate for ADAM after 50 of 150 epochs by a factor of 10 and for Momentum-SGD by a factor of 10 after 100 epochs.
In Dimension 3, a learning rate of  $10^{-2}$  was found numerically unstable for SGD without momentum. To compensate for the smaller learning rate and provide a fair comparison, the number of time steps was increased.
All experiments were performed on a free version of google colab or the experimenters' personal computers. One run of the model takes below fifteen minutes on a single graphics processing unit.
A.2 Summary and interpretation of additional simulations
In this Section, we present additional numerical experiments in various situations complementary to those presented in the main body of the text. These include: Wider neural networks (Appendix A.3), experiments with different optimizers (Appendix A.4), experiments with different initialization to explore effects of scale and symmetry and the role of explicit regularization (Appendices A.8 and A.5), experiments with  $\ell^1$ -loss instead of  $\ell^2$ -loss (Appendix A.7) and repeated experiments to visualize the stochastic variation between runs (Appendix A.6).
Additionally, we present and investigation into related settings where our theoretical understanding does not apply: In Appendix A.10, we consider linearized (random feature) dynamics to explore how close we are to a (truly non-linear) neural network model. In Appendix A.11, we consider neural networks with a single hidden layer and leaky ReLU activation instead of ReLU activation. In Appendix A.12, we consider ReLU networks with multiple hidden layers. For a detailed list, see the table of contents below.
A.3 Wide neural networks in one dimension 17
A.4 SGD and ADAM: Dimensions 3 and 15 17
A.5 Radial symmetry in Dimension 31 17
A.6 Gradient descent with Momentum 19
A.7  $\ell^1$  -loss and Huber loss 19
A.8 Initialization scaling and explicit regularization: high-dimensional radial data 20
A.9 He initialization 20
A.10 Linearized dynamics 21
A.11 Leaky ReLU activation 23
A.12 Deeper neural networks 23
Our goal is not to explore questions of loss function, initialization, optimization algorithm and the impact of hyperparameters in a systematic fashion, but rather to establish problems in which a minimum norm interpolant can be found in an explicit fashion as instructive benchmarks to numerically study such questions. As a proof of concept, we provide a partial exploration of the parameter space. For the moment, we find ourselves confined to ReLU networks with a single hidden layer, as this is the only case in which explicit minimum norm interpolants are available. Minimum norm interpolation describes the shape of functions between known data clusters and is thus more expressive than a study of generalization error which is naturally confined to data clusters.
The additional experiments corroborate our findings in the main text. Before the detailed presentation, let us briefly summarize the conclusions.
1. Across a variety of different optimizers, Xavier type (= Glorot type) initialization schemes and loss functions, a minimum Barron norm interpolant-like shape was attained, to varying degrees of accuracy and with different rescaling factors.
2. Solutions are fairly radially symmetric with standard deviation in radial direction at most 0.1 (SGD) and 0.2 (Adam).
3. A geometrically distinct shape is observed for random feature models in the same regime.
4. Explicit regularization has little effect, even for poorly chosen initializations of Xavier type with high gain. We conjecture that the uniqueness of the radially symmetric minimum norm interpolant induces a higher degree of rigidity and bias, compared to the one-dimensional case where the set of minimum norm solutions was diverse (and infinite).
5. Functions display larger variation in the radial direction for He initialization. In this regime, explicit regularization has more apparent and beneficial effects on both solution shape and radial symmetry. The solution does not reduce to the random feature model in this case either.
A.3 Wide neural networks in one dimension
In Figure 5, we present the same experiment as in Figure 2 for wider neural networks with  $m = 1,000$  neurons in the hidden layer.
A.4 SGD and ADAM: Dimensions 3 and 15
We repeat the experiment on Figure 3 for Stochastic Gradient Descent (SGD) optimizer without momentum and for the Adam optimizer of Kingma and Ba [2014]. The results are displayed in Figure 6 and 7 respectively. The results strongly resemble those obtained for the SGD optimizer with momentum in Figure 3.
A.5 Radial symmetry in Dimension 31
As indicated in Figure 4, we present computational results with the radially symmetric normal Xavier initialization in Figure 8.