SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: For wider neural networks, we observe less stochastic variation between runs, as the empirical distribution of neurons is closer to a continuum limit. Solutions are generally more convex and symmetric than their narrow counterparts. The gradient descent optimizer without momentum stands out for its tendency to select solutions with highly localized second derivatives and a preference for piecewise linear functions with few linear regions, while other training algorithms select 'smoother' solutions with curvatures which are dispersed more evenly throughout the domain.
Figure 6: We perform the same experiments as for Figure 3, but with the SGD without momentum optimizer. Learning rate was adjusted to  $10^{-2}$  for dimension 15, but for dimension 3 we just used  $10^{-3}$  learning rate and runned 10 times more epochs, due to stability of neural network training in dimension 3 case. The results are comparable, but the rescaling factors were chosen as  $1/1.15$  in dimension 3 and  $1/2.65$  in dimension 15.
Figure 7: We repeat the experiments of Figure 3, but with Adam. The numerical solutions resemble those found by Momentum-SGD, but better rescaling factors for numerical solutions are  $1/1.2$  in dimension 3 and  $1/2.75$  rather than  $1/1.05$  and  $1/2.55$ , i.e. the functions are 'flatter'.
Figure 8: Results between normal and uniform Xavier initialization are essentially identical in this experiment - compare Figure 4. (Approximate) radial symmetry is attained even when parameters are initialized in a fashion which is not radially symmetric.
A.6 Gradient descent with Momentum
In Figure 9, we present additional runs in the setting of Figure 3. Despite quantitative variation, the geometric shapes of solutions are stable over multiple runs and resemble the minimum norm interpolant  $f_{15}^{*}$  in all cases.
A.7  $\ell^1$ -loss and Huber loss
We repeat the experiment of Figure 3 with the  $\ell^1$ -loss function in the place of  $\ell^2$ -loss. To compensate for the lack of smoothness in the loss function, we reduce the learning rate by a factor of 10 to  $10^{-4}$  and increase the number of epochs by  $50\%$  to compensate. Results are reported in Figure 10.
Figure 9: Three realizations of numerical interpolations in the setting of Section 5.2, computed by SGD with learning rate  $\eta = 10^{-3}$  and momentum  $\mu = 0.99$ . In all cases, the minimum norm interpolant shape is attained approximately, but the radial averages briefly dip below zero and exhibit a local minimum which is not found in  $f_{d}^{*}$ . The variation between runs is notable, but not large.
Figure 10:  $\ell^1$ -loss leads to similar numerical results with a smaller rescaling factor of  $1/2.8$  rather than  $1/2.5$  for  $\ell^2$ -loss. Curiously,  $f(0) > 1$  for these algorithms, while  $f(0) \leq 1$  when optimizing  $\ell^2$ -loss. In Dimension  $d = 3$ , the non-smoothness leads to a minimization problem that is not well resolved by the numerical optimization algorithm.
Similarly, we repeated experiments with the Huber loss function. Since  $|f(x_i) - f^*(x_i)| < 1$  over the data set during the final stages of training, the loss function coincides with  $\ell^2$ -loss in the long run and all experiments are identical to  $\ell^2$ -loss. We therefore do not present additional plots.
In the initial stages of training, Huber loss is more stable numerically than  $\ell^2$ -loss, especially for large gain or He initialization (compare Section A.9).
A.8 Initialization scaling and explicit regularization: high-dimensional radial data
As noted in Section 5.1 and previously by Chizat et al. [2019], the choice of initialization affects the optimization process of neural networks. Motivated by our observations in the one-dimensional case, we consider the effects of initialization and explicit regularization in the radially symmetric setting (Section 5.2). Our results support the earlier claim that the effects of regularization are advantageous for poorly chosen initialization with high gain.
The experiments were performed in higher dimension 31 and with uniform Xavier initialization for the scenario in which it is most challenging to obtain radially symmetric solutions. As in Appendix A.7, we used  $\ell^1$ -loss rather than  $\ell^2$ -loss. To compensate for the non-smoothness of the loss function, we drop the learning rate by a factor of 10 twice during the training process. Similar results were observed for  $\ell^2$ -loss, but the effects of initialization and regularization were less pronounced compared to the  $\ell^1$ -case.
Plots for a single representative run are displayed in Figure 11. The explicit regularizer has the clearest effect in the high gain regime, where explicit regularization helps to achieve a better fit with the optimal transition curve and reduces the radial standard variation. Results were less sensitive to poor initialization than the corresponding experiments in one dimension. We conjecture that a higher degree of rigidity is introduced in this setting by the fact that there exists a unique minimum norm interpolant.
A.9 He initialization
All experiments so far were performed with the initialization scaling of Glorot and Bengio [2010]. Especially for deeper neural networks, the initialization scheme of He et al. [2015] is very popular. Hanin and Rolnick [2018] proves in particular that He et al. [2015]'s normalization avoids the vanishing and exploding gradients phenomenon at initialization in expectation. While this consideration does not apply to our shallow networks, we find it informative to compare the two schemes.
Figure 11: We vary initialization scale ( $\alpha \in \{0.25, \sqrt{2}, 10\}$  from left to right) and consider training without weight decay (top row) and with weight decay  $\lambda = 10^{-4}$  (bottom row). The rescaling factors are chosen to be  $1/3.9$  for  $\alpha = 0.25$ ,  $1/3.8$  for  $\alpha = \sqrt{2}$ , and  $1/4$  for  $\alpha = 10$ .
For shallow neural networks
$$
f _ {m}: \mathbb {R} ^ {d} \rightarrow \mathbb {R}, f _ {m} (x) = \sum_ {i = 1} ^ {m} a _ {i} \sigma \left(w _ {i} \cdot x + b _ {i}\right)
$$
the effect of initialization is as follows:
1. According to Glorot and Bengio [2010], the parameters  $a_i$  and  $w_{ij}$ ,  $1 \leq i \leq m$  and  $1 \leq j \leq d$  are chosen as random variables with mean 0 and standard deviation  $\sqrt{2 / (m + 1)}$  for  $a_i$  and  $\sqrt{2 / (m + d)}$  for  $w_{i,j}$ . In particular, if  $m$  is much larger than  $d$ , then the  $|a_i| \| w_i \| = O(1 / m)$ . As we add  $m$  terms of magnitude  $\sim 1 / m$ , we consider this the 'law of large numbers' scaling.
2. According to He et al. [2015], the parameters  $a_i$  and  $w_{ij}$ ,  $1 \leq i \leq m$  and  $1 \leq j \leq d$  are chosen as random variables with mean 0 and standard deviation  $\sqrt{2 / m}$  for  $a_i$  and  $\sqrt{2 / d}$  for  $w_{i,j}$ . In particular, if  $m$  is much larger than  $d$ , then the  $|a_i| \| w_i \| = O(1 / \sqrt{m})$ . As we add  $m$  terms of mean zero and magnitude  $\sim 1 / \sqrt{m}$ , we consider this the 'central limit theorem' scaling.
Many authors, such as Sirignano and Spiliopoulos [2020a,b, 2019], present the factor  $1 / m$  or  $1 / \sqrt{m}$  explicitly outside the neural network. As observed above, the effect of initialization can be significant. Unsurprisingly, results in the central limit regime, where all neurons contribute similarly at initialization, are more consistent and predictable. Experimental results are presented in Figure 12 in the one-dimensional setting and in Figures 13 and 14 in the case of radial symmetry. Notably, in high dimension, explicit regularization not only reduced radial variation, but also increased data compliance by reducing the rescaling factor  $r_d$ . In this setting, we observe the benefits of explicit regularization over relying on implicit bias only.
A.10 Linearized dynamics
Parameter optimization in neural networks depends heavily on the choice of initialization. While the dynamics are truly non-linear in the regime studied by Chizat and Bach [2018], Rotskoff and Vanden-Eijnden [2018], Mei et al. [2018], Sirignano and Spiliopoulos [2020a] and Wojtowytsch [2020], there are scalings for which the directions  $w_{i}$  remain close to their initialization for all time - see e.g. [E et al., 2019b] for a derivation. In this case, the solution produced by a neural network is similar to that of a random feature model. In this section, we numerically find the minimum norm interpolant of a random feature model by 'freezing' the inner layer coefficients at their random initialization. We find that the random feature solution differs geometrically from the Barron space solution, e.g. in that it is smooth at the origin, where the Barron space solution has a cone-like singularity of the form  $f(x) = 1 - c_d\| x\| _2$  for small  $x$ . In particular, we find that our experiments were set appropriately in the non-linear training regime. See Figure 15.
Figure 12: Experiments for He initialization in one Dimension in the same setting as Figure 2 with Xavier initialization. Left: No explicit regularization, right: Weight decay regularization  $\lambda = 0.002$ . Even for Glorot initialization with large gain, this regularizer was sufficient to induce convexity. For He initialization, it has a notable regularizing effect, but it is insufficient to impose convexity. We observe greater deviation from a linear function in the small intervals between known data points on either side of the big 'gap'.
Figure 13: Top row: Experiments for He initialization in dimension 31 in the same setting as Appendix A.8. Left: No explicit regularization with a rescaling factor  $r_d = 1/4.9$ , Right: Weight decay regularization with  $\lambda = 10^{-4}$  and rescaling factor  $r_d = 1/4.2$ . We observe that under He initialization the effects of the explicit regularizer is even more pronounced. Unlike in other experiments, the presence of regularization increases the rescaling factor and thus improves the fit to training data (for the radial profile).
Bottom row: We repeat the same as above with MSE loss rather than  $\ell^1$ -loss, using the Adam optimizer with learning rate  $10^{-5}$  instead of SGD with momentum, and using an overparametrized rather than underparametrized neural network. Without explicit regularization (left), the radial standard deviation is substantial, while explicit regularization leads to a more radially symmetric function, albeit at the price of a higher rescaling factor. For easy comparison to Figure 3, we present the optimal profile as rescaled in the main document as well. Both functions achieve training loss  $< 10^{-3}$ , but clearly generalization is poor without regularization: While the radial average is close to the target function  $f^{*} \equiv 0$  for inputs with  $\|x\| \geq 1$ , the radial variation is high.
Figure 14: We examine the effects of the explicit regularizer (weight decay penalty  $\lambda = 0$  for the top row and  $\lambda = 10^{-4}$  for the bottom row) while varying dimensions ( $d = 3, 15, 31$  from left to right) in the He initialization scheme. Optimizer settings were identical to the top row in Figure 13. The rescaling factors were  $r_3 = 1/1.15$ ,  $r_{15} = 1/2.1$ , and  $r_{31} = 1/4.2$ . As dimension grows implicit bias may be insufficient to find a minimum norm interpolant shape with reasonable scaling factor and may not enforce radial symmetry. In this case explicit regularizer may have an advantage.
Figure 15: A random feature model trained on the same dataset as the neural networks. The solutions produced this way are geometrically distinct from neural network solutions as they are 'flat' at the origin. The left two figures correspond to different initializations: Gain  $\alpha = \sqrt{2}$  (left) and gain  $\alpha = 5$  (right). Notably, the variation is higher in radial direction in high dimension and higher than for the comparable neural network model. Perhaps surprisingly, higher gain appears to induce a better implicit bias in this case. No explicit regularization was used. Here we initialized the random feature by the same law as the neural network rather than initializing the outer layer at zero, since our goal is to study neural network dynamics, not find the optimal random feature solution. For the right plot, the initialization was random normal with gain 5 in the inner layer and zero in the outer layer with unsurprisingly better results.
A.11 Leaky ReLU activation
As noted by Wojtowytsch [2022], minimum norm interpolation is not stable when passing to an equivalent norm. A Barron space theory can be developed in perfect analogy for networks with the leaky ReLU activation function, and it is easy to see that the 'Barron' spaces for both activation functions coincide with equivalent norms, depending on the negative slope of the leaky ReLU function. However, the minimum norm interpolant  $f_{d}^{*}$  with respect to the ReLU-based Barron-norm is not guaranteed to coincide with the minimum interpolant for the leaky-ReLU-based Barron norm. Experimentally, however, we observe strong agreement between the geometry of numerical solutions here.
A.12 Deeper neural networks
We train neural networks of depth  $L > 2$  to fit the same radially symmetric data as in Section 5.2. We see in Figure 17 that for depth  $L \geq 3$ , weight decay-regularized networks strongly resemble the interpolant  $f_{Lip}(x) = \max \{1 - \| x\| _2,0\}$  with minimal (Euclidean) Lipschitz constant. This function can be written as a composition of two Barron functions  $f = f_{bump}\circ f_{norm}$
$$
f _ {b u m p} (z) = \max  \{1 - z, 0 \} = \sigma (1 - z), \quad f _ {n o r m} (x) = \| x \| _ {2} = c _ {d} \mathbb {E} _ {\nu \sim \pi^ {0}} [ \sigma (\nu \cdot x) ]
$$
Figure 16: Neural networks with a single hidden layer and leaky ReLU activation  $\sigma(z) = \max\{z,0\} + 0.1\min\{z,0\}$  trained in the setting of Section 5.2. Without theoretical foundation, we observe that the shape of  $f_{d}^{*}$  is attained to high accuracy also in this setting with the same rescaling factors as in the ReLU setting. Left: Momentum-SGD, Middle: Adam, Right: Momentum-SGD for a wider network with  $m = 25,000$ .
Figure 17: Neural networks of width 250 and varying depth were trained to fit data generated as in Section 5.2 with weight decay regularizers  $10^{-2}$ ,  $10^{-3}$ ,  $10^{-4}$  and 0 (left to right). The initialization gain variable was chosen as  $\alpha = \sqrt{2}$  as required to avoid exploding and vanishing gradients in deeper networks. Evidently, a small amount of weight decay regularization provides useful geometric prior without diminishing the quality of data fit.
Unlike networks with one hidden layer, deeper networks have positive (or nearly positive) outputs everywhere. While two-layer networks follow the minimum norm interpolation shape closely by the origin and have radial variances which increase outside the unit ball, deeper networks have positive radial variance inside the unit ball, but are essentially radially symmetric outside - compare e.g. Figure 9.
where  $\pi^0$  denotes the uniform distribution on the unit sphere and  $c_{d}\sim \sqrt{d}$  is a dimension-dependent constant. We can thus approximate  $f_{Lip}$  efficiently by neural networks of depth  $L\geq 3$  as long as the first layer is sufficiently wide.
Unlike their shallow counterparts, neural networks with multiple hidden layers have no strong geometric prior without weight decay regularization. With weight decay, the observed behavior was relatively stable over a range of dimensions, initializations scalings and optimization algorithms. We are led to conjecture that  $f_{Lip}$  is a minimum norm interpolant in this setting. The statement remains imprecise at this point as no function space theory for deeper networks with weight decay regularizer has been developed to the same extent as Barron space theory.
B  $\Gamma$ -convergence
In this appendix, we recall the definition and a few properties of  $\Gamma$ -convergence, a popular notion of the convergence of functionals introduced by De Giorgi and Franzoni [1975] in the calculus of variations to study the convergence of minimization problems. Braides [2002], Dal Maso [2012] provide introductions to the theory and its applications. As the notion is likely not familiar to readers from the machine learning community, we provide some full proofs as well.
Definition B.1. Let  $(X,d)$  be a metric space and  $F_{n}, F: X \to \mathbb{R} \cup \{-\infty, \infty\}$  be functions. We say that  $F_{n}$  converges to  $F$  in the sense of  $\Gamma$ -convergence if two conditions are met:
1. (lim inf-inequality) If  $x_{n}$  is a sequence in  $X$  and  $x_{n} \to x$ , then  $\lim \inf_{n \to \infty} F_{n}(x_{n}) \geq F(x)$ .
2. (lim sup-inequality) For every  $x \in X$ , there exists a sequence  $x_{n}^{*} \in X$  such that  $x_{n}^{*} \to x$  and  $\lim \sup_{n \to \infty} F_{n}(x_{n}^{*}) \leq F(x)$ .
Intuitively, the first condition means that  $F(x)$  is (almost) a lower bound for  $F_{n}(x_{n})$  if  $n$  is 'large' and  $x_{n}$  is 'close' to  $x$ , while the second condition means that there is no larger lower bound that we could choose. The sequence  $x_{n}^{*}$  is often referred to as a 'recovery sequence'. Of course,
combining the liminf- and limsup-inequalities, we find that in fact  $F_{n}(x_{n}^{*}) \to F(x)$ . We employ  $\Gamma$ -convergence when dealing with minimization problems where uniform convergence fails, but we hope for convergence of minimizers to minimizers.
Often,  $\Gamma$ -convergence is considered as a continuous parameter  $\varepsilon$  approaches  $0^{+}$  rather than as the discrete parameter  $n$  approaches infinity. The definitions remain largely identical (with obvious substitutions).
To get a feeling for  $\Gamma$ -convergence, we consider a particularly simple situation by looking at two constant sequences of functions. Note that the sequence is constant, not the functions.
Example B.2. Let  $X = \mathbb{R}$  and consider the constant sequences
$$
F _ {n} (x) = f (x) = \left\{ \begin{array}{l l} 1 & x \neq 0 \\ 0 & x = 0 \end{array} \right., \qquad G _ {n} (x) = g (x) = \left\{ \begin{array}{l l} 0 & x \neq 0 \\ 1 & x = 0 \end{array} \right..
$$
We claim that
$$
\left(\Gamma - \lim  _ {n \to \infty} F _ {n}\right) (x) = f (x), \quad \left(\Gamma - \lim  _ {n \to \infty} G _ {n}\right) (x) = 0 \qquad \forall   x \in \mathbb {R}.
$$
If  $x_{n} \to x$  and  $x \neq 0$ , then  $F_{n}(x_{n}) = 1$  and  $G_{n}(x_{n}) = 0$  for all but finitely many  $n \in \mathbb{N}$ , meaning that  $F_{n}(x_{n}) \to 1 = f(x)$  and  $G_{n}(x_{n}) \to 0$ . It remains to consider the case  $x_{n} \to 0$ .
We see immediately that  $F_{n}(x_{n}) \geq 0 = f(0)$  for all  $n \in \mathbb{N}$ . Conversely, if we take  $x_{n}^{*} = 0$  for all  $n$ , then  $x_{n}^{*} \to 0$  and  $F_{n}(x_{n}^{*}) = f(0) \to f(0)$ . In total, we conclude that  $\Gamma - \lim F_{n} = f$ .
For  $G_{n}$ , we find that  $G_{n}(x_{n}) \geq 0$  for all  $n \in \mathbb{N}$ . Additionally, we can choose the sequence  $x_{n} = 1 / n$  such that  $G_{n}(x_{n}) = 0$  for all  $n \in \mathbb{N}$ . Altogether, we find that  $\Gamma - \lim G_{n} = 0$ .
More generally, if  $F_{n} = F$  for all  $n \in \mathbb{N}$  and some  $F: X \to \mathbb{R}$ , then  $\Gamma - \lim_{n \to \infty} F_{n} = \overline{F}$  is the lower semi-continuous envelope of  $F$ . In particular,  $\Gamma - \lim_{n \to \infty} F = F$  if and only if  $F$  is lower semi-continuous. The main useful properties of  $\Gamma$ -convergence are summarized in the following lemma.
Lemma B.3. Assume that  $F_{n}\to F$  in the sense of  $\Gamma$ -convergence,  $\varepsilon_n\rightarrow 0^+$  and  $x_{n}\in X$  is a sequence such that
$$
F _ {n} (x _ {n}) \leq \inf  _ {x \in X} F _ {n} (x) + \varepsilon_ {n}.
$$
Assume that  $x_{n} \to x^{*}$ . Then  $F(x^{*}) = \inf_{x \in X} F(x)$ . In particular, if  $x_{n}$  is a minimizer of  $F_{n}$  and the sequence  $x_{n}$  converges, then the limit point is a minimizer of  $F$ .
Clearly, this is most useful if we can guarantee that the sequence  $x_{n}$  converges. For many useful sequences of functionals, the existence of a convergent subsequence can be established by compactness. This is easily sufficient, as we can also pass to a subsequence in  $F_{n}$ .
Proof. Due to the liminf-inequality, we have
$$
F (x ^ {*}) \leq \lim  _ {n \to \infty} \inf  _ {n \to \infty} F _ {n} (x _ {n}) = \lim  _ {n \to \infty} \inf  _ {x \in X} F _ {n} (x).
$$
On the other hand, let  $x \in X$  be any point. Then, due to the limsup-inequality, there exists some sequence  $x_{n}^{\prime}$  such that
$$
x _ {n} ^ {\prime} \to x, \qquad F (x) = \lim  _ {n \to \infty} F _ {n} \left(x _ {n} ^ {\prime}\right) \geq \lim  _ {n \to \infty} \inf  _ {x \in X} F _ {n} (x).
$$
In particular  $\inf_{x\in X}F(x)\geq \lim \inf_{n\to \infty}\inf_{x\in X}F_n(x)$ . Combining the two estimates, we find that  $F(x^{*})\leq \inf_{x\in X}F(x)$ , which means that  $x^{*}$  is a minimizer of  $F$ .
For completeness, a few observations are in order.
1. The notion of  $\Gamma$ -convergence relies on the notion of convergence on the underlying space  $X$ , and  $\Gamma$ -limits can change when keeping the set  $X$  fixed, but passing to a different topology (e.g. a weak topology in infinite-dimensional spaces).
2.  $\Gamma$ -convergence is made for minimization problems, and it does not behave well under multiplication by negative real numbers: In general  $\Gamma - \lim(-F_n) \neq -(\Gamma - \lim F_n)$ , even if both limits exist. The reason is the asymmetry between the lim inf- and the lim sup-condition. To see this, consider for instance  $F_n$  and  $G_n - 1$  in Example B.2.
3. If  $F_{n} \to F$  and  $G_{n} \to G$ , it is not necessarily true that  $F_{n} + G_{n} \to F + G$ . While it remains true that  $\lim \inf_{n\to \infty}(F_n + G_n)(x_n) \geq (F + G)(x)$  if  $x_{n}\rightarrow x$ , it may no longer be possible to find a recovery sequence  $x_{n}$  for  $F_{n} + G_{n}$ . For example, if  $F_{n} = 1_{\mathbb{Q}}$  and  $G_{n} = 1_{\mathbb{R}\backslash \mathbb{Q}}$  for all  $n\in \mathbb{N}$ , then  $F_{n} \xrightarrow{\Gamma} 0$  and  $G_{n} \xrightarrow{\Gamma} 0$ , but  $F_{n} + G_{n} = 1$  for all  $n$  and  $F_{n} + G_{n} \xrightarrow{\Gamma} 1$ .
However, if  $F_{n} \xrightarrow{\Gamma} F$  and  $G_{n}$  converges to a continuous limit  $G$  uniformly, then  $(F_{n} + G_{n}) \xrightarrow{\Gamma} F + G$ . In particular, uniform convergence implies  $\Gamma$ -convergence. Namely, if  $G_{n} \to G$  uniformly,  $G$  is continuous and  $x_{n} \to x$ , then for given  $\varepsilon > 0$ , we can choose  $N \in \mathbb{N}$  so large that
(a)  $|G(x_{n}) - G(x)| < \varepsilon / 2$  for all  $n \geq N$  since  $G$  is continuous at  $x$  and
(b)  $|G_{n}(x_{n}) - G(x_{n})| < \varepsilon / 2$  for all  $n \geq N$  due to uniform convergence.
$$
\left| G _ {n} \left(x _ {n}\right) - G (x) \right| \leq \left| G _ {n} \left(x _ {n}\right) - G \left(x _ {n}\right) \right| + \left| G \left(x _ {n}\right) - G (x) \right| <   \varepsilon .
$$
In particular  $G_{n}(x_{n}) \to G(x)$ . Recall that  $G$  is guaranteed to be continuous if  $G_{n}$  is continuous for all  $n \in \mathbb{N}$ .
4.  $\Gamma$ -convergence is unrelated to pointwise convergence of functions: Neither does it imply pointwise convergence, nor is it implied by it. Namely, the sequence  $G_{n}$  in Example B.2 has the function  $g$  as a pointwise limit and the constant function 0 as a  $\Gamma$ -limit.
5.  $\Gamma$ -convergence is not a notion of convergence derived from a topology. Indeed, even if  $F_{n}$  is a constant sequence, i.e. if  $F_{n} = G$  for all  $n$ , it may happen that  $\Gamma - \lim_{n \to \infty} F_{n} \neq G$  (see  $G_{n}$  in Example B.2). The  $\Gamma$ -limit is related to  $G$ , though: It is the lower semi-continuous envelope of the function  $G$ . In fact, every  $\Gamma$ -limit is lower semi-continuous.