Flat Channels to Infinity in Neural Loss Landscapes
Authors: Flavio Martinelli
Alexander van Meegen
Berfin Simsek
Wulfram Gerstner
Johanni Brea
Venue: NeurIPS 2025 poster ()
Keywords: Loss Landscape
Critical Points
Optimization
Gated Linear Units
Flat Minima
Sharp Minima
Primary area: deep_learning
TL;DR: MLPs contain "channels to infinity" where pairs of neurons evolve to form gated linear units with diverging output weights, creating regions that appear like flat minima but actually have slowly decreasing loss value

Abstract: The loss landscapes of neural networks contain minima and saddle points that may be connected in flat regions or appear in isolation. We identify and characterize a special structure in the loss landscape: channels along which the loss decreases extremely slowly, while the output weights of at least two neurons, $a_i$ and $a_j$, diverge to $\pm$infinity, and their input weight vectors, $\mathbf{w_i}$ and $\mathbf{w_j}$, become equal to each other. At convergence, the two neurons implement a gated linear unit: $a_i\sigma(\mathbf{w_i} \cdot \mathbf{x}) + a_j\sigma(\mathbf{w_j} \cdot \mathbf{x}) \rightarrow c \sigma(\mathbf{w} \cdot \mathbf{x}) + (\mathbf{v} \cdot \mathbf{x}) \sigma'(\mathbf{w} \cdot \mathbf{x})$. Geometrically, these channels to infinity are asymptotically parallel to symmetry-induced lines of critical points. Gradient flow solvers, and related optimization methods like SGD or ADAM, reach the channels with high probability in diverse regression settings, but without careful inspection they look like flat local minima with finite parameter values. Our characterization provides a comprehensive picture of this quasi-flat region in terms of gradient dynamics, geometry, and functional interpretation. The emergence of gated linear units at the end of the channels highlights a surprising aspect of the computational capabilities of fully connected layers.

Flat Channels to Infinity in Neural Loss Landscapes
Flavio Martinelli'
Alexander van Meegen" Berfin Simşek?
Wulfram Gerstner' Johanni Brea
EPFL, Lausanne, Switzerland
2 Flatiron Institute, New York, USA
*equal contribution
{flavio.martinell] alexander vanmeegen, brep@ ch
Abstract
The loss landscapes of neural networks contain minima and saddle points that may connected flat regions or appear isolation. We identify and characterize special structure in the loss landscape: channels along which the loss decreases extremely slowly, while the output weights of at least two neurons, ag and aj. diverge to +infinity, and their input weight vectors, w, and uy,, become equal to each other. At convergence, the two neurons implement a gated linear unit: aao(w; x + a,o a,o(w; x) + cow x) + x) x)o'(w x). Geometrically, these channels to infinity are asymptotically parallel to symmetry-induced lines of critical points. Gradient flow solvers, and related optimization methods like SGD or ADAM, reach the channels with high probability in diverse regression settings, but without careful inspection they look like flat local minima with finite parameter values. Our characterization provides a comprehensive picture of these quasi-flat regions in terms of gradient dynamics, geometry, and functional interpretation. The emergence of gated linear units at the end of the channels highlights surprising aspect of the computational capabilities of fully connected layers.
a. y=
aj a==(1
(=-117)) 11aa)a
loss
Saddle line
lin.
1
2
Channel 0o
W
to infinity
W
W
V
Figure 1: Saddle lines à la Fukumizu & Amari [1] and channels infinity. Left: Duplicating
a neuron in network trained to convergence generates lines of saddle points in the loss landscape M. Duplicated neurons share the input weights of the original neuron while their output weights ya, (1 7) sum to the original neuron's 's output weight a. Middle: Loss landscape of duplicated network projected along the saddle line (in red) and the eigenvector of the smallest (most negative) eigenvalue of the loss Hessian. Parallel to the saddle line there are channels to infinity (green curve) along which the loss decreases very slowly. Following the channel, the output weights diverge to infinite norm and the input weights converge to a new value. Right: The solution at infinity implements a new function consisting single neuron and a gated linear unit. The gating function the derivative of the original activation function a.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Although neural network loss functions are known to be non-convex, machine learning practitioners usually find good solutions with gradient-descent-ln methods. This is puzzling, because the non- convexity is not obviously benign 2 3 One simple example of non-convexity are critical points resulting from embedded sub-networks. In fact, an arbitrarily large network can be constructed by duplicating hidden neurons of smaller networks, such that the optimal loss of any embedded sub-network appears as a manifold of critical points in the loss function of the full network [1] [ Typically, neuron duplication results in the transformation of local minima into line of saddle
points [5 d. but it is also possible that this line contains segments of local minima D 5 There are good theoretical arguments that local minima are unlikely, in particular in multilayer settings (see Appendix and (7)), and that the lines of saddle points are unproblematic [8]. Thus. the saddle lines are expected to be a benign non-convexity. Empirically, we confirm that these symmetry symmetry-induued induced saddle lines are unproblematic even in low-dimensional settings (see (eccionn 2 and Appendix A). However, we find that there are many channels the loss landscape that are asymptotically parallel to symmetry- -inducer lines of saddle points, and lead to local minima at infinity (see Figure I). These minima at infinity differ from the well-known solutions at infinity of separable classification problems [9]. They are characterized by input vectors of at least two neurons becoming equal to each other at finite values, while their output weights diverge to + infinity. Our main contributions are:
1. We identify and characterize a novel structure in the loss landscape of fully connected layers: the channels to infinity (section 3).
2. We show empirically that gradient-based training reaches these channels with high probabil- ity from the random initialization schemes used in practice (section 3).
3. We explain the functional role of the channels to infinity: they endow the network with novel computational capabilities by implementing gated linear unit (section 4).
1.1 Related Work
Neuron splitting creates continuum of critical points: Fukumizu and Amari I were the first to report the mapping of critical points of the loss function of a neural network to a line of critical points of the loss function of new neural network with extra neuron. Moreover, they studied the Hessian of the loss on this line of critical points and showed that regions of local minima appear, if a certain condition is satisfied. When this condition is not satisfied, neuron splitting creates strict saddles enabling escape [10 11I which is utilized to propose alternative algorithms to train neural networks by sequentially growing them
With similar reasoning, splitting of multiple neurons [5] and neuron-splitting in deep networks 14 were analyzed.
22
13
Sub-optimal minima: For extremely wide neural networks, the loss function does not possess any sub-optimal minimum under general conditions [5
(see [8] for a review). However, there are families of sub-optimal minima for von-overpuramcttrrize and salllly-overprrmmeterizedd neural networks 19 20, 21). Neuron splitting turns sub-optimal minima into saddles 5 7), and an overparamcterization with a few extra neurons makes the optimization problem significantly easier In the context of the spiked tensor model, an intricate structure of saddle points allows gradient flow circumvent sub-optimal local minima (24]
T6 17
22 23).
Flat regions loss-landscapes: Precisely- flat regions in loss landscapes are described in various works, from manifolds of symmetry-induced critical points 5 6 10 to manifolds of global minima
in overparameterized 25 26 27
28
29 3D) and teacher-student setups 6. 23 Empirically, flat
regions in loss landscapes were found in interpolations between solutions of different training runs, a phenomenon typically called mode-connectivity [31 [32,, more prominently after permutation alignment [33 34). These locally flat manifolds have been discovered to be star-shaped [35 36 or to
present orthogonal tunnels that spread in the parameter norm 37
Attractors at infinity in neural networks: The well-known minima at infinity in separable classification problems [ differ from the ones we investigate here. In linear networks, and in one- -i-dimensioonl
sigmoidal deep networks trained with a single datapoint, local minima at infinity do not occur [88. but there are examples where minima at infinity can arise with two or more datapoints [38 39
is possible to eliminate finite-norm local minima from landscapes through the addition of specific neurons directly connecting input to output 40 and eliminating local minima at infinity with regularization 42).

---

1.2 Setup
To expose the key ideas, we consider neural networks with a single hidden layer and mean--quared error loss. The main insights generalize to multiple layers and do not depend on the loss function (see sectiion 2). More explicitly, we consider the loss landscape
(0; =
y
aoolwy - x!)))²
The parameters are 0 (wi..... W,, a1,.... ar), the data D = (x(1)))))) y )) is fixed, and is
continuous activation function. The input samples a!1) E IR++1 1 are composed of d-dimensional input vectors and a constant dd-1 to include bias term wa+1 w x Ei=r Wh + Wd+1-
2 Neuron duplication introduces lines of critical points
We start with an empirical investigation of the symmetry-induced inducedd saddle lines. Fukumizu and Amari (α), Theorem 1) showed that any critical point of the loss function with r neurons implies an equal-loss line of critical points of the loss function with r + 1 neurons. Formally, if 0" (wij.... w ai ................................ ..)) is a critical point of the loss function, i.e.. VoL, (0"; D) 0, then the parameters
02,0) 0 (Wi..... wi W ai. az 0) +
0, 0.0 1.+11
(2)
vectors
r+1 weights
r+1 vectors r+1 weights
of neural network with one additional neuron are also a critical point, i.e., VgL,+1 for any E R, and L, DD) (0"; The variable 7 parametrizes the line I (0 RR that points in direction (O.....
-1); we call this line the saddle line. The stability of the symmetry-induced critical points 0 of the L,+1 loss depends on the specific choice of 7, and the spectrum of a symmetric (d + 1) x (d + 1)-dimensional matrix (ΠI: Theorem 3). If and only this matrix is positive or negative definite, there is region of local minima on the saddle line, which we call a plateau saddle, because it is bounded by strict saddle points (see (igure 2b).
Given the amount of available duplications in network, the number of saddle lines in the loss landscape grows factorially with network width (6. What are the chances of finding stable region of the saddle line plateau saddle in the loss landscape? To obtain a comprehensive view of all minima, we trained an extensive set of small networks of increasing widths on d = 2, scalar regression problem (Figure 2. Appendix A). In a setup where neurons have no bias and the output is a scalar, we find that many saddle lines contain plateau saddles, where gradient dynamics converge. This is evident in Figure 2p, where networks of different sizes converge to identical loss values, since they compute identical functions. These converged networks contain at least two neurons of equal plateau saddle
input weight vector (Equation 2). the signature of a plateau saddle. Convergence occurs with probabilities from 10% to 30% across random initializations (Figure 2p, inset).
The loss landscape around plateau saddles is studied in Figure 21, where we perform a perturbation analysis on solution found in FFiuure 2, and analyze its stability. Plateau saddles are attracting (all eigenvalues of the Hessian are non-negative), flat, regions of the landscape (Figure 2b). they can be present the segment ? E (0, 1), or outside, or nowhere (M; Theorem 3). This distinction between segments of the saddle line is important, because at its boundaries (7 = 0, 1), some eigenvalues of the Hessian meet at zero, and swap sign; a signature of a highly degenerate point in the landscape. In all other regions of 7, eigenvalues do not cross and instead exhibit an eigenvalue repulsion phenomenon
43
(see Figure A4).
Since the number of unique minima can increase exponentially with network width, it is unfeasible to comprehensively characterize the set of minima for larger networks (24. However, we verify empirically (Appendix A) Figure A1). that saddle lines lose almost all plateau-saddles if we add biases to the network. similar conclusion was predicted with theoretical arguments for the case of multi-output or multi-layer MI.Ps 77]. Although these saddle lines do not seem to pose directly any problem realistic settings, these equal loss lines through the entire parameter space may have special impact regions nearby.
3

---

a
Networks
b
1.0
networrk with duplicate neurons
Saddle
line
Plateau-saddle
0.8
105
loss
Strict
saddle
network size
Target
0.6
loss
0
d
0.4
distance from saddle line
trajectory
final
Unique minima
46
10
32
19
16
10
10
8
2
2
10
0.2
-2
1
network size
frequency
90
projection on saddle line
Figure 2: Stable plateau-saddles and their loss landscape in ML.P without bias. (a) Networks of hidden neurons and scalar output are trained on the shown 2D regression target (logarithm of the rosenbrock function, sce/Appendix A). Training follows full-batch gradient flow dynamics until convergence to a critical point. A quantification of unique solutions in weight-space (up to permutation symmetries) is shown at the bottom.
(b) Loss levels of converged networks: each diamond shows the loss of a converged network, color-code indicates network size. The only source of randomness is the initialization. Many identical-les solutions found by networks of different sizes. Inset: Frequency of converged solutions exhibiting duplicated neurons. (c) Loss landscape along the duplication parameter 7 and the direction of smallest eigenvalue of the Hessian arémin (7) corresponding to one of the converged solutions shown in b. Small perturbations are stable only within the plateau-saddl: region, 7 (0, 1). (d) Gradient-fol floww trajectories following a small perturbation from the saddle line in the direction me for the example shown in c. Perturbations outside the plateau-saddle region, (0, 1), escape the saddle line and land in other minima.
3 Seemingly flat regions in the loss landscape as channels to infinity
Where do gradient descent dynamics converge, if initialized near saddle line? Using ODE solvers to integrate the gradient flow dynamics 0 = (O; we discovered channels in the loss landscape that are characterized by:
1. diverging output weights of at least two neurons, |a + + oo of opposed signs a,az < 0,
2. decreasing distance between corresponding input weight vectors d(w,, wy) -
Due to these properties we refer to these structures as channels to infinity.
Figure shows a detailed example of the landscape around a saddle line. To first produce saddle line we (i) train an MLP until convergence into a finite-norm local minimum 0, (ii) apply neuron- duplication, 0 + 0", various 7 values (Equation 2p, and (iii) compute the loss function on planar slice spanning 0 and the vector a'Cminn where a E IR and Ciin is the smallest (most negative) eigenvalue of the loss Hessian, - DD) In Figure 3/. we see the landscape characterized by a channel of decreasing loss running parallel to the saddle line. These channels continue indefinitely the direction of the saddle line, Figure 3p. Since this view is neglecting all other dimensions the loss landscape, conducted a perturbation analysis from the saddle line to track the gradient-flow trajectories. In this specific example, only one of the two channels attractive. Trajectories of dynamics trapped in the channel are characterized by very slow decrease in loss (Figure 3d)) and

---

a
b
loss x10-2
Channel to infinity
x10-2
Saddle line
5.5
5.7
loss
5.4
5.5
1
C
-0.50
0.25
0.00
0.25
0.50
0.2
0.2
2
0.0
0.0
3
0.2
Q
-0.
d
e
100
102
104
7
cos(AA, r)
a
loss
0.05
a
0.04
0
10-2
100
102
104
toDE
10 1
loss loSS
cos ()
10 4
10-7
0
1
1.5
3
5 10
25
10
Figure 3: Channels infinity. (a) Loss landscape of 4-4-1 MI.P trained on a regression task (Appendix B). The saddle line (red straight line) is found via neuron splitting from local minimum of a 4-3-1 MLP. The surface a slice of the loss along the splitting parameter y and the direction of smallest (negative) eigenvalue of the Hessian a'min- Most other eigenvalues are positive. At first glance, it looks as if there were two channels to infinity parallel to the saddle line, but the analysis in the next panels reveals that there is only one (the green curved line). (b) Loss profile along aCmin color-coded at different values of 7. Note that the loss is not continually decreasing for positive a, indicating that this is not channel infinity.
(c) A top-view of the landscape for large y reveals that the local picture of the loss landscape in (a) holds also for very large n. (d) The two-dimensional projection of the loss landscape in panels (a)-(c) does not show how the loss depends all the other free parameters of the 4-4-1 MI.. Therefore, we look at gradient-flow trajectories following a small perturbation along emn from the saddle line 7 1.5. The perturbation direction is shown as green and orange arrows on the surface plot in panel (a). After the green perturbation (a-), the gradient trajectory moves inside channel infinity towards increasing values of 7 following descent with extremely small slope (green channel to infinity). The orange perturbation (a.4) converges to finite-norm minimum, which confirms that the landscape positive a is not channel infinity. (e) Cosine distance between parameter updates A and direction of the saddle line (T) for the a perturbation: after an initial high-dimg high-dimensional trajectory, parameter updates AO are parallel the saddle line. The ODE dynamics reveal an extremely slow divergence of
updates that become parallel to the saddle line (Figure 3e). To report 7 for trajectory that is not on a saddle line, we measure the projected 7 by averaging the y estimates ar/a, (for neuron r), and (a, a,+1 of)/2a, where at is the output weight of the original duplicated neuron and a,,, ar+1 are the output weights after duplication. Note that + too + Ja,1 aa,+1 o0.
(a: ar+1)/a 1) (for neuron + 1), leading
Figure 4 shows summary statistics of channels infinity found across different training conditions; importantly, we start in all cases from random initial conditions (Glorot normal initialization). We trained MLPs with different architectures varying layer sizes, input dimensionality, shallow and deep
MI.s- and across various datasets the modified rosenbrock function (a log-polynomial) and four types Gaussian processes (GP) characterized by different kernel scalings E 0.1, 2, 10 (see Appendix B for more details). All experiments were performed with biases and softplus activation
5

---

b
a
10
2.5
60%
early training
C
100
[ai] ++ |aj]
frequency
late training
% channels
51
10
50
30%
20
10"
ar. 0
10
x
2
0%
H:. Cari sodl
rosenbrock
GP (s 0.
GP (s 0.5))
GP (s
GP = 10.0)
0.00
0.50
1.00
10
10
10
ming d(w, Wj)
Inside channels subspace
fraction of update
dataset
density P
e
finite channel
0.1
10
10
Ama
charnels
Ami
-10
0.0
10
-10
10
10
10
10
10²
10
10
t0
10
10
10
10
loss
tODE
tODE
f
g
Location of channels between layers
Dimensionality of channels
L1
103
Linkk
L1&L3
6%
7%
L1&L2
37%
count
10²
9%
10'
9%
L18L28L3
None
L3
L2
123
12
12
123
123456
18%
GP (s 0.1)
GP (s 0.5)
2.0)
10.0)
rosenbrock
GP (s
GP
L28L3
Figure 4: Frequency and properties of channels to infinity. (a) As first criterion to identify channels to infinity, we consider the cosine distance of the pair of closest input weight vectors within a network and the sum of absolute output weights corresponding that pair. Putative channel solutions are identified by having a large weight norm and small distance in input weights (top left section of the graph). (b) As a second criterion,
consider channel solutions networks that had parameter updates mostly within the subspace spanned by the putative channels, parallel to the saddle line. After filtering for networks that satisfy (a), compute the percentage of updates that lie inside the channels subspace. At late stage of training. most of the network updates are parallel to the saddle line (c.f. Figure 3b). (c) Estimated probability of converging and architectures spanning different input and hidden dimensions and number of hidden layers (see Appenddxx B for details). (d) Distribution of finite-norm minima and channel minima training loss. There is no evident difference between types of minima (see Appendix B). (e) Trajectories of maximum and minimum Hessian eigenvalues for finite-none nore and channel minima. Channel solutions have larger maximum cigenvalues, indicating that they are sharper than the finite-norm minima. Channels sharpen as training progresses. They are extremely flat regions, as indicated by the small magnitude of the minimum eigenvalue. (f) For MIPs with three hidden layers, channels appear in all layers, and multiple layers the same time. (g) Channels do not always involve pairs of neurons, they can be formed by an arbitrary number of neurons with diverging output weights and converging input weights. In these cases, the flat regions are multi-dimensional (see) Aecendipxiix for details on multi-dimensional channels). (b-c-d-e) show results for the GP (s=0.5) dataset; see Appendix B for other datasets.
channel for various datasets
function. We identify channels following three criteria: high parameter norm, (ii) low distance between at least a pair of input weight vectors, (jii) updates only within the parameters contributing to channels (more details in Figure 4p.b, and[Appendix B}. We observe a substantial probability of reaching channels in all tested dataset. More channels are found when the target function is rough, as evidenced by the positive correlation with the scale of the GP kernels s (Figure 4 FFiurre B8). The distribution of losses between channels and finite-norm local minima seem to be similar (Figure 40),

---

suggesting that these solutions may not be worse than other local minima. Maximum and minimum eigenvalue of the Hessian for all networks trained on the GP(s 0.5) dataset are shown in Figure 4p. We observe clear difference between finite-norm solutions (red) and channels: training progresses the channels become both sharper and flatter than finite-norm minima (see Figure B12 for a slice of the landscape along the steepest direction). In multilayer MLPs, channels can appear in any layer, and also simultaneously in different layers (Figure 4D). Finally, channels are not limited to be one-dimensional: multiple pairs of neurons can generate higher dimensional channels to infinity Tiguuee 4g). We tested networks with smooth activation functions, such as softplus, erf, sigmoid, and found channels appearing in all settings (see Isee/Appendi B for further details, including brief discussion of relu activation functions).
In summary, striking features of these channels are their extreme flatness along the direction of the saddle line, and the increasing sharpness as the parameter norm diverges to infinity. As the gradient flow dynamics does not seem to stop in these channels, we hypothesize that they lead to local minima at infinity.
Channels to infinity converge to gated linear units
What is the functional role of the channels? To better understand the convergence in these channels, we consider two neurons i and j and reparameterize them as
a. (w; (.)):)))))))) xx ajo(wa x)
2 (o((w + + xx) + o - x))
(o
+ (A) x) aaww (w (A) .z))
with v = (w,, + wy)/2, -|0; - w A
w. w.-w w = c(a; - aj),
+ aj. The
ww
second term in Equation 3Jis the central finite difference approximation of the derivative in direction A. In the limit e + 0 with fixed w, A, a, c, the central difference converges to the derivative,
o x)- a,0(w, x)) =0,, co(ww x)) + a(A x)o'(w xx)
Interestingly, the second term implements a gated linear unit, with gate (ww x) and linear trans- formation v a(A z). If (zx) log(1 + exp(#)) is the softplus activation function, the gate is given by the standard sigmoid o' (x) = 1/(1 + exp(-x)) used in GLU 45). Importantly, the limit 0 with fixed corresponds exactly
ww,l = 2c), and diverging output weights a; aj = a/c, which we observed in many solutions (Figure 4). Note that the limit e + 0 with fixed implies that the readout weights diverge with 1/e.
converging input weights (ww: tww-w,]
To clarify the geometric relation between the channels and the saddle line we write the above reparameterization in the full parameter space
(Wi..... w. W. a1,.... c/2, c/2) + (O,..., <A, - -E,, 0, //........./..._a///.
(5)
where we chose neurons i and j to be the last two neurons w.l.o.g. For small the second term becomes 2. (O.... 0, 0, O..... 1, - 1) which is parallel to the second term in Equation the channels become asymptotically parallel to the saddle line with y 1/c.
a
priori, is unclear whether the loss function in in/Equation I Thas minima at the end of channels to infinity; it could be that the optimal in Equation 3 is at a finite value. Empirically, we never saw the ODE solvers stop in the channels, but they make rather slow progress, and with reasonable compute budgets we never reached values of 1/e above few hundreds, even with small networks. To obtain further empirical evidence for the optimum at infinity, we used a jump procedure, which alternates between dividing e by constant factor, and integrating the gradient flow for mixed amount of time, to move quickly along the channels (see Appendix C). With this jump procedure we see that the loss continually decreases, c, a, w. A seem converge, and the approximation error in Equation 4] decreases quadratically with (see Figure 5).
Proving convergence to the gated linear unit in Equation is difficult for general settings. However, expanding the loss in around = 0, leads to C. (0; D) (0,; D) + (h D) + O(e²). where On are the parameters for - 0. and h On; D) denotes the leading order correction term. We see this scaling of the loss with e² in Figure 5] If 0, a local minimum of the loss, it is attractive in e, h(Oo; D) > 0 (to prove full stability, one would need to look at the spectrum of the entire Hessian
7

---

x10 +10"+4.047 +4.047
x10
-7.708
CoS W)
x10
+6.88
10
-10
11
2.5
-15
10
10"
10'
10
10
10
10
approximation error
loss - loSS
X10 -9
x10
x10
1
Amax
0.01
0.01
0.01
10
10
10"
10
1/c
1/c
1/c
Figure 5: Convergence in to gated linear units. Moving along a channel to infinity with the jump procedure described in Appendix C Shows that c, a, and the cosine similarity cos A w) converge to constant values, and that the loss and the approximation error decrease with e² and the sharpness diverges with 1/e², predicted by the theory. For this example, network with 8 input dimensions and 8 hidden softplus neurons (81 parameters) trained on the rosenbrock target function was used.
0n). This scaling also characterizes the asymptotic flatness along the channel: combining dC,, ede with d(a, az) ~ - -2- leads to dd(//((;; Qz) ~ -c², i.e., the slope decreases asymptotically with e along the channel.
To gain further insights, we turned to a toy example, where a network with two hidden neurons without biases (activation function o (a) = erf(2//v2) is used to fit the one-dimensional target function fx) = erf (5xx+2.5)
-erf ((5x 2.5)/ 2 (see Figure 61). This network has only 4 parameters: a1 02g W1, W2 or, equivalently, C, a, w, e (A the wiggle of the teacher function at x = O cannot be well approximated by a single erf function, but the combination of erf and its derivative erf' leads to an accurate approximation (Figure 6a). To avoid finite data effects, we base the analysis on the population (or infinite- -ata) loss
one input dimension). Importantly,
00 E(a1, 02 W1, W2) = 1- [f(x) - a - (20(w)x))
2 p(x)dd,
(6)
where p(z) is the standard normal density function. We can express the loss function in terms of normal integrals, find the optimal (w, c) and a" (w, c) analytically for any value of w and c, and expand the loss f(w,e) = É(c' (w, e.,,,),, aw,, <), w, around (h(ww)+OL (see Appendixx Cfor h(w) and the derivation). We find that the loss E(w, 0) has three critical points for positive (the loss is symmetric f(w, 0) = 2(- 0)): two correspond to local minima, and one
to find f(w, e) E(w, 0)
saddle point of the full loss E(c, a, w. c) (see (see Figure 6c). At the local minima wf and w2 we find h(wi)= > 0 and h(w)) > O. which shows that the loss function has indeed stable fixed points at e = O, or, equivalently, at diverging di and ag We emphasize that this example the optimal solution is at a local minimum at infinity (at the end of channel 2 in Figure 6p); we note that this is consistent with Proposition 1 in (39. Note that a small weight regularization would prevent finding the optimal solution due to the flatness of the channel.
In this toy example, we can also see that stochastic gradient descent (SGD) or ADAM 46 easily finds the "entrance" of the channels to infinity (Figure 6p). but quickly gets stuck there. This occurs because the trajectory reaches the 'edge of stability [47]: along the channel, the sharpness (maximum eigenvalue of the Hessian) increases until it reaches the maximal value Amax 2/n for given step size 17 (Figure 61). Since the Hessian contains terms that scale with a:a; 1/e², its largest eigenvalue diverges with Amax 1/e² along the channel, leading to emin vn.. Curiously, the increasing sharpness coexists (in orthogonal dimensions) with the increasing flatness of the channel (compare Figure 3/ and Figure B12).
8

---

a
0.06
arget channel
saddle
channel
charnel
chennel
0.04
loss
-1
0.02
-2
-2
1
2
-2
-1
0
2
X
X
W
b
d
2/n
0-
q=1.8e-01
n-1 De-01
n=1.3e-01
5e-02
3
10
2x10
2
Infinite data
Amax
8
1
finite dats
100
SGD
0
-1
4x10
10
0 10
20
30
40
50
10"
10
10
iteration
Figure 6: Minima at infinity and edge of stability. (a) The solution at infinity (channel 2) can be the global minimum, as this example, where a one-dimensional target function (blue curve) is fitted with neural network with two erf neurons without biases (4 parameters).
(b) The loss landscape has three stable solutions: (i) the plateau saddle for y E (0, 1) (white line on the left converging to infinity for negative a, and (iii) low-loss channel 2 to infinity for positive a. Gradient flow with finite data follows closely gradient flow with infinite data; SGD and ADAM get stuck early in the channels. (c) In the limit
0, 0.65), (ii) channel
0 we can express the loss l(w, 0) as a function of w (see Section 4), which shows clearly that the channel solutions are stable w. Additionally, in the green region, the function h(w) in the e-xxpansion of the loss positive, showing that the local minima are indeed reached in the infinite time limit. (d) Using GD with a finite step size n7, the optimization cannot go beyond the 'edge of stability' where the maximum eigenvalue of the Hessian equals Amax = 2/n (dashed lines; color indicates learning rate 7)).

---

Discussion
Neural network loss landscapes are non-convex, but the empirical success in training shows that their structure is benign also outside the deeply wer-parameterized regime. We uncovered generic mech- anism contributing to this benign nature of the landscape: extremely flat channels to infinity. While these channels lead to local minima, they endow the network with new computational capabilities in the form gated linear units, that arise as the limit of the central finite difference approximation of directional derivative. Thus, the channels can give rise to good local minima. Channels involving multiple neurons can lead to yet another set of capabilities based on higher-order derivatives (see Appendix B). but a full characterization of these multi-neure channels is an interesting topic of future research.
We developed our theory for the stylized setup of single-hidden-lyyee MLP, mean-squared errr loss, and smooth activation functions. However, the channels to infinity arise from a simple interaction of input and output weights of pairs of neurons. This mechanism neither depends on the type of loss nor the number of layers. The pair of neurons contributing to the channel can be located in an arbitrary fully connected layer of deep network; indeed, this is the case empirically (Figure 4D. We expect that channels to infinity arise also large-scale models, within MLP layers of transformer or convolutional architectures, and even between channels of convolutional layers, but quantification of the occurrence probability of channels to infinity in these large scale settings ongoing work. Furthermore, groups of similar neurons, corresponding to condition 2 of our channel identification (Appendix B). have already been observed in large convolutional networks [48]
Flatness of minima has been linked to good generalization 49 The channels to infinity being very flat in some directions, and progressively steeper others- provide an interesting perspective on this: commonly used optimizers get stuck at the edge of stability [47]. thus they appear to be flat local minima. Already at the beginning of the channel, the network implements (a finite difference approximation of) the gated linear unit, which can be beneficial for generalization. If many channels emerge during training, this could be an indication that the network architecture suboptimal for the given task; a stylized example would be a teacher with gated linear units and a student with softplus neurons. Adding regularization has a similar effect as a finite step size: from a certain point the regularization outweighs the decrease in loss such that the dynamics get stuck inside the channel. Nonetheless, the network implements an approximation the gated linear unit.
We envision our novel view on the loss landscape to be insightful for practical applications where expertise of the landscapes is crucial: model fusion [31] adversarial robustness 51, continual leamingg [52 53 and federated learning [54] 34].
The code is available http:////..aaaao/o/ooooaaaaaolar
10

---

Acknowledgements
This work was supported by the Swiss National Science Foundation grant CRSII5 198612, 200020- 207426 and 200021-236436. The authors would like to thank Antonio Orvieto for fruitful discussions on the edge stability.
References
[1] Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 13(3):317-377, 2000.
[2] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[3] Anna Choromanska, MIkael Henaff, Michael Mathieu, Gerard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 192-204. PMLR, 2015.
[4] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[5] Kenji Fukumizu, Shoichiro Yamaguchi, Yoh-ichi Mototake, and Mirai Tanaka. Semi-flat minima and saddle points by embedding neural networks to overparameterizzaion. Advances in Neural Information Processing Systems, 33:63868 13876, 2019.
[6] Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning. pages 9722-9732. PMLR, 2021.
[7] Henning Petzka and Cristian Sminchisescu. Non-attracting regions of local minima in deep and wide neural networks. Journal of Machine Learning Research, 22(143):1-34, 2021.
[8] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. International conference on machine learning, pages 1724-1732 PMLR, 2017.
[9] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The Implicit Bias of Gradient Descent on Separable Data. Journal of Machine Learning Research, 19770):1-57, 2018
[10] Yaoyu Zhang, Zhongwang Zhang, Tao Luo, and Zhiqin J Xu. Embedding principle of loss landscape of deep neural networks. Advances Neural Information Processing Systems, 34:14848-144 14859, 2021.
[11] Yaoyu Zhang, Yuqing Li, Zhongwang Zhang, Tao Luo, and Zhi-Qin John Xu. Embedding principle: a hierarchical structure of loss landscape of deep neural networks. arXiv preprint 1.Xi::2111.1557,, 2021.
[12] Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures. Advances in neural information processing systems, 32, 2019.
[13] Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural archi- tecture optimization: Escaping local optimum with signed neural splitting. arXiv preprint 10332 2020.

---

[14] Eiji Mizutani and Stuart Dreyfus, An analysis on negative curvature induced by singularity in multi- -lyle neural-network learning. Advances in Neural Information Processing Systems, 23, 2010.
[15] Timothy Poston, C-N Lee, Y Choie, and Yonghoon Kwon. Local minima and back propagation. In IJCNN-9I-Seatie International Joint Conference on Neural Networks, volume 2, pages 173-176. IEEE, 1991.
[16] Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv: / Ieo..60055.611 2016.
[17] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International conference on machine learning, pages 2603-2612. PMLR, 2017.
[18) Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding, and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5)995-108, 2020.
[19] Itay Safran and Ohad Shamir. Spurious local minima common in two-layer relu neural networks. In International Conference on Machine Learning, pages 4433-4441, PMLR, 2018.
[20] Yossi Arjevani and Michael Field. Analytic study of families of spurious minima in two-layer relu neural networks: a tale of symmetry ii, Advances in Neural Information Processing Systems, 34:15162-15174, 2021.
[21] Berfin Simsek, Amire Bendjeddou, and Daniel Hsu. Learning gaussian multi-index models with gradient flow: Time complexity and directional convergence. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[22] Itay M Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild par-tearraettrizzatinnn the optimization landscape shallow relu neural networks. In Conference on Learning Theory, pages 99344 PMLR, 2021.
[23] Flavio Martinelli, Berfin Simsek, Wulfram Gerstner, and Johanni Brea. Expand-and-clusterr Parameter recovery of neural networks. In Forty-first International Conference Machine Learning, 2024.
[24] Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, and Lenka Zde- borová. Who is afraid of big bad minima? analysis of gradient-flow in spiked matrix-tenss models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[25] Radford M. Neal. Bayesian Learning for Neural Networks. Springer New York, 1996.
[26] Christopher Williams. Computing with infinite networks. In M.C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9, MIT Press,
1996.
[27] Jaehoon Lee. Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks gaussian processes. International Conference on Learning Representations, 2018.
[28] Alexander G G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahra- mani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018
[29] Yaim Cooper. The loss landscape of overparameteriize neural networks. arXiv preprint urXiv:1804 10200, 2018
[30] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116 2022.
12

---

[31] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances neural information processing systems, 31, 2018.
[32] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[33] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
[34] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
[35] Zhanran Lin, Puheng Li, and Lei Wu. Exploring neural network landscapes: Star-shaped and geodesic connectivity. arXiv preprint arXiv:2404.06391, 2024.
[36] Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad, and Seong Joon Oh. Do deep neural network solutions form star domain? In The Thirteenth International Conference on Learning Representations, 2025.
[37] Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. Advances in Neural Information Processing Systems, 32, 2019.
[38] Cédric Josz and Xiaopeng Li. Certifying the absence of spurious local minima at infinity. SIAM Journal on Optimization, 33(3):14166 1439, 2023.
[39] Lek-Heng Lim, Mateusz Michałek, and Yang Qi. Best k-layer neural network approximations. Constructive Approximation, 55(1):583-604, 2022.
[40] Shiyu Liang, Ruoyu Sun, Jason D Lee, and Rayadurgam Srikant. Adding one neuron can eliminate all bad local minima. Advances in Neural Information Processing Systems, 31, 2018.
[41] Kenji Kawaguchi and Leslie Kaelbling. Elimination of all bad local minima in deep learning. In International Conference on Artificial Intelligence and Statistics, pages 853-863. PMLR, 2020.
[42] Shiyu Liang, Ruoyu Sun, and R Srikant, Revisiting landscape analysis in deep neural networks: Eliminating decreasing paths to infinity. SIAM Journal on Optimization, 3274)2277772877, 2022.
[43] J. von Neuman and E. Wigner. Uber merkwürdige diskrete Eigenwerte. Uber das Verhalten von Rigenwerten bei adiabatischen Prozessen. Physikalische Zeitschrift, 30:467 January 1929.
[44] Johanni Brea, Flavio Martinelli, Berfin Şimşek, and Wulfram Gerstner. MI.PGradientFlow: Going with the flow of multilayer perceptrons (and finding minima fast and accurately), January 2023.
[45] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 933 931. PMLR, 06-11 Aug 2017.
[46] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
[47] Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference Learning Representations, 2021.
[48] Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gra- dient noise attracts SGD dynamics towards simpler subnetworks. shirty- Conference on Neural Information Processing Systems, 2023.
13

---

[49] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awe aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[50] Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In International Conference on Learning Representations, 2020.
[51] Ren Wang, Yuxuan Li, and Sijia Liu. Exploring diversified adversarial robustness in neural networks via robust mode connectivity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2346-2352, 2023.
[52] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In Interna- tional Conference on Learning Representations, 2021.
[53] Haitao Wen, Haoyang Cheng. Heqian Qiu, Lanxiao Wang, Lili Pan, and Hongliang Li. Optimiz- ing mode connectivity for class incremental learning. In International Conference on Machine Learning, pages 36940-36957. PMLR, 2023.
[54] Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.086 06440, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We thoroughly tested intuition behind channels to infinity with thousands simulated networks, reported all relevant metrics and developed a simple yet solid theory interpret the results.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We speculate on the presence channels in scaled up testing but do not provide simulated evidence. Partly because this work is first theoretical stepping stone and partly because further evidence in scaled up experiments will require overcoming additional technical challenges (ODE solvers for large network sizes are computationally expensive). This work has ethical, privacy, fairness concerns nor uses any specific dataset.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: To the best of our knowledge, proofs are correct and discussed in detail in the appendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
Justification: Link to code for reproducibility is in the appendix, setups described in the appendix. No datasets or models need to be released for this work.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Same as above.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Same as above.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: some error bars are reported the quantification of channel frequencies over different datasets, to get idea of the estimate. The message of this paper is not focused on specific metries expressed in numbers, rather more the existence of a specific structure in the loss. Use of statistical tests is not needed.
15

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Discussed the appendix. This work was solely performed on CPUs.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httptttpsr.ooorrssccooo/ooo///lt
Answer: [Yes]
Justification: Given the guidelines, we do not see any ethical concern stemming from this paper.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Given the foundational nature of research, we do not any potential negative impact directly stemming from this.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Same above.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: All software used was open source. Other assets were not used.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Does not apply.
14. Crowdsourcing and research with human subjects
Question: crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Does not apply.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16

---

Justification: Does not apply.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Does not apply.
17

---

A Neuron duplication introduces lines of critical points
A.1 Adding bias to neurons drastically reduces the number of stable plateau saddles
In the main text we highlighted specific attractive regions, plateau saddles, that can be found via neuron duplication. These attractive regions were first hypothesized theoretically by Fukumizu & Amari [ for 1-hidden-layer, scalar-output MLPs. Figure 2 numerically explores the frequency with which plateau saddles are reached under the common Glorot normal initialization in the Fukumizu & Amari setup without biases.
The stability of saddle lines is dependent the positive (negative) definiteness of the (d x d)- dimensional matrix: Bj
(w,xz) कق(ق āf( (f(x))((()))))))))))))(())) (x))) (II) Theorem 3), where Wj,aj aj are the input and output weights of neuron to duplicate, C, the loss of the original network, (*) is the target function, f())) is the network output, (-)D denotes the expectation w.r.t. the distribution of the training data p(z) N EE= ō(x x0)), and w, R.. It is difficult to determine in general whether the B3 matrix is positive (negative) definite or not, and thus whether the saddle line is stable or unstable. Empirically, we notice that the simple addition of bias term to the neurons drastically reduces the number of plateau saddles in the loss landscape, as shown in Figure Al (in this case B; is (d + 1 x d + d)dmennsioon.).
0
The stability in multi-output and multi-layer settings has also been theoretically explored 12 13
In addition to By being positive (negative) definite, another condition must hold on a (di-1 x de+ 1)- dimensional matrix D) being zero (77: Theorem 9), where d1-1 and d1+1 are the number of neurons in the previous and next layer respectively. In particular, Petzka & Sminchisescu (7] argue that the likelihood of all eigenvalues of B3 being positive (negative) in conjunction with the likelihood of D being zero in practical settings is extremely low. This conclusion is further supported by our numerical observations in Figure where we find little to no exact duplicate neuron.
A.2 Simulation details: Figure 2
We trained networks of 5 different architectures, with E {1, 3,4,5, neurons in single hidden layer. To guarantee enough coverage of (presumably) all unique minima in the landscape, each architecture was simulated 50-27 2" times. Initializations were drawn from the Glorot normal distribution and we used the mean-square
error loss L(0) (IE) (w, + b,) + c- fx))) >. where 0 (w, a, b. c) and (Ix) sigmoid (x)) softplus(r) is an asymmetric activation function
introduced in 66 23. The target function f(xx) ()) is modified version of the 2D Rosenbrock function:
f((11, log10 [(a x1)² + b(22 - 2 +c)² +
f1 x22) zscorep f (21,)2)
where a = b, c d
d= = 0.1 and (scorep[((x)) =
(f()) p|)) The modified
f(a)=(((=))>
Rosenbrock function was chosen due to its complicated, non-symmetric profile, leading rich variety of solutions found by the networks.
Each simulation was performed on single AMD EPYC 9454 48-Core Processor CPU core, using
ODE solvers to solve the gradient flow equation: 0 Vo (C((()) + R(O)) with the Julia package MLPGradientFlow.] [44]. R())= = (110 maxnorm)* if ||0|| maxnorm, else R(O) = 0, is a regularizer active only when maxnorm threshold is reached. This allows us to verify convergence of our simulations and halt them when the norm exceeds too high values. The dataset consisted of N samples drawn once for all seeds, with input distributed [-v3,+v3] Training was performed full-batch meaning that the only source of randomness is the initialization seed. Both input and output data have mean zero and standard deviation one. Hyperparameters of the simulations are provided in Table I where: patience is the number of iterations to wait before stopping the ODE solver if no improvement in the loss is observed, reltol and abstol are the relative and absolute tolerances for the ODE solver, maxnorm is the maximum norm of the gradient flow trajectory; we consider trajectories that exceed this norm as infinite-norm solutions.
regular 2D grid with æ1,
Evidence for the precision of convergence of our simulations is shown in Figure A3 We report extremely low gradient norms and non-negative minimum cigenvalues for all simulated networks.
18

---

Without bias
With bias
1.0
1.0
Parameter norm
norm
0.8
infriite finite
Infinte finite
0.4
Loss
0.1
Frequency
90
Frequency
90
unique minima
% duplicate neurons
unique minima
% duplicate neurons
46
243
134
19
30%
61
30%
16
10
27
15%
9
15%
3
0%
0%
2
5
3
3
Network size
Network size
Network size
Network size
Figure Al: Plateau saddles mostly disappear with the addition bias term: side-by-side comparison between the experiment of Figure 2|d its version with neurons with bias. The amount of solutions sharing the same exact loss level across different network sizes drastically decreases when adding bias term: horizontally overlapping solutions of different colors are barely present on the right panel. This is evidence that most of the saddle lines in the landscape do not contain stable regions, quantification of duplicate solutions (bottom) shows little to presence of plateau saddles. In contrast, in both experiments, a substantial amount of solution trajectories are unbounded in parameter-norm (gray-scale insets).
The network used Figure 2 was selected from the simulations of Figure 2p.b. Specifically, it d = 2, r = 2 network with finite parameter norm (see inset of left plot of Figure AI). Perturbations were performed after splitting the neuron with the negative output weight (blue arrow color in Figure 12 at different y values: 0* - 0" Figure 2 experiments show training convergence and trajectories of gradient flow after initializing at a point 0 near the saddle line. Specifically: 0 0d + @Cguin where Cmin is the eigenvector of the Hessian of 0 with the minimum eigenvalue, and a = 10 The training procedure follows the same details as in isubsection A.2 with the exception of using slower ODE solver, Heun (55]. for the early phase of the simulation. We chose Heun to obtain more trajectory steps near the saddle line. An interesting eigenvalue repulsion phenomenon is detailed in figure Figure A4
19

---

a
Convergence after small perturbation
b
Original local minimum
Distance from fism saddle saddee line
trajectory final
10"
(),
10
E (0,
10"
-2
Projection on saddle line
C
0.7
Eigenvalues
0.2
m/2
Loss
Angle
0.0
0.3
-0.1-
-2
-2
-2
Figure A2: Additional details for Figure 2 (a) Example of neuron duplication for loss function
stable only within the plateau-saddle region, Y E (0, 1). (b) Gradient-flow trajectories following a small perturbation from the saddle line the direction of O'emin' identical to [Figure 2t. (c) Losses reached after perturbations from different y values (left; colors as in panel b), eigenvalues the Hessian as function of y (middle), and rotation angle each cigenvector measured from the starting point 7 1/2 (right).
shown in Figure 2p: small perturbations
Table Hyperparameters of simulations in Figure 2
maxtime
patience
reltol
abstol
maxnorm
ODE solver
104
1h
106
10-3
10-6
103
KenCarp58 (56)
r=
r 2
r 3
[ 4
r =
50
20
25
50
10
10
10
10
15
10
10
10
-55
10
10
10
10
10
10
10
15
10
10
Converged gradient norm
200
20
50
50
100
10
0
-10
10
10
10
10
10
10
10
10
10
10
-10
10
-10
10
0
10
10
Converged minimum eigenvalue
Figure A3: Evidence of convergence to local minima channels to infinity for simulations in
Figure 2) the Loe norm max; Vo, C(0) is used to quantify the gradient norm.
20

---

0.0002
0.0130
0.0000
-0.0002
0.0100
-0,0005 0.0 0.0005
0.95
0.97
0.2
0.0
-0.1
-2
0
1
3
Y
Figure A4: Eigenvalue swapping if E {0, 1} and eigenvalue repulsion if y É {0,1): eigenvalues are colored based on continuity their corresponding eigenvectors, as two eigenvalue approach each other and swap order, this is the only way keep track of their identity. A particular characteristic of the saddle line the extra degeneracy happening at = and 7 = On top of the obvious zero-eigenvalue corresponding to the eigenvector in the direction of the saddle line (green line), another d eigenvalues cross zero at = and y = 1. This d-dimensional space corresponds to the subspace spanned by the input weight vectors of the duplicate neuron that is being silenced (since a1 = ya, 02 = (1 11-7)a, either one the two neurons is silenced at y = 0 or = 1). For y & {0, 1}. eigenvalues do not cross but repel each other. This phenomenon is sign of non-degeneracy of the system for y & {0, 1}; akin to eigenvalue repulsion phenomena in other fields (3
21

---

B Seemingly flat regions in the loss landscape as channels to infinity
B.1 Simulation details: Figure 3
The perturbation procedure used in Figure is similar to the one of Figure 2p.d, but with a different network. The network used in this example was selected from the simulations of Figure 4 We plotted the loss landscape of the network along the saddle line (see next paragraph for a discussion on how to find saddle line from a given channel), and along the eigenvector emin (7) of the Hessian corresponding to the minimum eigenvalue. This eigenvector is computed at every value of along the saddle line. Note that usually all eigenvectors rotate as ? changes (Figure A2j), meaning that our loss surface is plotted along rotating reference frame. The orange and green perturbations are performed by adding mine where a = 10-5. What reported to be loss, in panel is the loss of the network when the trajectory reaches the maxnorm. A zoomed-out view of the landscape, including negative values of y and the interval y E [0, 1] is shown in Figure B5.
B.2 Finding the closest saddle line from a given channel: an open challenge
Given the parameterization of a channel, identifying the parameterization of a saddle line parallel to said channel]is not trivial problem. As we have seen, usually saddle lines are non-attractive regions of the landscape, hence they cannot be found via simple optimization. Moreover, despite channels and saddle lines being asymptotically parallel subspaces, they are not necessarily close; all parameters can change along a trajectory that originates near a saddle line and ends in a channel (see early trajectories in Fiiure 3 Figure B6). In order to find the parameterization of the saddle line from which a channel originated, we used the following heuristic: (i) From the converged solution in a channel, merge the pair of neurons with the closest cosine similarity in input weights. This is done by substituting the pair of neurons by a new neuron with the average input weight vector w = (w, + 1/2 and summed output weights a = (a. + a,- 1) (ii) Train the resulting merged network with the same training procedure as in (sussection A.2 for few iterations. (iii) Verify that the parameterization reached, when duplicated, generates a saddle line that has some escape trajectories converging to the channel solution started from. This procedure is not guaranteed to work, as in some cases step (ii) leads to other channel solutions, and/or step (iii) leads to different channels parallel to the one started from. In conclusion, while generating channel solutions is straightforward (simply duplicate a neuron to create a saddle line then perturb). finding the parameterization of the saddle line that lead to a specific channel solution is not trivial. The example of Figure 3ss a successful case of this procedure. The network used was one of the networks obtained in the simulations of Figure 4
B.3 Simulation details: Figure 4
Five types of datasets were used in Figure 4 a multidimensional, modified version of the rosenbrock function and 4 gaussian processes (GP) with different kernel sizes. The modified d-dimensional
rosenbrock function is defined as follows:
fx(x) logio
Σ aa-r x2 + b(x; of + c)2 + +d
(8)
(x) = zscorep (x)),,
where a 1. b 3. 1. d 0.1. The Gaussian process datasets are generated using the packkce AbstractGPs..D package, using the Matern32 kernl::
k: (x, x²) (1 +1++0000000 v33 d(x x') exp(ex/333/3) d(xr, xe') x²)),
with .(...) the Euclidean distance, and s E 10.1, 0.5, 10} scaling factor. Some examples of 2D GP datasets are shown Figiuurr B8 Given that channel solutions implement derivatives of the activation function, we were wondering whether their probability of convergence is related to the non-smoothness of the target function. Indeed this seems to be the case Figure 4p. All datasets were fitted with various architectures of MLPs with hidden layer and r E (2, 4, 8, 16) neurons, for
In our exploration we found there exist multiple saddle lines parallel to given channel. But not all of these. after small perturbations, lead the dynamics back to the original channel.
22

---

different imput dimensions d E 42,4, 16). In particular, every GP dataset was re-drawn for every network, but kept the same across different initializations the parameter vectors. Both the softplus and erf activation functions were used for these simulations. We also tested deeper network on the rosenbrock dataset, with and 3 hidden layers with 4 neurons each hidden layer and softplus activation function. For each configuration, we ran 50 fits with different random initializations of the parameter vectors. total, we trained 4420 networks. The training followed the same procedure as in subsection and Table II converged gradient norms can be seen in Figure B7
Throughout the analysis, we considered solutions as channels if they satisfy the following conditions:
1. parameter norm larger than 10 (see subsection A.2 for the definition of parameter norm),
2. a cosine distance between input weights cos(w; w /|| w/ hw::l///le (1l) of less than 10-
3. and the fraction parameter updates inside the channels subspace bigger than 90%.
See Figure B9for for a version of panel in Figure 4 computed for all datascts. The fractiin of update inside the channels subspace in Figure 4b was computed as paujjcccelll ||| for each network that we deemed to have reached channel based criteria 1 and 2. Here, A refers to the update of all parameters and 2 2 projected refers to the update of parameters inside the subspace spanned by the channel. This subspace spanned by the output weight dimensions of the neurons participating in the channel (the corresponding projection operator is ligehannel indices) and the input weight vectors of the neurons participating in the channel (the corresponding projection operator W WWWW W where We contains the channel input weight vectors). The analysis of Figure was performed only by detecting channels with two participating neurons, but we noticed that some channel solutions contained triplets or even higher numbers of weights that were close each other, leading to multi- dimensional channels. It also often happens that multiple pairs of weights are close to each other, leading to multiple -i-dimensional channels. To detect nulti-dimensional channels we fixed set threshold of 0.01 for the cosine distance between input weights and then created groups of neuron indices which each element satisfies the distance threshold with least one element in the group (Figure B1T.. We ran this procedure on all networks that exceeded the maximum parameter norm. The size of these groups reported in Figuur 4g for all datasets. Finally, to detect channels between layers of deep MLPs in Figure 4f, we used the criteria and 2 described at the beginning of this paragraph.
B.4
Relu activation
Due to the non-differentiability of the relu activation function at O, the associated landscape can contain cusps which gradient flow dynamics get stuck. However, these cusps disappear in the infinite data limit (see subsection C.3), indeed, in this case there are examples where channels exist relu networks (see Figure B13).
23

---

-2
5.7
x10
o
5.5 loss
γ
1
0.2
0.0
3
-0.2
α
Figure B5: Zoomed-out view of channel landscapes: loss landscape of network of Figure Many other structures appear in the interval E [0, 1] that are not explored in this paper. Note that the landscape is symmetric around the point 7 = 1/2.
Orajectory
100
10-1
0
10-2
1.5
Figure B6: Channels are parallel to saddle lines but not near them: L2 distance of trajectories of Figure to the saddle line. While the 2D picture of the landscape may give the illusion that channels close saddle lines, in reality there many other dimensions that move substantially during the escape from the saddle.
24

---

100
50
50
10
16
10
-1
10
10
15
10
10
10 -5
10
16
10
10
10"
20
60
[4 444)
16
10
p- p
10
10
10
10
10
10
15
10
Converged gradient norm
Figure B7: Evidence of convergence to local minima or channels to infinity for simulations in Figure the Lee-oom max; Ve, [(0) used to quantify the gradient norm.
S 1/10
S 1/2
=
S
Figure B8: Examples of 2D GP datasets: 2D GP datasets used in Figure 4 The kernel size is controlled by the scaling factor the higher the bumpier is the dataset.
25

---

GP (s= 0.1)
GP (s= 0.5)
GP (s 2.0)
10"
225
22.5
|ai + |ajl
10
a: aj
x a: 2
10
10
10 -2
10
0 10
10
10
10
0
10"
10"
10
10
GP (s= 10)
rosenbrock
10
>2.
|ail + |ayl
10'
10
10"
10
10
10
10
10"
10
10
10
ming d(w,, Wj)
Figure B9: Channel features in all datasets.
60%
GP (s = 0.1)
GP (s 0.5)
GP (s 2.0)
frequency
0%
0.00
0.25
1.00
0.00
0.25
1.00
0.25
GP (s= 10)
rosenbrock
early training
30%
late training
0%
0.00
0.25
0.50
0.75
1.00
0.50
0.75
1.00
Fraction of update inside channels subspace
Figure B10: Fraction of updates inside channels subspace (as Figure 4b) across all datasets.
26

---

GP (s 0.1)
GP (s= 0.5)
GP =.0)
aail + alll + laki
10
10
10
10
10
10
10
0
10
2
GP (s = 10)
rosenbrock
|ail + |ajl + |ax]
10"
8"
10'
10"
10"
10
10"
10"
minijk d(w,, W,. Wx)
Figure B11: 2-dimensional channel features in all datasets, stemming from triplets of similar input weight vectors.
0.2
>10
0.1
7.5
0.0
loss
-0.1
0.2
50
100
150
200
Figure B12: View of a sharpening direction of the channel: slice of the landscape for channel
2 of Figure 6 allon I and almazz where emaz is the eigenvector corresponding to the maximum eigenvalue along the channel Amaz(7) (7).
27

---

f(X1, X2)
0
-2
-2
2
2
0
0
-2 -2 X1
X2
Figure B13: Example function learnable by 2-2-1 relu MLP that converges in channel. The shown target function can be implemented by relu MLP with 2 input, 2 hidden and I output neurons where f(x) = co x) + (v x'o' (w x), with c = 0, w = [1, 0j. v = [0. 1]. =relu. In order to fit this function approximately with a 2-2-1 relu MLP we employed the infinite-das formulation of ffssubsettion C.3 with teacher weights wi = (1, w2 = (1,c) and aj = 1/e, a2 -1/e, with
10
28

---

C Channels to infinity converge to gated linear units
C.1 Jump procedure along channels to infinity
To move quickly along the channel, we iterate the following three steps multiple times:
Use the reparametrization below Equation t obtain et9), at)), w0(), A and (!) for given values of a aj w. w in step of the ODE solver.
(f)
2. Move approximately in the direction of the channel by lowering ((++1) = ((0) while
keeping c(++1) clt), (*+1)) a w w(t), A (t+1) A This point may not be at the "bottom of the channel", because the other parameters also move slightly when lowering c.
3. Compute the corresponding parameters a: ((++1),
a w (t+1) w (t+1) and continue the ODE solver from this point to move again closer to the "bottom of the channel".
((+1)
C.2 Expansion of the loss in
We start with the reparamcterization in main text Equation 3
aw, x) + ayo x) = (o((w + (A) . x) + a(w - cA) x))
2
+ 2x
(o((w + +A) x) - =(ww - +A) x))
(10)
with w = (w; + w,)/2, e = ||w0 wy 11/2, A
w w.-w, w.
= e(a; - -ay), and = as + aj.
ww:-w
Performing Taylor expansion in é leads to awo(w, x) awo(w, x) co x)) + al A z))o (w)
2 c A x=)a0" (w x)) + i x))0")"0 m)) + O(d). (11)
Importantly, all contributions of (e) cancel such that the leading corrections are O
This implies that the network output is
f(:; 0o. t) so(a;; 0o) a f2(2;; 0o) + O(E*)
(12)
where 0o are the parameters for + 0, fo(æ; 0o) lim.-o fax; 0, is the output in the limit,
and fz(æ; 0u) = lim,-o e [f; 0, E) - fu(æ; 0o) denotes the leading order correction. For
single-hidden-laye layer perceptron fz(2; 0u) i(A xx) o0" x) + (A =>"o"((w æ). Inserting
12) into the loss L) = \ Em (1) (o,c), ( and expanding in c shows
C, (0: D) = L. (0,: D) + 2 ((0: D) + ((d)
(13)
with h(Oo; D) = + EN ()) 0n) B7 (o (i); 0o), y [. We note that the expansion the loss
B direct consequence of the O error of the central finite difference approximation
C.3 Infinite data (population) loss
To avoid finite data effects, we consider the population loss
/ (), 0* (')))()
((0) lim N N
(14)
for a given data distribution p(æ) lim N/ 1== implemented by teacher network. To ease the notation, we denote expectations w.r.t. the data distribution p(æ:) by angular brackets, (f())) f(x (p))p(r))))))))))))))))) da); angular brackets with subscript D
8(x (() and target function s(x; 0*)
reserved for finite data sets.
29

---

For a mean-squared emror loss and a single- www--hhdddddde-e perceptron teacher and student networks, the population loss is
(())
Σ
ajj(ws x x
Σ a x+ x+b;,) b,)
= 2 j=1
1 2 [1(x;
aj(ouun a + b7)o(wn x + bx))
j=1 k=1
ay(o(wy x + b . a + bn))an
(15)
J=
where we made the bias by explicit, i.e., Rd for this subsection, and assumed matching activation function of teacher and student.
For Gaussian data distributions p(æ:) II Maa 0, 1) the preactivations 2j w Gaussian with zero mean and covariance (23 ( wj Wi. Thus, the expectations are a function of the biases and input weight overlaps,
(un + (
b&)) = y(b bR. w wj, W. We, We u.&)
(16)
For (2) erf(2/ (2) and o (z) max(0, 0, 2) the function y can be computed analytically (see C.3.1
and C.3.2) Inserting Equation intonto Equation 13 leads to
((0) = (f (f(=00))) 07);
ajaxo(bf bk, wz wz.. wz uk, we wkk
wy agang(b, bk. wy wy wk, wk wk)
(17)
We investigate the properties of the landscape using gradient flow, 0 Vol(()) where (()) is given by Equation 17
C.3.1 Gaussian expectations with error function
We want compute the expectation (66 with a(2) erf(2/v2). However, the expectation is
simpler compute for the cumulative standard normal distribution G(z) } + erf(z/v)]]: afterwards we use (2)) = 2G(z) - 1 to get the expectation for the eπror function. For the case without biases the expectation is known [26]; for the case with biases the result to the best of our knowledge new.
Without loss of generality write 2; $/ (VI+ -I-py) where w; w. =0,, W: w
o, and x, y are i.i.d. standard normal. Thus, we want to compute
(G(p + G1) + 22)) = 1 d') [ dy GyG'())G(u + 01(vI + pr + vI pg)
v2
xo[p2= 02(vI+ +px VI Py)
(18)
V2
where G'(z) e f=² is the standard normal distribution. We use formula 20,010.3 from
V25
Owen's table
57
- dr (z) Gl dr) BvN( VI+ vir
where
VI+8
2 2222prto0 20-22)
BvN k) 2T S dx 1 dye
is the bivariate standard normal CDF, to
get
(G(m) + G2)G + +22)) 1 dx G'(x)
11 + TIVT+P
112 + 12vI#p
6:0. (1=0)
BvN
V2
/2
(19)
(1-p)
+ (1-p))
(1 (1-p)
2
2
2
2
30

---

Next, we use that Se dz BvN(a + br, c+ dz: ((@(a + bx - y) dz - 2) 2,) where y, 2 are bivariate Gaussian with zero mean, unit variance, and correlation (yz) = p and (=)) is the Heaviside step function. The random variables
y- bz 2 v 11
z-de pbbd vI+d" are still bivariate
Gaussian with zero mean, unit variance, and correlation أ VI- 1++4 Thus, changing variables leads to the identity ((O(a+ v)) dx -2))) 2)),,)) (0(a- vI+ 12)0 v1 ()))) g.a.
Using ((cx) = ((x) yields
(vI+=
p + bd
dx (z) BaN(a + ba, c+ dr; p) BvN
(20)
VI+ 8 VI+ d
VI+ 62 D
Note that 20,010.3 from [57] is the special case of (20) for p 0 since BvN(a + bx, c dz;0) G(a + Gr) + dz). Applying (20) to (19) leads to
(G(n1 + 21 Guz2 + 22)) BvN
V
A1
/1+0z ip
12
/1+of//14
0102
(21)
We use the identity (3.2) from 177. BvN(u1, p2: = T(m, P2/ p1) + T(p2, (1/H2)
T(M,, M pivi-pp - Tu2, 14 1W2
to express BvN(μ) P2; p) terms of Owen's
uaVI-p"
T function T(h, a) -oe G'(a) (lor) I+x²
dx for an efficient numerical implementation of ByN(
based on Gauss-Legende quadrature.
For (2) = erf(z/ (2f(z//v2)) we use 2G(z) - erf(z/v2), leading to
g(,,, |2, 0102 BNN
P1
P2
P
0102
VI+ of
V1
02
V1
+
o7 V
2G
Jn1
-2G
P12
+1
(22)
VI+
V1+00 02
where we used 10,010.8 from 57 1 dr G' (z) + bxx) = G(
VT+
C.3.2 Gaussian expectations with ReLU function
We want to compute the expectation (16) with (2) = max(0, (0, z). An alternative result can be found in [58] this (simpler) expression is to the best of our knowledge new.
First, we use to the homogeneity (az) (z) to consider the case with unit variance; the variance simply multiplies the final expectation and modifies the mean as μ4 -> pz/o;, Following subsubsection C.3.1 we write 2;
-/(( (V1 + pr + v1 pg). Thus, we want compute
(0(11 + 21 ( + 22))
00
dzge e x0-22 [s] 2p 20
dži
(01 + 212 + 22). (23)
2T /1
First, we shift integration variables, 7/4 - 24, to simplify the domain of integration in Equation 23 and rearrange terms to obtain
(a(1 + (1)0 + 22)) =
2 2 0² 100 džje S 21 L dē2 211 21 11= (1=2--a))-a pla w1))? 22. (24)
The 2/ integral can be written as Jo dx G' (ote(ai-pa) + vT- )z for which we can use formula VI-p²
x
(101) from [57] Jod dr Ga a + br)x (G) + aG(a) - aO(b). Using that > and
G(a) 11 -G(oa) this leads
(a(11 + 21)0(12 + =2)) = 100 džze d f(1 21
v1-pt pa
2x E 2ा 2 (Nat (())))))))))
122
p(21 - -m1)
G P2 ppl
p
21)
(25)
2T
v1 p1p p²
v1
31

---

Rearranging terms we obtain
(0(m + 21) 22)) =VI -V1--G((aa) L dz G¹( - 11 112
21
21
V1 1²
V1 p2
+ mill I dzp G')ii( 42 pp1
-1-p2
+ (w2 + pH1) L dep G'(z))((
H2
p
2)
v
p2
V1-
p2
+ p f danG(
12
+
P
2))
21
(26)
V1- V1-p²
For the first integral we can again formula (101), So dr dr bo)x के (a) -Ci(-a) for b. > O, for the second we need (10,010.2), 1 de G)) for the third (10,011.1), So dx (a + ba)a = VI4
VI- Aeee = BvN(h, k::))
b G' VT )G( -
Gla + (c), and for the fourth (10,0In.2) with help from 59 get rid of typos, Io dr G')) + bx)x² GG
GG()) + T(c + 2) + T( VHt b + <ltt")) + /T
)0(-a) + + G' (( vI+b
G( ab-(1+6%)g + G¹( ((( + cG'(c) + be) for vI+b
vi+6
<> 0. This leads to
VI- p²
200 20-14)
(o(u) + 21) + 22))
2T
+ (z
11 -P1/2
+ 122 (H))
1/2 111
V1-
V1 p²
1 (((()) + 2 +
+
G(uz) - TT(1,
P
- T T
#2
(p2) (27)
p²
for μl < O. Using formula (3.1) from 57 for BvN (11, P2: p) the expression reduces to
p²
19 122001u7t0
(a(p1 + 21)0(12 + 22))
2(-1²)
+ (MMne + () M2:
2T
+ (1zg)
11 112
+ (21
H2 111
(28)
V1 p²
V1 p2
which works for arbitrary sign of μl-
C.3.3 Minimum at infinity
Here, we derive the stability condition for the minimum at the end of a channel. To this end, we consider the simplified setting where the input is scalar, neurons without biases, leaving four parameters: a1, 02 W1, Wy. We work in the reparameterizzttoo
R. and the network consists of two
from (qquation which for scalar input w = (wi + (22) E (w2 w2) /2, = c(a, - 02).
and = 01 + Together with Equation 11 we get [z; 0o. t) = fo(:; 0n) + 6 f2(;; 0o) + O with
fo(æ; 0o) = co (wex) + axo (wx),
(29)
fa( 0o) = 2 x000 ==>o(wx)) + a
c
(30)
6
and 0g (a, c, w). Note that for scalar input A and E IR instead of e 2 0. Expanding the loss in we obtain
15
(0n, c) 2 ([o((; 0p) f(x; 03)) + (0)00o0000000 - 0xx O(6*), (31)
reflecting the structure of Equation 13
To reduce the number of parameters from 0.- = (a, c, w) to w we assume a separation of time scales: the dynamics of the readout weights a, are instantancous in relation to the dynamics of the input
32

---

weights w, e. While the separation of time scales changes the details of the trajectory, it leaves the critical points invariant. In particular, this means that a stable minimum at 0 with fast readout weights implies a stable minimum at e - O without separation of time scales.
Under the assumption of separate time scales, the readout weights a, e minimize the loss (31 for any given set of input weights w,e e. Since (31) is quadratic in a, the minimization is straightforward to perform, Furthermore, it is sufficient consider the minimization of the (1) contribution because the OEE") correction to the loss due to an (E²)) correction to a, e vanishes by construction of the minimum. Thus, we get f(w, c) f(w, 0) + chh(w)) + O(e)) where
1
Eww, 0) (Ifo(z: 0o(w)) f(;; (x;
(32)
h(w)= = (f2(#;
fo(w)) (a; )1),,
(33)
with Oo(w) = (ao(w), co(w), w) and ao (w) and cw(w) determined by the linear system
ao(="o (wx)) co (oorw)rrr (wx))) (f(x;; 0- (=o'(wx))
(34)
ao aok (ooo(w))
co (o(((())) (J(,; 0* (o)))orx))
(35)
All expectations are w.r.t. a one dimensional standard normal distribution which can be solved efficiently using Gauss-Hermite quadrature such that f(w, 0) and h(w) can be determined numerically. Stability requires h(w)) > at a minimum of t(w, 0).
C.4 Simulation details: Figure
We picked arbitrarily one of the networks with r hidden neurons (softplus activation function)
with biases (81 parameters in total), trained the modified Rosenbrock function in 8D (see Equa-
tion 8). which had a small minimal cosine distance d( d(w,, wy and large la.] + layl (see top left in
Figure 4fA). Starting with these parameter values, we applicd the jump procedure described Jump procedure along channels to infinity to obtain the results in Figure 5AA
C,5 Simulation details: Figure 6
For the d-dimensional toy example in Figure 5B we rely on Equation 17 with the analytical expression of (rom(tuuation 22] and derivatives thereof, to solve the gradient dynamics and compute the loss in the population average (infinity data) setting. The plot of the loss function (middle Figure 3B) is produced by fixing y and a (the projection onto the eigenvector with the smallest eigenvalue on the saddleline y 2), and minimizing the loss in the orthogonal, remaining 2 dimensions. The trajectories are orthogonally projected onto the y a-plane. The curves for finite data (gray) were produced with 4096 standard normally distributed input samples that were fixed throughout the simulation. The same dataset was used to run SGD (learning rate 7 = 0.1) and ADAM (default parameters 7/ = 0.000, 8 0.9, 82 0.999, e 10-8) with batchsize 16. We ran SGD and ADAM for fixed duration, which resulted in approximately 50'000 training epochs. To determine the stable region (green in |Figure 5B right). we used Herumie quadrature compute the integrals in Equation 33 for different values of w.
C.6 Second-order derivatives
Obtaining higher order derivatives from multiple neurons requires hierarchy of divergent readout weights and convergent input weights: a second order derivative leads to an (c²) contribution the finite difference which requires readout direction which diverges c to get an O(1) contribution to the output in the limit. To capture this hierarchical structure, we change notation in this subsection compared to the main text.
We consider a single-hidden- layer perceptron with r hidden units
f(x; 0) = a oo ( x))
(36)
with a E R",, a E Rd+1, W E (d+1), and o is applied plement-wise. As before the neurons in this network can be part of a bigger network with multiple layers.
The divergent readout weights need a set of orthogonal axes {up)r= along which they can diverge with different speeds without influencing each other. We perform change to the orthonormal basis
33

---

given by (up)p=o In this new bases the hierarchy of divergent readout weights and convergent input weights
eup = W up: where ||رp|| (1) and a = O1). Inserting Equation 37
aap Upp a,
(37)
into Equation 36 leads to
r-1
r-1
flx;00, Σ
Up
C
el (w x) x)uU)
(38)
cP
p=u
where Po (p)::W))=i)) Eventually the p-th term in the series will give rise the p-th derivative in the limit e + O.
C.6.1 First derivative with two neurons
For the familiar result of first derivative with neuron network change basis
V2 (1),
V2 (-1)
1
UA =
(39)
This leads with Equation 38
Ou, = Σ
1
Upulp oowwo x) c(w2 x)u1
cp
p=
Σ p=0
Uip Uiplo (usowo x) + eui1 (w) (o)oowowwww x) + OE+))
60
where we wrote the inner product explicitly in the second line and expanded e. Since #4a 1/v2 does not depend on the inner product in the second term reduces to the inner product up -u 6p1i in combination with Σl= Uip v2800 we get
fæ: Ou <) = V2ayo (wo x/ x/V2) + a ai(w1: x)o' (wo x/V2) + O
(40)
Note that the error is indeed (E because El= u = Connecting this result with the notation in the main text, we see /200, 0n w wo/ /2, and A w1.
C.6.2 Second Derivative with Three Neurons
For the second derivative with a three neuron network we change basis
i
2 = 22
U2 = V6
-2
(41)
v3
Now we have an OEe²) contribution in the preactivation, (wo æ)wo + e(wi x)²1 + e: x) which leads
2
((x)
Ckp
Uip a (wsown x) + eu;i (wi : x)
0=
1=0
uu x) (woww x) + 5 2 (wi (ugowo x)) + O
Using again ortonormality up un 5p1 in combination with uwo 1/v3. Ei=o Uip v30 and E u22n? 1/6 we get the limit e +
f(x) = 33 x/v3) + (ai(w] x) + anlws x))oo(wo x\V3
1
+ 12 a2lwi o))o"((s 2 V3).
(42)
Note that it is necessary to have the We x contribution to (62)), otherwise the (z(w2 a)) term would diverge with 1/c.
34

---

Appendix References
[55] Endre Süli and David F Mayers. An introduction to numerical analysis. Cambridge university press, 2003.
[56] Christopher A Kennedy and Mark H Carpenter. Higher-order additive runge-kutta schemes for ordinary differential equations. Applied numerical mathematics, 136:183-205, 2019.
[57] Donald B. Owen. A table of normal integrals. Commun. Stat. Simul. Comput., 94):3889 1980.
[58] Kirsten Fischer, Alexandre René, Christian Keup, Moritz Layer, David Dahmen, and Moritz Helias, Decomposing neural networks as mappings of correlation functions. Phys. Rev. Res., 4:043143, Nov 2022.
[59] Donald B. Owen. A special case a bivariate non-central t-distribution. Biometrika, 52(3/4):437 446, 1965.
35


Reviews:
[{"review_id": "dIcAES8cmF", "reviewer": "Reviewer_jaHr", "summary": "This paper explores the loss landscape around $\\gamma$-lines.   A $\\gamma$-line is created when a small network is trained to a local minimum and then one of the nodes is cloned, creating a line in parameter space of networks computing the same function.  The paper's experiments show that $\\gamma$-lines arise naturally in the training process.  Extensive experiments on small networks shed insight into the loss landscape around these $\\gamma$-lines and proved evidence that they are accompanied by parallel nearly-flat channels that lead to local optima at infinity.  The paper also shows that, in a limit,  these channels mimic a kind of gated linear gate.", "questions": "Some minor questions.\nIn figure 2, it appears that different numbers of runs were used for the different network sizes.  Could this help explain the different numbers of unique minima discovered?\n\nFigure 4D shows that updates eventually become essentially parallel to the $\\gamma$-line, does the distance from the $\\gamma$ line remain small?\n\nWhy might $\\gamma$-lines lose plateau-saddles when biases are added?", "limitations": "The behavior explored is in the absence of regularization.  Any resonable regularization would seem to prevent to formation of \\gamma$-lines and channels.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The channel structure uncovered by the paper is interesting, and the experimental setup (although on toy networks) is reasonable. \nThe experiments convincingly demonstrate that $\\gamma$-lines can arise naturally, and that the associated channels are likely to have\nthe nearly-flat but converge at infinity behavior.  It seems to be a valuable contribution to the study of loss landscapes in un-regularized networks.  The significance of the connection to gated linear units is unclear.\n\nAlthough the paper is generally well written, Figure 3a was confusing to me.", "was_revised": true, "final_justification": "My initial review mostly stands: this appears to be a solid contribution to the understanding of neural net loss landscapes.  \nThe author's comments gave me a greater appreciation for the possible value of the connection to gated linear units, but \nI still feel that the other aspects of the paper are stronger, but now see how others more enthusiasm for the channel phenomenon's relationship to gated linear units.", "rebuttal": "Thank you for carefully reading our paper, the fair and positive evaluation, and your questions and suggestions. We are pleased to read that our work \"*seems to be a valuable contribution to the study of loss landscapes*\". We address your concerns in detail below. If our answers will increase your confidence in your judgment we would appreciate if you could raise your score to reflect this.\n\n**Significance of the connection to Gated Linear Units**: \n\nWhen we saw the channels for the first time, and realized that they occur in a significant number of cases, we naturally wondered about their functional role. The connection to Gated Linear Units answers this question: the channels implement a new function that is the derivative of the activation function multiplied by a linear projection of the input. When naively looking at the architecture of an MLP, fitting such functions does not seem possible. As a concrete example, a network with a *finite* number of softplus neurons can perfectly fit targets composed of Gated Linear Units. Fig. 6B (left panel) provides another stylized example.\n\n**Regularization**: \n\nIndeed, regularization would prevent the dynamics to reach the \"end\" of the channel. However, even if the dynamics \"get stuck\" in the channel, the network would still employ a finite-difference approximation of the Gated Linear Unit, which provides a clear understanding of the underlying computation (see lines 226-227).\nIn other words, the landscape would contain a channel whose \"end\" is at finite values due to regularization. \nWe will expand the discussion of regularization (as also suggested by Reviewer `YsiZ`) to make sure it is clear that the idealized behavior (convergence to GLUs) can only occur in the absence of regularization. Concretely, we will add this brief discussion to the end of the paragraph in lines 223-227: \"*Adding regularization has a similar effect: from a certain point the regularization outweighs the decrease in loss such that the dynamics get stuck inside the channel. Nonetheless the network implements an approximation of the gated linear unit.*\"\n\n**Number of runs vs. number of unique minima in Fig. 2**: \n\nWe ran $50 \\cdot 2^r$ simulations per network, where $r$ is the hidden layer size. We experimentally verified that conducting more simulations for the smaller networks does not lead to the discovery of new minima. Furthermore, we have empirically noticed that the number of unique minima follows roughly $2^r$, and we estimated that running 50 times that number of simulations would allow good coverage of the landscape. There may be minima whose basins of attraction are too small or unreachable from standard Glorot initializations; those would not be quantified in our figure.\n\n**Distance between channels and $\\gamma$-line**:\n\nThe distance between the channel and the $\\gamma$-line does not necessarily remain small because the other parameters can move significantly; this is quantified in supplemental Fig. B5.\n\n**Effect of biases on plateau-saddles**: \n\nThe stability of the $\\gamma$-line is dependent on two sub-matrices that can be derived from the Hessian of the loss (Fukumizu \\& Amari, 2000); in Appendix A.1 we briefly describe their structure. For a plateau-saddle, these matrices must be positive or negative semi-definite (eigenvalues must be all positive or all negative), or zero. In general it is very difficult to make predictions on the signs of the eigenvalues because they depend on the whole training dataset. For the case of multi-dimensional output weights, Petzka \\& Sminchisescu 2021 (section 5.5) simply note that constraints on all eigenvalues are too strong to be satisfied by any practical example with noticeable probability. We speculate that the same accumulation of constraints can explain our results for adding biases to the network. Our empirical evidence, in combination with Petzka \\& Sminchisescu 2021, shows that symmetry-induced $\\gamma$-lines are not a source of local minima in loss landscapes. Note that this only applies to plateau-saddles, not the channels to infinity.\n\n**Fig. 3a**: \n\nFor the final version we will combine Figs. 2 and 3 to improve the flow of the manuscript (see also the comment by Reviewer `YsiZ`), which includes dropping 3a, because the main intuition that the plateau-saddle looses stability is also conveyed in 3c.\n\nThank you very much again for your constructive feedback!"}, {"review_id": "M9megkITO5", "reviewer": "Reviewer_Po89", "summary": "This paper discusses a particular type of structure that the authors have discovered in the loss landscape of neural networks with dense layers, that the authors call channels to infinity. These structure indeed appear as narrow valleys (\"channels\") with a very slow downward slope along a single line that points to a minimum at infinity. The authors offer an explanation of how these channels may arise in the context of an overparameterized dense layer and examine their property in the infinite limit (they show that two perceptron-like units behave effictvely as a perceptron unit plus a gated linear unit). They perform numerical experiments to look for such structures in networks trained on relatively simple tasks, finding them to be rather common.", "questions": "* Although the treatment is general with respect to the activation function, the ones that is used by the authors is very unusual, softplus(x)+sigmoid(4x). Thus, in the empirical estimation of the relevance of the phenomenon, it is unclear to me how much the results would change by using a more common activation function. Indeed, the authors themselves acknowledge that (line 220): \"It is also an open question, to what extent channels to infinity arise in MLPs with the popular, non-smooth ReLU activation function.\" I guess that the authors may have chosen not to use ReLUs in order to get rid of extra symmetries that affect those networks, as these would create extra flat directions (one per hidden unit). On top of this, because of the symmetry, the norm of each unit is no longer well-defined in a sense, since the norm of each unit can be changed arbitrarily by using a transformation of the kind $ReLU(\\alpha x) = \\alpha ReLU(x)$ for any $\\alpha > 0$. In such scenario, defining what would qualify as a \"channel to infinity\" as described in this paper is not even clear, I suppose.\n* Besides the activation function, the size of the network and the complexity of the training task may also play a significant role.\n* Maybe I missed it, but is it trivial that a channel to infinity is always associated to a $\\gamma$ line? From the discussion in the appendix it seems like identifying such lines is problematic.\n* The authors claim that SGD and ADAM tend to get to a channel to infinity and get stuck there (line 201). Wouldn't momentum be able to fix this entirely? (Also possibly some modification of ADAM that worked in a rotated parameter space, so as to adapt the gradient step and increase it in the flat direction, but that's way more speculative)\n* Figure 5c shows that (purported) channel and non-channel minima have similar training loss. But what about the test loss, or test accuracy? Any difference in that regard? How does the test accuracy change along the line, is it also affected only slightly?\n* The procedure for identifying the channels to infinity in fig. 5 is based on thresholding the distance between pairs of hidden units and their norms (5a). This seems to be a necessary but not sufficient condition. Then, there is an estimate of how many of these are associated to lines that go parallel to some $\\gamma$ line (figures 5d and B9 and B11). The latter seems to identify at most 60% (depending on the dataset and other details) of networks for which this is the case. So if I understand correctly a more conservative estimate of the number of these channel to infinity minima should be scaled by 60% or less. Or am I missing something?\n* The suggestion that channels to infinity may explain the observed star-shaped connectivity in loss landscapes (line 228) is unconvincing to me in light of the results found in this paper https://arxiv.org/abs/2202.03038 where a star-like (or \"octopus-shaped\", as the authors put it) structure was observed in a network with removed symmetries where all neurons were normalized; and also in this paper https://arxiv.org/pdf/2305.10623 where a star-shape structure was shown theoretically to exist in a simple non-convex model, in that case with a single hidden unit.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n* Clearly written, reproducible, appendices complete, overall very well presented\n* Interesting addition to the body of work on the geometry of the loss landscape. At the very least, these finding suggest that the kind of structures revealed here should be taken into account in the discussions and analyses of the loss landscape. Possibly, more of these kinds of structures may be uncovered in the future.\n\nWeaknesses:\n* Despite my last comment above, to me the significance of this contribution remains unclear, mostly because:\n  - I'm not sure that the evidence is sufficient to actually claim this to be a relevant phenomenon in practical scenarios (more on this in the \"questions\" box)\n  - It's also unclear if and how these structures affect the learning process and the generalization properties of the network\n  - The fact that the addition of biases drastically reduces the effect (as shown in Appendix A) suggests that, indeed, the phenomenon may \"go away\" in more realistic scenarios\n  - I also have some significant doubts about the proposal (made in the discussion) that these kind of structures may explain the star-shaped geometry of the space of solutions observed empirically in other works", "was_revised": true, "final_justification": "After the rebuttal all my concerns were adequately addressed. Having seen the other reviews, there seems to be a general consensus. Therefore, my initial doubts were solved (I'm increasing my significance score from 2 to 3) and I'm now recommending acceptance (I'm increasing the rating from 3 to 5).", "rebuttal": "We thank you very much for your detailed reading of our manuscript, your insightful questions and suggestions, and the balanced evaluation. We truly appreciate your effort put into reviewing our work and we are pleased to read that you found it \"*interesting*\" and \"*very well presented*\". We address your questions and concerns in detail below. In particular, we would like to clarify two important points regarding the activation function and the effect of biases that might have biased your rating.\n\n**Effect of adding biases**: \n\nAdding biases does *not* have a negative impact on the number of channels, all experiments in Fig. 5b are with biases. In contrast, adding biases strongly affects the presence of plateau-saddles (local minima on $\\gamma$-lines) -- they indeed seem to \"go away\" in more realistic scenarios, as expected from inspection of the structure of the Hessian matrix (see lines 113-114 and supplemental Fig. A1 and our response to reviewer `jaHr` on this point). \nThus, simply adding biases does not suggest that the channels go away in more realistic scenarios, in contrast, plateau-saddles disappear.\n\n**Unusual activation function**: \n\nThis comment seems to be based on a misunderstanding. We used a range of different activation functions throughout the manuscript: Figs. 2, 3 are indeed based on $\\mathrm{softplus}(x)+\\mathrm{sigmoid}(4x)$ but Figs. 4, 5 are based on $\\mathrm{softplus}(x)$ (see appendix B3; a preliminary analysis for $\\mathrm{erf}$ is consistent with the $\\mathrm{softplus}$ results), and Fig. 6 is based on $\\mathrm{erf}(x)$ (see line 183). Thus, our results show that channels to infinity occur with different activation functions. \n\n**Significance for practical scenarios**: \n\nWe expect channels to infinity to be present in the loss landscape of large neural networks. Indeed, preliminary experiments applying the jump procedure (Appendix C.1) to a VGG pretrained on CIFAR10 leads into a channel. However, the tricky question is under which circumstances the training dynamics lead into a channel --- addressing this requires significant additional work. Furthermore, the channels may not be trivial to identify, because standard training with SGD and regularization may lead only to the very beginning of it.\n\n**Generalization**:\n\nTrain and test loss are almost identical in our setups because the data is relatively low-dimensional and densely sampled. In particular, the distribution of the test loss for channels and finite minima is similarly overlapping as the distribution of the training loss.\n\n**Momentum**: \n\nThe reason why SGD and ADAM get stuck is their finite step size in combination with the increasing sharpness (maximum eigenvalue of the Hessian) of the channels. As shown in the literature on the \"edge of stability\" [Cohen, et al.; Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability], the maximum sharpness is inversely related to the step size $\\eta$. Note that this effect occurs with and without momentum (see their Eq. 1). Extending our scaling analysis (lines 180-181; Fig. 6 top row) we can show that the sharpness diverges with $1/\\epsilon^2$ in the channels. \nFurthermore, SGD training in the setup of Figure 6 converges exactly at a point in the channel corresponding to the edge of stability ($\\lambda_\\mathrm{max} = 2/\\eta$).\nRegarding your proposal of an optimizer operating in \"a rotated parameter space\", we believe this is very similar in spirit to the jump procedure that we used to proceed when the ODE solver got stuck (lines 169-172; appendix C.1).\n\n**ReLU activation**: \n\nWe avoided the ReLU activation due to its non-smoothness (the homogeneity can be accounted for, see next point). ReLU loss landscapes present non-differentiable cusps at points in parameter space defining the boundary between a ReLU neuron being activated or dead for a specific datapoint. \nWe noticed, especially nearby saddles where trajectories can get very close to each other, that ReLU network trajectories would get stuck at cusp-like points near the $\\gamma$-line. \nHowever, given that softplus is a smooth approximation of ReLU, we speculate that ReLU landscapes can still present channel structures. \n\n**Identifying channels**: \n\nThis is an excellent point and we thank you very much for raising it. Indeed the threshold on weight similarity provides only a necessary condition. \nThus, we will scale the numbers reported in Fig. 5b down by including only trajectories that have > 90\\% fraction of the updates parallel to the saddle line. This reduces the number of confidently identified channels to 80-90\\% the reported number (depending on the dataset), which is still a significant fraction of runs. \nOn top of this, we improved the channel identification criteria in two important ways: \n1) We now account for the approximate homogeneity of softplus (in the limit of large inputs ReLU and softplus coincide) which allows to rescale input- vs. output-weights to correctly identify putative channels with high norm input weights. \nSpecifically, channels of high input weight norm softplus neurons can have weights grow both parallel to the saddle line *and* parallel to the direction of the input weight vector (becoming effectively a two dimensional space). \n2) The originally selected threshold for the rosenbrock data was including solutions that were not channels (see dashed line in Fig B8). We selected better thresholds for each dataset and saw a drastic reduction of runs that had updates not parallel to the saddle line (or the extra manifold described above). The down-scaling by 80-90\\% already includes these two corrections.\n\n**Association between channels and $\\gamma$-lines**: \n\nQuite often, a small perturbation from a $\\gamma$-line leads to a channel. However, it is an open question, whether the attractive region of all channels contains points that are arbitrarily close to a $\\gamma$-line. In some preliminary work, we tried with mixed success to find $\\gamma$-lines nearby the region of attraction of given channels. A direct association between $\\gamma$-lines and channels is that they are asymptotically parallel. To see this we can rewrite the reparameterization introduced in section 4 as $\\mathbf{\\theta} = (\\mathbf{w_1}, \\ldots, \\mathbf{w}, \\mathbf{w}, a_1, \\ldots, c/2, c/2) + (\\mathbf{0}, \\ldots, \\epsilon\\mathbf{\\Delta}, -\\epsilon\\mathbf{\\Delta}, 0, \\ldots, a/2\\epsilon, -a/2\\epsilon)$ where the update given by the second term is asymptotically equal to $\\frac{a}{2\\epsilon}(\\mathbf{0}, \\ldots, 0, 1, -1)$, which is parallel to a $\\gamma$-line.\n\n**Star-shaped connectivity**: \n\nWe would like emphasize that the connection to star-shaped connectivity was clearly indicated as a speculation (line 228). That being said, we agree that the connection is a bit premature since channels could only explain aspects of star-shaped connectivity---clearly, it could not explain the structure observed by Annesi et al. for a spherical perceptron (but note that they consider geodesics on the sphere, not in euclidean space)---and, re-reading our formulation from this point of view, we agree that the link to star-shaped connectivity is premature. We will remove this speculation from the discussion.\n\n**Conclusion**\n\nWe thank you again for the careful reading and the insightful questions and comments. We hope our rebuttal convinced you that the channels can be of practical significance; in particular we hope we were able to clarify that the channels occur for commonly used activation functions and that they also occur with biases. If this is the case please consider raising your overall score to reflect this."}, {"review_id": "bQy1OTfSud", "reviewer": "Reviewer_YsiZ", "summary": "This paper identifies a special structure in the loss landscape of neural networks: channels along which the loss decreases extremely slowly, accompanied by the divergence of output weights of at least two neurons to infinity. The work begins with an empirical investigation of the symmetry-induced saddle lines created by neuron splitting techniques. It then describes the “channel” structure observed in these networks, showing that along these channels the loss decreases slowly. The authors interpret this structure as effectively implementing a gated linear unit. Finally, they discuss the implications of this phenomenon and demonstrate how common optimizers such as SGD and Adam converge to these structures in practice.", "questions": "# Major Questions\n\n- What would happen if you add weight decay to the training? Naturally, the minima of the regularized loss could not lie at infinity anymore, as at some point the weight decay would outweigh the benefit of decreased loss from following the channels to infinity. I think it’s worth expanding line 198 into a short discussion on this.\n\n- You mentioned that along the channels there is increasing sharpness. Does it saturate at some value? Have you considered the connections to the edge of stability literature? How does the learning rate interact with how far “down” the channels the network goes?\n\n# Minor Comments\n\n- While reading Section 2, I found myself jumping between Figures 2 and 3. If possible, could you consider a more natural organization of the subfigures or a way to place these figures closer together and where they are referenced in the main text? Both figures have a lot of content; you might even consider merging them and moving some subfigures to the appendix.\n\n- In Figure 3, consider using a different colormap in (b) for γ than what is used for the loss landscape in (a).", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n- I think this is a very creative and original study of the loss landscape of neural networks that could prove to be very insightful for understanding trained neural networks, providing insights for both future theoretical and empirical studies.\n\n- Very strong figures that help visualize the central ideas of the work, which would otherwise be harder to follow.\n\n# Weaknesses\n\n- It seems that all experiments were done in very small model settings using synthetic data and often with ODE solvers rather than gradient descent. While I understand the motivation for simplified settings, I would suggest adding at least a couple of experiments using natural datasets or investigating the weight structures of pre-trained networks to strengthen this work.\n\n- While in some sense the detailed figures are a strength, many have too many subfigures, leading to overly long captions that break up the narrative flow and make it more difficult to follow the main points of the work. I would suggest deferring some of these subfigures to the appendix.\n\n- The greatest weakness of this work is that it is unclear what the practical significance of these channels is. While you suggest that these channels might be beneficial for generalization or could explain recent discoveries of star-shaped connectivity in loss landscapes, these arguments are purely speculative. I believe stronger evidence of these channels in more realistic networks would improve the impact of this work.", "was_revised": true, "final_justification": "I think this is a very creative work with a lot of potential, but \"the greatest weakness of this work is that it is unclear what the practical significance of these channels is\". The authors have discussed this partially in their rebuttal, but I think they could spend more time in the discussion making this argument clear. Overall, I still recommend acceptance for this work.", "rebuttal": "Thank you very much for your careful reading of the manuscript, your insightful detailed, and helpful suggestions. We are pleased to read that you found our work to be \"*very creative and original*\" and that it \"*could prove very insightful*\". We address your concerns and suggestions in detail below. \n\n**Investigating pre-trained networks**: \n\nThis is an interesting point, but it is difficult to address it adequately. We expect channels to infinity to be present in the loss landscape of large neural networks. Indeed, preliminary experiments applying the jump procedure (Appendix C.1) to a VGG pretrained on CIFAR10 leads into a channel. However, the problem is to identify under which circumstances the training dynamics lead into a channel --- addressing this requires significant additional work. Furthermore, the channels may not be trivial to identify, because standard training with SGD and regularization may lead only to the very beginning of it.\n\n**Edge of stability**: \n\nThis is an excellent suggestion. Extending our scaling analysis (lines 180-181; Fig. 6 top row) we can show that the maximum eigenvalue of the Hessian $\\lambda_\\mathrm{max}$ diverges with $1/\\epsilon^2$ in a channel. Thus, the sharpness does not saturate and, following the \"edge of stability\", any optimizer with finite learning rate will eventually get stuck. Using the scaling of the sharpness, we can quantify where the optimizer gets stuck for a given learning rate $\\eta$: according to the edge of stability, the maximum eigenvalue scales inversely with the learning rate, $\\lambda_\\mathrm{max}\\sim1/\\eta$, leading to $\\epsilon_\\mathrm{min} \\sim \\sqrt{\\eta}$. We empirically verified this on the setup of Fig. 6: we find indeed that GD trajectories get stuck at $\\lambda_\\mathrm{max}=2/\\eta$. Thus, this setup presents a stylized example of the edge of stability phenomenon, where the geometrical properties of the landscape are defined by the channel.\n\n**Weight decay**: \n\nIndeed, weight decay would prevent the dynamics from continuing along the channel. Nonetheless, the computation would be a finite-difference approximation of the directional derivative that gives rise to the Gated Linear Unit, which provides a clear understanding of the underlying computation. We will add the following brief discussion to the paragraph in lines 223-227 to expand the brief statement from line 198: \"*Adding regularization has a similar effect: from a certain point the regularization outweighs the decrease in loss such that the dynamics get stuck inside the channel. Nonetheless the network implements an approximation of the gated linear unit.*\"\n\n**Figures 2 \\& 3**: \n\nThis is a very helpful suggestion. We will follow your advise and combine parts of Figs. 2 and 3 into a single figure. To this end, we will move the top right panels of Fig. 2 and panels a, b, e from Fig. 3 into the appendix. Fig. 3 c, d will be combined with the remainder of Fig. 2, thereby summarizing the main results on saddle-lines in a single figure.\n\n**Conclusion**\n\nThank you very much again for the work that you put into this review and the helpful comments. We are particularly grateful for your suggestion of the link to the \"edge of stability\", which we believe provides an interesting link to the literature and improves the significance of our results. If you agree, we would appreciate if you would revise your scores to reflect this."}, {"review_id": "Opih1lUU2l", "reviewer": "Reviewer_eBLi", "summary": "Training neural networks occurs on a surface called the *Loss Landscape*.\nThis paper zooms on regions related to minima of subnetworks and uncovers a novel structure called *Channel to infinity*.\nDuplicating a neuron in a subnetwork yields a symmetry with degree(s) of freedom on the output of the duplicated neurons.\nAt a subnetwork minima, this process generates a saddle line which in turn gives rise to the channels on both sides of unstable regions of the saddle line.\nChannels, having a slowly decreasing loss, push the parameter under gradient to infinity by making the output weights of two neurons infinite while equalizing their input weight vectors.\nMoreover, at infinity the two entangled neurons simplify to a GLU unit.\n\nAlongside a description of the phenonemon, the paper conducts various empirical analyses on the stability of the saddle line, the natural reachability of the channels, the different properties a pair of channels can exhibit, their flatness/sharpness and descriptive statistics touching on frequency or location in an architecture of entangled neurons.", "questions": "- You study plateau-saddles and networks that have neurons with equal input weight vectors. Do you think this relation could be generalized, e.g. 3 neurons which would perform computations doable by 2 (but where no 2 neurons could be replaced by 1) and do you expect existence of an analogous structure in the loss landscape ?\n- In the same vein, have you considered the case where there is no nonzero local minima for a 1-subnetwork (minus 1 neurons) but there is one for a 2-subnetwork ?\n- In \"Best k-Layer Neural Network Approximations\" by Lim et al, it is shown that for neural networks in general, the infimum might not be reachable and that minimizing the loss necessarily makes the parameter diverge. Could you comment on the relation with your work ?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Quality\n\nThe submission is high quality and seems technically sound (I have partly checked the maths) and the experiments are conducted with rigor: complete description, use of solvers when necessary and clear setup.\nThe paper is well narrated with theory guiding when it can a primarly experimental study.\nThe plots are a bit crowded but well executed and each subplot has added value.\n\n## Clarity\n\nThe manuscript is very clear, although dense, and the organization is good as well.\nThe reader can find all relevant details of experiments in appendix and the authors have careful described their results.\n\nOne thing is not totally clear: I expected the channels to be sharp orthogonally to the saddle direction, as it is argued a couple of times (line 151, 203, fig 5e) but on Figure 4c it seems that the flatness converges rapidly and stays constant along the sampled direction corresponding to $e_{min}$ which is explicitly selected to have maximum negative second order variation.\nIf the authors could clarify why we don't see sharpening after $10^2$ that would comfort my opinion of the paper.\n\n**Minors / suggestions**\n- Figure 4d caption: $e_-$ might be a typo,  $\\pm e_{min} $ would be clearer to denote a pertubation along the direction of the minimal eigenvalue.\n- line 164 is not clear: $a$ is not constant so why would the $a_i$ and $a_j$ diverge ? \n- the term \"attractive\" is mentionned a couple of times but not clearly defined. In my understanding it qualifies a minimum to which some parameter can converge but defining it in a few words and contrasting it with a stable minimum would clarify.\n- the function $h$ line 176 could be specified shortly like \"where $h$ is a function ...\"\n\n## Significance\n\nDescribing the loss landscape is a key for understading training dynamics and generalization.\nIn this regards, the paper is both impactful and interesting.\nThe phenomenon is actually observed on a simple setup.\nFurthermore, it provides a case study of a gap between gradient flow and gradient descent as the authors report that the discretized versions can become stuck.\nIt might also foster the discussion on flatness in parameter space, since these channels are at the same time flat and sharp, depending on the direction.\nSimilarly, it sheds light on mode connectivity since channels can be composed of low loss regions that end up in a global minimum at infinity.\n\nHowever, as hinted in the \"Related Work\" section, the applicability to the interpolating regime, which is common with overparametrized networks, is limited.\nIn my understanding, the phenomenon of *channel to infinity* happens in the vicinity of local minima (including global ones) of subnetworks, provided these minima do not reach 0 loss. However one can argue that with sufficient capacity, a subnetwork with one less neurons would only have global, $0$-loss minima and so channels to infinity would not be present.\nMoreover, works have shown that even in mild overparametrization it seems not easy to reach local minima (e.g. \"Spurious Local Minima are Common in Two-Layer ReLU Neural Networks\" by Safran and Shamir) so entering a channel to infity might also be quite unlikely.\nThe experiments are limited to simple cases (which is fine) but it is therefore also not clear how frequent entering in a channel happens in practice: whether it happens for different kinds of architecture, nor the precise interplay with discrete optimizers and their peculiarities (edge of stability, loss catapult).\n\nAll in all the paper provides insights **and** raises new questions: it is significant.\n\n## Originality\n\nLoss landscape have been tremendously studied, this works uncovers a novel structure therein.\nMethodologically, the use of ODE solvers and the jumping technique to overcome very slow convergence is original for the field.", "was_revised": true, "final_justification": "Loss landscapes are a central object of study in deep learning. This paper offers a novel, rigorous, and thought-provoking contribution to their structure and therefore I believe should be accepted.\n\nOn top of its main contribution, which is the description of so-called \"channels to infinity\" and their properties it has connections to sharpness/flatness (fig 5e), mode connectivity and derivative computation by neural network (convergence to GLU and appendix C.5), which are valuable.\nThe use of ODE solvers, the channel-jumping technique while not central is original.\nIt also highlights an important discrepancy between gradient flow and gradient descent, showing that discretized optimization can become trapped at the entrance of a channel.\n\nThe paper's applicability remains limited in the context of (overparameterized or) interpolating networks. Its broader impact on practical deep learning may be constrained by the somewhat detached nature of the setting: : the focus is on non-interpolating regimes, traversing a channel requires continuous optimization, no guidance or modification of the standard training procedure is proposed and it does not offer an explanation of previously observed behaviors in deep learning. As other reviewers noted, regularization tends to truncate channel structures, and adding bias terms diminishes the effect.\nDespite these limitations, which are acceptable in my view, I found the paper compelling, well-narrated and of general high quality as indicated in my original review.", "rebuttal": "Thank you very much for your thorough review of our work, your careful assessment, and the positive evaluation. It was humbling to read that \"*loss landscape have been tremendously studied, this works uncovers a novel structure therein.*\" We will address your questions, concerns, and suggestions in detail below.\n\n**Sharpening of channels:** \n\nIndeed, the channels are very sharp in some orthogonal directions, and these directions become progressively sharper along the channel. This is not visible in Fig. 4c because the Hessian is evaluated at the saddle-line for a negative eigenvalue, i.e., it shows the direction which \"leads away\" most strongly from the saddle-line. This direction simply does not correspond to the direction of maximum curvature inside the channel in the high-dimensional parameter space. To give a more formal argument, we extended our scaling analysis to the eigenvalues of the Hessian which shows that the maximum eigenvalue diverges as $\\lambda_\\mathrm{max} \\sim 1/\\epsilon^2$.\n\n**Discrete optimizers:** \n\nThis is a very helpful comment! We can directly combine the above scaling of the sharpness with the main result of the 'edge of stability': the maximum eigenvalue scales inversely with the learning rate $\\eta$, i.e., $\\lambda_\\mathrm{max} \\sim 1/\\eta$. This results in a simple relation between the learning rate and the maximum depth: $\\epsilon_\\mathrm{min} \\sim \\sqrt{\\eta}$. We empirically confirm in the setup of Figure 6 that gradient descent converges to the level in the channel in which $\\lambda_\\mathrm{max} = 2 / {\\eta}$, and that the maximum eigenvalue diverges with $1/\\epsilon^2$.\n\n**Vicinity of local minima:** \n\nWe would like to clarify that it is *not* necessary to find local minima of smaller sub-networks in order to find channels. We sincerely apologize because this misunderstanding clearly arose from a crucial omission on our side: The experiments in Fig. 5, including the quantification of channels in 5c, all start from random initial conditions (Glorot normal). Thus, channels are found in significant number starting from random initial conditions; it is not necessary to first identify minima of sub-networks. We will add this important point by appending \"*; importantly, we start in all cases from random initial conditions (Glorot normal initialization)*\" to line 135. We believe this also answers your question (ii), but please let us know in case we misinterpreted this question.\n\n**Interpolating regime:**\n\nWe agree that it seems unlikely to find a channel in the interpolating regime, certainly deep in the interpolating regime with large manifolds of zero-loss solutions. We will change the first sentence of the discussion (line 205) to: \"*Neural network loss landscapes are non-convex, but the empirical success in training shows that their structure is benign also outside the deeply over-parameterized regime.*\"\n\n**Generalization to multiple neurons:**\n\nThe generalization of similar structures to multiple neurons is an interesting question. Empirically, we find channels composed of multiple neurons in various settings (see Fig 5g), which effectively makes them multi-dimensional. From a theoretical perspective we show in appendix C.5.2 how three neurons could together implement a second derivative using two divergences with different speed ($1/\\epsilon$ and $1/\\epsilon^2$) along two particular directions in parameter space. Empirically, we verified that a few neurons could fit to zero loss a target that is a linear combination of the activation function and its first and second order derivative.\n\n**Relation to Lim, Michalek, and Qi (2022):**\n\nThank you very much for pointing out this interesting result, which we were unaware of. First, Lim, Michalek, and Qi explicitly construct an example of a minimum at infinity with multiple (six) data points for their Theorem 1, hence we will change line 67 to \"*but there are examples, where minima at infinity can arise with two or more datapoints [Josz23,Lim22]*\". However, their construction is different from a channel to infinity because only a single readout weight diverges, in contrast to the joint divergence of two readout weights that characterize a channel. Furthermore, Lim, Michalek, and Qi show for a single-hidden layer network that any sequence of weights converging to a global minimum at infinity is unbounded. This is consistent with the example in Fig. 6 where the upper channel (channel 2) leads to the global minimum, hence we will add \"*; we note that this is consistent with Proposition 1 in [Lim22]*\" to line 197. Importantly, channels do not only occur in combination with a global minimum at infinity: the lower channel (channel 1) in Fig. 6 provides an explicit example. Thus, channels are consistent with Proposition 1 of Lim, Michalek, and Qi but are a more generic phenomenon because they are not tied to a global minimum at infinity. Moreover, in contrast to Lim et al, our example of Fig. 6 shows that solutions at infinity can arise also with infinite data.\n\n**Minor suggestions:**\n\nThank you very much for bringing unclear notations and wordings to our attention! (i) We will fix the notation of mineigval direction following your suggestion. (ii) Throughout section 4 we consider the limit $\\epsilon \\to 0$ at fixed $a$ (convergence of $a$ is shown in Fig.\\ 6a, middle panel). This was indeed not clearly defined in the text and we will fix the definition of the limit to \"$\\epsilon\\to0$ with fixed $\\mathbf{w}$, $\\mathbf{\\Delta}$, $a$, $c$\". (iii) Precisely, with 'attractive' we mean a point where gradient flow can converge, e.g., on the plateau saddle. We will add \"*attracting (all eigenvalues of the Hessian are non-negative)*\" to line 103.  (iv) Yes, very good suggestion; we will add \"*, and $h(\\mathbf{\\theta}_0; \\mathcal{D})$ denotes the leading order correction term*\" to the end of the sentence in line 177.\n\n**Figures:** \n\nThe impression that the figures are too crowded was also echoed by the other reviewers, in particular Figs. 2 and 3 seemed to obstruct the flow of the manuscript. To address this we will combine them into one figure and move the top right panels in Fig. 2 as well as panels a, b, e in Fig. 3 to the supplement. Please do let us know if you have further suggestions to improve the readability!\n\n**Conclusion**:\n\nThank you very much again for meticulously checking and deeply engaging with our work. We sincerely appreciate your constructive criticism as well as your helpful questions and suggestions, and we hope our rebuttal was able to comfort your opinion of our work."}]

Author rebuttal:

Meta-review: This paper identifies a special structure in the loss landscape of neural networks: channels along which the loss decreases extremely slowly, accompanied by the divergence of output weights of at least two neurons to infinity.  The authors interpret this structure as effectively implementing a gated linear unit.  They discuss the implications of this phenomenon and demonstrate how common optimizers such as SGD and Adam converge to these structures in practice.

Reviewers praised the interesting nature of the main concept, the quality of presentation including the figures, and reproducibility.

The had reservations about the practical significance, the impact of biases, the connections to other phenomena, and the limited experimental setup.

The authors handled the rebuttal and the subsequent discussion well, and some of the reviewers increased their ratings.

Overall, this is a clear accept, with all four reviewers being in favor of it, and only one of them weakly.  Nevertheless, the authors should address all the feedback received when revising the paper.
Decision: Accept (poster)