Exact Solutions of a Deep Linear Network
Authors: Liu Ziyin
Botao Li
Xiangming Meng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Linear Network
Exact Solution
Collapse

Abstract: This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that zero is a special point in deep neural network architecture. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than $1$ hidden layer, qualitatively different from a network with only $1$ hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.

Exact Solutions of a Deep Linear Network
Liu Ziyin', Botao Li?, Xiangming Meng
Department of Physics, The University of Tokyo
2 Laaboratoire de Physique I'Ecole normale supérieure, ENS, Université PSL., CNRS, Sorbonne Université, Université de Paris Cité, Paris, France Institute for Physics of Intelligence, Graduate School of Science, The University of Tokyo
Abstract
This work finds the analytical expression of the global minima of deep linear network with weight decay and stochastic neurons, fundamental model for understanding the landscape of neural networks. Our result implies that zero is special point in deep neural network architecture. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero network with more than hidden layer, qualitatively different from network with only hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient ease the optimization of neural networks in general.
Introduction
Applications of neural networks have achieved great success in various fields. One central open question is why neural networks, being nonlinear and containing many saddle points and local minima, sometimes be optimized easily (Choromanska et al., 2015a) while becoming difficult and requiring many tricks to train in some other scenarios (Glorot and Bengio, 2010; Gotmare et al., 2018). One established approach is study the landscape of deep linear nets (Choromanska et al., 2015b), which are believed to approximate the landscape of a nonlinear net well. A series works proved the famous results that for deep linear net, all local minima are global (Kawaguchi, 2016; Lu and Kawaguchi, 2017; Laurent and Brecht, 2018), which is regarded to have successfully explained why deep neural networks are so easy to train because implies that initialization in any attractive basin can reach the global minimum without much effort (Kawaguchi, 2016). However, the theoretical problem of when and why neural networks can be hard train is understudied.
In this work, we theoretically study a deep linear net with weight decay and stochastic neurons, whose loss function takes the following form in general:
dddy,dammdd do Σ
D Σ 7h O
E, E,E,,n (0) (D)
(D)
(D)
(2)) (1)
12 in in 4;
1=1
Lg
La
(1)
where E, denotes the expectation the training set, U and W the model parameters, D is the depth of the network,' is the noise in the hidden layer (c.g., due to dropout), di is the width of the i-th layer, and y is the strength of the weight decay. Previous works have studied special cases of this loss function. For example, Kawaguchi (2016) and Lu and Kawaguchi (2017) study the landscape of Ly when is constant (namely, when there is no noise). Mehta et al. (2021) studies
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
IIn this work, we use "depth" to refer to the number of hidden layers. For example, a linear regressor has depth 0.

---

y
trivial globall minimum; nontrivial solution
trivial global minimum: no nontrivial solution
A8-
Rec/etts
trivial global minimum;
Iwy 120
Unccrree Dw@
parceptron (D-0)
with nontrivial solution
nontrivial global miniiuum; 0 local miniiumm
Swy de
nontrivial global minimum; 0 is saddle
Depth
0.03
0.02
0.04
0.06
0.08
0.10
1
2 and more
Figure I: Left: A summary of the network landscape that is implicd by the main results of this work when one increases the weight decay strength while fixing other terms. We show that the landscape of depth-1 net he precisely divided into two regimes, while, for D there exists least three regimes. The solid blue line indicates that division of the regimes is precisely understood. The dashed lines indicate that the conditions we found are not tight and may be improved in the future. Right: ResNet18 on CI.ARIO. The performance of a linear regressor never drops to that of trivial model, whereas the performance of ResNet1 8 drops to the level of a trivial model, like deep linear net with similar depth.
Lo with (a more complicated type of) weight decay but without stochasticity and proved that all the stationary points are isolated. Another line of works studies Lo when the noise is caused by dropout (Mianjy and Arora, 2019; Cavazza et al., 2018). Our setting is more general than the previous works in two respects. First, apart from the mean square error (MSE) loss Lo. La regularization term (weight decay) with arbitrary strength is included; second, the noise is arbitrary. Thus, our setting is arguably closer the actual deep learning practice, where the injection of noises latent layers is common, and the use of weight decay is virtually ubiquitous (Krogh and Hertz, 1992; Loshchilov and Hutter, 2017). One major limitation of our work is that assume the label y to be 1-dimensional, and it can be an important future problem to prove whether an exact solution exists or not when is high-dimensional.
Our foremost contribution is to prove that all the global minimum of an arbitrarily deep and wide linear net takes simple analytical form. In other words, we identify in closed form the global minima of Eq. (I) up to a single scalar, whose analytical expression does not exist in general. We then show that it has nontrivial properties that can explain many phenomena in deep learning. In particular, the implications of our result include (but are not limited to):
1. Weight decay makes the landscape of neural nets complicated;
we show that bad minima² emerge as weight decay is applied, whereas there is no bad minimum when there is no weight decay. This highlights the need to escape bad local minima in deep learning with weight decay.
2. Deeper nets are harder to optimize than shallower ones;
we show that a D 2 2 linear net contains a bad minimum at zero, whereas a = 1 net does not. This partially explains why deep networks are much harder to optimize than shallower ones in deep learning practice.
3. Depending on the task, the common initialization methods (such as the Kaiming init.) can initialize a deep model in the basin of attraction of the bad minimum at zero;
common initialization methods initialize the models a radius of roughly w/ around the origin; however, we show that the width of the bad minimum is task- dependent and can larger than the initialization radius for tasks with a small margin [[E])))]]
4. Thus, the use of (effective) weight decay is a major cause of various types of collapses in deep learning (for example, see Figure 1).
Organization: In the next section, we discuss the related works. In Section 3, we derive the exact solution for two-layer net. Section 4 extends the result to an arbitrary depth. In Section 5, we study and discuss the relevance of our results to many commonly encountered problems in deep learning. The last section concludes the work and discusses unresolved open problems. All proofs are delayed to Section B. Moreover, additional theoretical results the effect of including bias term is considered in Section D.
Unless otherwise specified, we use the word "bad minimum" to mean local minimum that is not a global minimum.

---

Notation. For a matrix W, we use W to denote the i-th row vector of W. || denotes the L2 norm if Z is a vector and the Frobenius norm if Z is matrix. The notation signals an optimized quantity. Additionally, we use the superscript and subseript - interchangeably, whichever leads to simpler expression. For example, be and (b*) denote the same quantity, while the former is "simpler." "
2 Related Works
In many ways, linear networks have been used to help understand nonlinear networks. For example, even at depth 0, where the linear net is just linear regressor, linear nets are shown to be relevant for understanding the generalization behavior of modern overparametrized networks (Hastie et al., 2019). Saxe et al. (2013) studies the training dynamics of a depth-1 network and uses it to understand the dynamics of learning of nonlinear networks. These networks are the same as linear regression model in terms of expressivity. However, the loss landscape is highly complicated due to the existence of more than one layer, and linear nets are widely believed to approximate the loss landscape of a nonlinear net (Kawaguchi, 2016; Hardt and Ma, 2016; Laurent and Brecht, 2018). In particular, the landscape of linear nets has been studied as early as 1989 in Baldi and Hornik (1989), which proposed the well-known conjecture that all local minima of a deep linear net are global. This conjecture is first proved in Kawaguchi (2016), and extended to other loss functions and deeper depths in Lu and Kawaguchi (2017) and Laurent and Brecht (2018). Many relevant contemporary deep learning problems can be understood with deep linear models. For example, two-layer linear VAE models used understand the cause of the posterior collapse problem (Lucas et al., 2019; Wang and Ziyin, 2022). Deep linear nets are also used to understand the neural collapse problem in contrastive learning (Tian, 2022). We also provide more empirical evidence in Section 5.
3 Two-layer Linear Net
This section finds the global minima of two-layer linear net. The data point is a d-dimensional vector x Id drawn from an arbitrary distribution, and the labels are generated through an arbitrary function y = y(x) e R. For generality, let different layers have different strengths of weight decay even though they often take the same value in practice. We want to minimize the following objective:
A Σ USe Σ WW - v) The 1wll1 nulIUPP.
La,d, LU, EE
where d1 is the width of the hidden layer and e, are independent random variables. Yu > 0 and Tu > 0 are the weight decay parameters. Here, we consider a general type of noise with [[4] = and B[{Ge,] ]] = o + 1 where diy is the Kronecker's delta, and o² > O. For shorthand, we use the notation Ag := B[xxx[=]]] and the largest and the smallest eigenvalues of Ao are denoted as amax and amin respectively. a, denotes the i-th eigenvalue of Ao viewed in any order. For now, it is sufficient for us to assume that the global minimum of Eq. (2) always exists. We will prove a more general result in Proposition 1, when we deal with multilayer nets.
3.1 Main Result
We first present two lemmas showing that the global minimum can only lie on a rather restrictive subspace of all possible parameter settings due to invariances in the objective.
Lemma 1. At the global minimum of Eq. (2), U3-
Tw Σ, W for all j.
7u
Proof Sketch. We use the fact that the first term of Eq. (2) is invariant to a simultaneous rescaling of rows of the weight matrix to find the optimal rescaling, which implies the lemma statement. O
This lemma implies that for all j. [Uy] must be proportional to the norm of its corresponding row vector W. This lemma means that using weight decay makes all layers of deep neural network balanced. This lemma has been referred to as the "weight balancing" condition in recent works (Tanaka et al., 2020), and, in some sense, is a unique and potentially essential feature of neural networks that encourages sparse solution (Ziyin and Wang, 2022). The following lemma further shows that, at the global minimum, all elements of must be equal.
'While we formally require and to nonzero, one can show that the solutions we provided remain global minimizers in the zero limit by applying Theorem 2 from Ziyin and Ueda (2022).
3

---

Lemma 2. At the global minimum, for all and j. we have
- Uj:
Proof Sketch. We show that if the condition is not satisfied, then an "averaging" transformation will strictly decrease the objective.
This lemma can be seen as a formalization of the intuition suggested in the original dropout paper (Srivastava et al., 2014). Namely, using dropout encourages the neurons to be independent of one another and results in an averaging effect. The second lemma imposes strong conditions on the solution of the problem, and the essence of this lemma is the reduction of the original problem to lower dimension. We are now ready to prove our first main result.
Theorem 1. The global minimum U, and W. of Eq. (2) is U, = and W. = if and only if
[[[zy]| $ Tw.
(4)
When [[a]]] Yw, the global minima are
U. br; W. rE] b (o² + di) Ao The [oo
(5)
6b
where r (tll,..,,+] is an arbitrary vertex of a di -dimensiooal hypercube, and b satisfies:
[0 (o²+ Ao mo E B[ay]]
Yu
(6)
Yw
Apparently, 0 is the trivial solution that has not learned any feature due to overregularization. Henceforth, we refer to this solution (and similar solutions for deeper nets) as the "trivial" solution. We now analyze the properties of the nontrivial solution 6* when it exists.
The condition for the solution to become nontrivial interesting: [[E[ay]] Yo The term [][my]]| can be seen the effective strength of the signal, and Youw is the strength of regularization. This precise condition means that the learning of a two-layer can be divided into two qualitatively different regimes: an "overregularized regime" where the global minimum is trivial, and "feature learning regime" where the global minimum involves actual learning. Lastly, note that our main result does not specify the exact value of b"" This is because b' must satisfy the condition in Eq. (6), which is equivalent to a high-order polynomial in b with coefficients being general functions of the eigenvalues of Ag, whose solutions are generally not analytical by Galois theory. One special case where an analytical formula exists for is when An o See Section C for more discussion.
3.2 Bounding the General Solution
While the solution to 6* does not admit an analytical form for general Ao, one can find meaningful lower and upper bounds to b* such that we can perform asymptotic analysis of b*. At the global minimum, the following inequality holds:
|| [62 (o² d1) amax the [] B[ s [1 (o² + +di Ap + The 1)- [ryy]l
s || 60 (o² + d1) amin / +w Tw E[
(7)
where amin and Gmax are the smallest and largest eigenvalue of Ag. respectively. The middle term is equal Tu/"ru by the global minimum condition in (33), and so, assuming Omin > 0, this inequality is equivalent to the following inequality of b":
Two
7a
Tu
7
(8)
(02 + di Amax
(02
+ d1 )mmi
Namely, the general solution 5" should scale similarly to the homogeneous solution in Eq. (105) treat the eigenvalues of Ag constants.

---

Exact Solution for An Arbitrary-Depth Linear Net
This section extends our result to multiple layers. We first derive the analytical formula for the global minimum of a general arbitrary-depth model. We then show that the landscape for a deeper network highly nontrivial.
4.1
General Solution
The loss function
d.d. d.d.dg
D - y )) rullU12
)²
Eg B, 2l0),....
( (1) (WD) U i ian 22...
Σ
(D)
Σ ri 1
where all the noises e are independent, and for all i and j. " and of We first show that for general D, the global minimum exists for this objective.
Proposition 1. For D and strictly positive Yuu M.... Tn. the global minimum for (9) exists.
Note that the positivity of the regularization strength crucial. one of the 7/8 is zero, the global minimum may not exist. The following theorem is our second main result.
Theorem 2. Any global minimum of Eq. (9) is of the form
U bbrD:
W(+) b,r,r
(10)
( - [[y] (b. II bi) (by IT bi)33 (o² +di) Ao Tod)
where ş 12-224 dr, 3² TP- do2 + d.), by 2 0 and b, 2 O. and r, = (=1, =1) is an arbitrary vertex of a di-dimensional hypercube for all i. Furthermore, let bj := wII/d]l3/d and bp := by by satisfies
M = yddx-1b..
(11)
Proof Sketch. We prove by induction on the depth D. The base case is proved in Theorem 1. We then show that for general depth, the objective involves optimizing subproblems, one which is a 1 layer problem that follows by the induction assumption, and the other is a two-layer problem that has been solved in Theorem Putting these two subproblems together, one obtains Eq. (10). O
Remark. We deal with the technical case of having bias term for each layer in Appendix D. For example, we will show that if one has preprocessed the data such that E[r) [r] = and EE E[y] = 0, our main results remain precisely unchanged.
The condition Eq. (1I) shows that the scaling factor b, for all i is not independent of one another.
This automatic balancing of the norm of all layers is consequence of the rescaling invariance of the multilayer architecture and the use of weight decay. It is well- well-known that this rescaling invariance also exists in neural network with the ReLU activation, and this balancing condition is also directly relevant for ReLU networks.
Condition (11) implies that all the by can be written in terms of one of the b;:
D bw 1=2
D
b I b
cosgn
I bb = cosgn by
[ b,
6"
(12)
2=
2=2
(19g (radgd1" P/2
p/2
where VTn. N22 7. TT2,d.va
and 2 0. Consider the first layer (i = 1), Eq (11) shows that the
global minimum must satisfy the following equation, which is equivalent to a high-order polynomial in that does not have analytical solution in general:
||[[z]] y [222 3² (o² + di) Ao + nw. T 1|2 dzb².
(13)
Thus, this condition is an extension of the condition (6) for two-layer networks.
At this point, it pays to clearly define the word "solution," " especially given that it has special meaning in this work because now becomes highly nontrivial to differentiate between the two types of solutions.
5

---

Definition 1. We say that non-negative real b is solution if it satisfies Eq. (13). A solution is trivial if and nontrivial otherwise.
Namely, a global minimum must be solution, but a solution is not necessarily a global minimum. We have seen that even in the two-layer case, the global minimum can be the trivial one when the strength of the signal is too weak or when the strength of regularization is too strong. It is thus natural to expect O to be the global minimum under a similar condition, and one is interested in whether the condition becomes stronger weaker as the depth of the model is increased. However, it turns out this naive expectation is not true. In fact, when the depth of the model is larger than 2, the condition for the trivial global minimum becomes highly nontrivial.
The following proposition shows why the problem becomes more complicated. particular, we have seen that in the case of a two-layer net, some elementary argument has helped us show that the trivial solution is either a saddle or the global minimum. However, the proposition below shows that with D 2 2, the landscape becomes more complicated in the sense that the trivial solution is always local minimum, and becomes difficult to compare the loss value of the trivial solution with the nontrivial solution because the value of b* is unknown in general.
Proposition 2. Let D 2 2 in Eq. (9). Then, the solution U O, W(D) = =0, ... W() = is local minimum with diagonal positive-definit Hessian y1.
Comparing the Hessian of D and D 1, one notices a qualitative difference: for D 2 2, the Hessian is always diagonal (at O); for D = 1, in sharp contrast, the off-diagonal terms are nonzero in general, and is these off diagonal terms that can break the positive-definitensss of the Hessian. This offers different perspective on why there is a qualitative difference between D = 1 and D 2.
Lastly, note that, unlike the depth-1 case, can longer find precise condition such that b 0 solution exists for a general Ag. The reason is that the condition for the existence of the solution is now high-order polynomial with quite arbitrary intermediate terms. The following proposition gives a sufficient but guronger-hhan-neeeseee
condition for the existence of a nontrivial solution, when all the o, intermediate width d, and regularization strength 7/4 are the same."
Proposition 3. Let of a² O. di do and 7 y for all i. Assuming amin 0, the only solution is trivial if
D-1
D 1
(D- (D-1))l (D-1)|[[2yy]
D+l
[[[ry]Idd
D-1
<7
(14)
2D
2D 2 Pamin
Nontrivial solutions exist if
D 1 |[xxy]ld
- (D-1)|B[ay]] 2Ddo do) Pamax
2.
(15)
2D
Moreover, the nontrivial solutions are both lower and upper-boundeds
1
|[[ do(o2 do) Omax
D41
S
(16)
dg
E
zy
Proof Sketch. The proof follows from the observation that the h.s.s. of Eq. (13) is a continuous function and must cross the r.h.s. under certain sufficient conditions.
One should compare the general condition here with the special condition for D 1. One sees that for D 2 2, many other factors (such as the width, the depth, and the spectrum of the data covariance A₂) come into play to determine the existence of a solution apart from the signal strength [[ry and the regularization strength
"This is equivalent to setting co ddg The result qualitatively similar but involves additional factors of cu if o., d, and % all take different values. We thus only present the case when de de, and 7 are the same for notational concision and for emphasizing the most relevant terms. Also, note that this proposition gives a sufficient and necessary condition if Au = o is proportional to the identity.
which equal to zero if E[ry] y. and if [[[yy]+ 7. With this definition, this proposition applies to two-layer net as well.
Foor D 1, we define the lower-bound as lim limp--1t
||

---

0.0
Dreea
2.00
0.35
1.75
0.30
1.50
1.2
1.00
0.75
0.50
-4. 0.2 0.0
0.2
0.4
-0.75
-0.75-0.50-0
0.25
5.00
0.5 0.50
0.75
The training loss as function hidden layer. For simplicity, dropout is not implemented. The non-linear activation functions we considered are ReLU, Tanh, and Swish. The left and right panels use diffcrent data. Left: X are Gaussian random vectors, and y= v- is linear function of Right: are Gaussian random vectors, and y tanh(x) nonlinear functions of data; the weight obtained as Gaussian random vector.
Figure 2:
network with different activation functions
4.2 Which Solution is the Global Minimum?
Again, we set = 7 0, of = o² > 0 and d, do > 0 for all for notational concision. Using this condition and applying Lemma 3 to Theorem 2, the solution now takes the following form, where b b200 O,
U v dobrp: w( br.rf-i
(17)
W() =r,E[]]"]]] de 200 dd2 Ao +b]+
2
The following theorem gives sufficient condition for the global minimum to be nontrivial. It also shows that the landscape of the linear net becomes complicated and can contain more than local minimum.
Theorem 3. Lei of = =22 0²> 0, do and 74 =7>0 all i and assuming Umin Then, if
-0t 1=(2200 + do) D-1
D-1 amaz
(18)
dB-1(D P
the global minimum of Eq. (9) is one of the nontrivial solutions.
While there various ways this bound can be improved, it is general enough for purpose. In particular, one sees that, for a general depth, the condition for having a nontrivial global minimum depends not only on the E[ry] and but also the model architecture general. For a more general architecture with different widths etc., the architectural constant co from Eq. (13) will also enter the equation. In the limit of D relation (18) reduces
|E[xy]1| 2
(19)
which is the condition derived for the 2-layer case.
Implications
Relevance nonlinear models. We first caution the readers that the following discussion should be taken with a caveat and is based on the philosophy that deep linear nets can approximate the nonlinear ones. This approximation certainly holds for fully connected models with differentiable activation functions such as tanh or Swish because they are, up to first-order Taylor expansion, a deep linear net around zero, which the region for which our theory the most relevant. We empirically demonstrate that close to the origin, the landscape of linear nets can indeed approximate that of nonlinear nets quite well. To compare, we plug the solution in Theorem 4 to both linear and nonlinear models of the same architecture and compare the loss values at different values of 6 around b O. For simplicity, we only consider the case D = 1. The activation functions we consider are ReLU, Tanh, and Swish (Ramachandran al., 2017), modern and differentiable variant of ReLU. See Fig. 2.
The regressor r R is sampled as Gaussian random vectors. We consider two methods of generating y; the first one (left) is y v- r. The second one (right) y = v lanh(x)) where the weight e e I Gaussian random vector. Fig. 2 shows that the landscape consisting of Tanh is always close to the linear landscape. Swish not as good as Tanh, the Swish landscape shows similar tendency to the linear landscape. The ReLU landscape is not so close to the linear landscape
is obtained
7

---

either for b > or b < O, but it agrees completely with the linear landscape on the other side, as expected. Besides the quantitative closeness, it is also important to note that all the landscapes agree qualitatively, containing the same number of local minima at similar values of b.
Landscape of multi-layer neural networks. The combination of Theorem 3 and Proposition 2 shows that the landscape of deep neural network can become highly nontrivial when there is a weight decay and when the depth of the model is larger than 2. This gives an incomplete but meaningful picture of a network's 's complicated but interesting landscape beyond two layers (see Figure I for an incomplete summary of our results). In particular, even when the nontrivial solution is the global minimum, the trivial solution is still local minimum that needs to be escaped. Our result suggests the previous understanding that all local minima of deep linear net are global cannot generalize to many practical settings where deep learning is found to work well. For example, a series of works attribute the existence of bad (non-global) minima to the use of nonlinearities (Kawaguchi, 2016) or the use of a non-regular (non- (non-differentiable) loss function (Laurent and Brecht, 2018). Our result, in contrast, shows that the use of simple weight decay sufficient to create a bad minimum." Moreover, the problem with such minimum is two-fold: (1) (optimization) it is not global and so needs to be "overcome" and (2) (generalization) it is a minimum that has not learned any feature at all because the model constantly outputs zero To the best of our knowledge, previous to our work, there has not been any proof that a bad minimum can generically exist in a rather arbitrary network the data." 7 Thus, our result offers direct and solid theoretical justification
without any restriction for the widely believed importance of escaping local minima in the field of deep learning (Kleinberg et al., 2018; Liu et al., 2021; Mori et al., 2022). In particular, previous works on escaping local minima often hypothesize landscapes that are of unknown relevance to an actual neural network. With our result, this line of research can now be established with respect to landscapes that are actually deep-learning-relevan
Previous works also argue that having a deeper depth does not create bad minimum (Lu and Kawaguchi, 2017). While this remains true, its generality and applicability to practical settings now also seem low. Our result shows that as long as weight decay is used, and as long D 2 2, there is indeed a bad local minimum at O. In contrast, there is no bad minimum at 0 for a depth-2 network: the point 6 0 is either saddle or the global minimum." Having deeper depth thus alters the qualitative nature of the landscape, and our results agree better with the common observation that a deeper network is harder, if not impossible, to optimize.
We note that our result can also be relevant for more modern architectures such as the ResNet. Using ResNet, one needs to change the dimension of the hidden layer after every bottleneck, and a learnable linear transformation is applied here. Thus, the "effective depth" of a ResNet would be roughly between the number of its bottlenecks and its total number of blocks. For example, ResNet18 applied to CIFARIO 10 often has five bottlenecks and 18 layers in total. We thus expect it to have qualitatively similar behavior to deep linear net with depth in between. See Figure 1. The experimental details are given in Section A.
Learnability of neural network. Now we analyze the solution when D tends to infinity. We first note that the existence condition bound in (15) becomes exponentially harder to satisfy as D becomes large:
(20) When this bound is not satisfied, the given neural network cannot learn the data. Recall that for two-layer net, the existence condition is nothing but [|[[,y]| > ?² independent of the depth, width, or stochasticity in the model. For a deeper network, however, every f'cctor comes into play, and the
[[[x)]] me Dlog[(o ddo)/ + 0(1).
"Some previous works do suggest the existence of bad minima when weight decay is present, but direct proof exists yet. For example, Taghvaci et al. (2017) shows that when the model is approximated by a linear dynamical system, regularization can cause bad local minima. Mehta et al. (2021) shows the existence of bad local minima in deep linear networks with weight decay through numerical simulations.
"In the case nonlinear networks without regularization, a few works proved the existence bad minima. However, the previous results strongly depend on the data and are rather independent of architecture. For example, one major assumption is that the data cannot be perfected and fitted by a linear model (Yun et al., 2018; Liu, 2021: He al., 2020). Some other works explicitly construct data distribution (Safran and Shamir, 2018: Venturi al., 2019). Our result, in contrast, is independent of the data.
*Of course, in practice, the model trained with SGD can still converge to the trivial solution even if it is a saddle point (Ziyin et al., 2021) because minibatch SGD is, in general, not good estimator of the local minima.
8

---

architecture of the model has strong (and dominant) influence on the condition. In particular, a factor that increases polynomially in the model width and exponentially in the model depth appears.
A practical implication is that the use of weight decay may be too strong for deep networks. If one increases the depth or width of the model, one should also roughly decrease Y according Eq. (20).
Insufficiency of the existing initialization schemes. We have shown that 0 is often a bad local minimum for deep learning. Our result further implies that escaping this local minimum can be highly practically relevant because standard initialization schemes are trapped in this local minimum for tasks where the signal B[zy] is weak. See Inequality (16): any nontrivial global minimum is lower-bounded by a factor proportional to (7/l|E[z)) "0]]] /dn, which can be seen as an approximation of the radius of the local minimum at the origin. In comparison, standard deep learning initialization schemes such as Kaiming init, initialize at radius roughly 1/vdo. Thus, for tasks B[ry] << >/Vdo. these initialization methods are likely initialize the model in the basin of attraction of the trivial regime, which can cause a serious failure in learning. To demonstrate, we perform numerical simulation shown in the right panel of Figure 3, where we train D= 2 nonlinear networks with width 32 with SGD on tasks with varying |[[[,]]]. For sufficiently small [|[]l]] the model clearly is stuck at the origin." In contrast, linear regression is never stuck at the origin. Our result thus suggests that it may be desirable to devise initialization methods that are functions of the data distribution.
1.00
0.55
trivtal souution
0.00
D-0
Tann
0.85
0.05
0.10
Figure 3: Training loss of D 2 neu- ral networks with ReLU and Tanh activa- tions across synthetic tasks with different [[[ay]]]- We see that with the Kaiming initialization, both the Tanh net and the ReLU net are stuck at the trivial solution in expectation our theory. optimized linear regressor (D = 0) better than the trivial solution when ||E[ry]] ]| > O. See Section A for experimental details.
contrast, an
Prediction variance of stochastic nets. A major extension of the standard neural networks is to make them stochastic, namely, to make the output a random function of the input. In a broad sense, stochastic neural networks include neural networks trained with dropout (Srivastava al., 2014; Gal and Ghahramani, 2016), Bayesian networks (Mackay, 1992), variational autoencoders (VAE) (Kingma and Welling. 2013), and generative adversarial networks (Goodfellow et al.. 2014). Stochastic networks are thus of both practical and theoretical importance to study, Our result can also be used for studying the theoretical properties of stochastic neural networks. Here, we present a simple application of our general solution to analyze the properties of a stochastic net. The following theorem summarizes our technical results.
Theorem 4. Let of o² > 0, d; = dg and %/4 = 7 > O for all i. Let Ag = T. Then, at any global minimum of Eq. (9), holding other parameters fixed,
1. in the limit of large do. ==rl[(=))=0005)
2. in the limit of large o², Var [ar[f(=)]= - 0 (o-j0))
3. In the limit of large D, Var [a)]]())] = O 222 -do)(d))
Interestingly, the scaling of prediction variance in asymptotic 0² is different for different widths. The third result shows that the prediction variance decreases exponentially fast in D. particular, this result answers a question recently proposed in Ziyin et al. (2022b): does stochastic net trained on MSE have a prediction variance that scales towards 0? We improve on their result in the case of deep linear net by (a) showing that the do' is tight in general, independent of the depth or other factors of the model, and (b) proving a bound showing that the variance also scales towards zero as depth increases, which is novel result of our work. Our result also offers an important insight into the cause of the vanishing prediction variance. Previous works (Alemi et al., 2018) often attribute the cause to the fact that wide neural network is too expressive. However, our result implies that this is not always the case because a linear network with limited expressivity can also have a vanishing variance as the model tends to infinite width.
Collapses in deep learning. Lastly, we comment briefly on the apparent similarity between different types of collapses that occur in deep learning. For neural collapse, our result agrees with the recent
'There are many natural problems where the signal is extremely weak. One well known example is the problem of future price prediction in finance, where the fundamental theorem of finance forbids large [|B[zy]]| (Fama, 1970).
9

---

works that identify weight decay as main cause (Rangamani and Banburski- 2022). For Bayesian deep learning. Wang and Ziyin (2022) identified the cause of the posterior collapse in two-layer VAE structure to be that the regularization of the mean of the latent variable is too strong. More recently, the origin and its stability have also been discussed as the dimensional collapse in self-supervised learning (Ziyin et al., 2022a). Although appearing in different contexts of deep learning, the three types of collapses share the same phenomenology that the model converges to a "collapsed" regime where the learned representation becomes low-rank constant, which agrees with the behavior the trivial regime we identified. We refer the readers to Ziyin and Ueda (2022) for study of how the second-order phase transition framework of statistical physies offer possible unified explanation of these phenomena.
Conclusion
In this work, we derived the exact solution of deep linear net with arbitrary depth and width and with stochasticity. Our work sheds light on the highly complicated landscape of a deep neural network. Compared the previous works that mostly focus the qualitative understanding of the linear net, our result offers a more precise quantitative understanding of deep linear nets. Quantitative understanding is one major benefit of knowing the exact solution, whose usefulness have also demonstrated with the various implications. The results, although derived for linear models, are also empirically shown be relevant for networks with nonlinear activations. Lastly, our results strengthen the line of thought that analytical approaches to deep linear models can be used to understand deep neural networks, and is the sincere hope of the authors to attract more attention this promising field.
Acknowledgement
Ziyin is financially supported by the GSS scholarship of the University of Tokyo and the JSPS fellowship. Li is financially supported by CNRS. X. Meng is supported by JST CREST Grant Number JPMJCR1912, Japan.
References
Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., and Murphy, K. (2018). Fixing a broken ELBO. In International Conference Machine Learning, pages 168- PMLR.
Baldi, P. and Hornik, K. (1989), Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58.
Cavazza, J., Morerio, P., Haeffele, B., Lane, C., Murino, V., and Vidal, R. (2018). Dropout low-rank regularizer for matrix factorization. In International Conference on Artificial Intelligence and Statistics, pages 435 445. PMLR.
Choromanska, A., Henaff, M., Mathicu, M.. Arous, G. B., and LeCun, Y. (2015a). The loss surfaces of multilayer networks. In Artificial Intelligence and Statisties, pages 192-204.
Choromanska, A., LeCun, Y., and Arous, G. B. (2015b). Open problem: The landscape of the loss surfaces of multilayer networks. In Conference on Learning Theory, pages 1756-17000 1760. PMLR
Fama, E. F. (1970). Efficient capital markets: A review of theory and empirical work. The journal of Finance, 25(2):383-417.
Gal, Y. and Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances neural information processing systems, 27.
10

---

Gotmare, A., Keskar, N. S., Xiong, C., and Socher, R. (2018). A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint ar.1v:1810.13243.
Hardt, M. and Ma, T. (2016). Identity matters in deep learning, arXiv preprinl arXiv:161
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2019). Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv: 1903.08560.
He, F., Wang, B., and Tao, D. (2020). Piecewise linear activations substantially shape the loss surfaces of neural networks, arXiv preprint Stti::2003.122.33
Kawaguchi, K. (2016). Deep learning without poor local minima. Advances in Neural Information Processing Systems, 29:586-599
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. vrXiv:1312.611
arXiv preprint
Kleinberg, B., Li, Y., and Yuan, Y. (2018). An alternative view: When does sgd escape local minima? In International Conference on Machine Learning pages 2698-2707. PMLR.
Krogh, A. and Hertz, J. A. (1992). A simple weight decay can improve generalization, In Advances in neural information processing systems, pages 950-957.
Laurent, T. and Brecht, J. (2018). Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pages 2902-2907. PMLR.
Liu, B. (2021). Spurious local minima are common for deep neural networks with piecewise linear activations. arXiv preprint arXiv:2102.1325:
Liu, K., Ziyin, L., and Ueda, M. (2021). Noise and fluctuation of finite learning rate stochastic gradient descent.
Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101
Lu, H. and Kawaguchi, K. (2017). Depth creates bad local minima. 172.v:1702.08580
arXiv preprint
Lucas, J., Tucker, G., Grosse, R., and Norouzi, M. (2019). Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse.
Mackay, D. J. C. (1992). Bayesian methods for adaptive models. PhD thesis, California Institute of Technology.
Mehta, D., Chen, T., Tang, T., and Hauenstein, J. (2021). The loss surface of deep linear networks viewed through the algebraic geometry lens. IEEE Transactions Pattern Analysis and Machine Intelligence.
Mianjy, P. and Arora, R. (2019). On dropout and nuclear norm regularization. In International Conference on Machine Learning, pages 4575-4584, PMLR.
Mori, T., Ziyin, L., Liu, K., and Ueda, M. (2022). Power-law escape rate of SGD.
Ramachandran, P., Zoph, B., and Le, Q V. (2017). Searching for activation functions.
Rangamani, and Banburski-Fahese A. (2022). Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2243-424.. IEEE.
Safran, I. and Shamir, O. (2018). Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning, pages 4433-4441. PMLR.
Saxe, A. M., McClelland, I.,, and Ganguli, S. (2013). Fxact solutions to the nonlinear dynamies of learning in deep linear neural networks. arXiv preprint

---

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958
Taghvaei, A., Kim, J. W., and Mehta, P. (2017). How regularization affects the critical points in linear networks. Advances in neural information processing systems, 30.
Tanaka, H., Kunin, D., Yamins, D. L.,, and Ganguli, S. (2020). Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Prucessing Systems, 33:6377-6389.
Tian, Y. (2022). Deep contrastive learning is provably (almost) principal component analysis. arXiv preprint arXiv:2201.12666
Venturi, L., Bandeira, A. S., and Bruna, J. (2019). Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, 20:133.
Wang. Z. and Ziyin, L. (2022). Posterior collapse of a linear latent variable model. arXiv preprint arXiv:2205.04000
Yun, C., Sra, S., and Jadbabaic, A. (2018). Small nonlincarities in activation functions create bad local minima in neural networks. arXiv preprint 18X2::880.003480
Ziyin, L., Li, B., Simon, J. B., and Ueda, M. (2021). SGD with Constant Large Learning Rate Can Converge to Local Maxima.
Ziyin, L., Lubana, E. S., Ueda, M., and Tanaka, H. (2022a). What shapes the loss landscape of self-supervised learning? arXiv preprint arXiv:2210.00638.
Ziyin, L. and Ueda, M. (2022). Exact phase transitions in deep learning. arXiv preprint 12X10:2205.12510.
Ziyin, L. and Wang, Z. (2022). Sparsity by Redundancy: Solving L1 with a Simple Reparametrization. arXiv preprint 110.022210.01212.
Ziyin, L., Zhang. H., Meng, X., Lu, Y., Xing, E., and Ueda, M. (2022b). Stochastic neural networks with infinite width are deterministic.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] The experiments are only for demonstration and are straightforward to reproduce following the theory.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] The fluctuations visually negligible.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are done on single 3080Ti GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ekTY9f8YgWd", "reviewer": "Reviewer_UdKB", "summary": "This paper analyzes the global minima of deep linear networks with weight decay. Under the assumption of linear architecture, $l_2$ regularization and population risk, the paper takes advantage of the symmetry and invariance in the network and derives the analytical expression of the minimum points. Depending on the regularization strength, for two-layer networks, zero is either the global minimum or a saddle point; for deeper networks, zero is always a local minimum and can be global (see Figure 1). The paper also tries to connect these theoretical results with some phenomena in deep nonlinear networks.", "questions": "1. An important result of this paper is that zero is a saddle point (not local minimum) when $D = 1$ if the strength of the signal is more significant. It would be better to mention it in Theorem 1 or its discussion.\n2. In Figure 1 Left, the labels of the inequality may need update.\n3. In Figure 2 Right, does the linear regression also have l2 regularization?", "limitations": "Please see the “Weaknesses” Section above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a set of assumptions under which the analytical expression of the minimum points can be derived, and the corresponding properties can be analyzed.\n2. I think the most interesting contribution of this paper is to point out that the weight decay, i.e., the $L_2$ regularization, may introduce a local minimum at zero for deep networks. It is not surprising since the regularization term in Eqn (1) is quadratic while $L_0$ has higher order. It is good to formulate the phenomenon as a rigorous theory under the assumptions.\n3. The presentation of the paper is pretty good. The settings are clearly stated and the results are supported by rigorous proofs. There are also detailed comments and discussions about the meaning and possible implications of the theoretical results.\n\nWeaknesses:  \nMy main concern is that the assumptions in this paper may over simplified the problem. The proofs are straightforward (though I believe Theorem 2 is not trivial), and heavily depend on the symmetry (thanks to the assumptions) that does not hold in general cases. If considering weaker assumptions, I guess we may still prove that zero is a local minimum, since the regularization is quadratic and the square loss has higher order, but the quantitative results in this paper may not be extended. Since the implications in Section 5 are all under weaker assumptions, the theoretical results may not support the discussions here strongly enough. In addition, I guess a ResNet architecture may avoid the local minimum at zero since the square loss is not in high order now.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bj8mj0uybUX", "reviewer": "Reviewer_QUdB", "summary": "This paper studies deep linear neural networks with weight decay and stochastic neurons. The authors show that the analytical global minima of square loss can be found for shallow neural networks (thm1) and deep neural networks (thm2). The analysis has some implications on the role of weight decay and the depth of neural networks.", "questions": "The paper assumes $\\mathbb{E}[\\epsilon_i] = 1$ and $\\mathbb{E}[\\epsilon_i\\epsilon_j] = \\delta_{ij}\\sigma^2 +1$ with $\\sigma^2>0$. If $\\epsilon$ are all 1, which is the case of neural networks with deterministic neurons, then $\\sigma^2$ = 0 which does not fit into this regime. Is it correct?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI like the setting in this paper, which is clean and simple but can manifest interesting properties of neural networks. The results are very interesting, especially the part where bad minima emerge with weight decay. \n\nWeakness:\n1) I understand the difficulty of analyzing global minima hence some assumptions are needed, e.g., diagonal A_0 for the exact form of b^* for the shallow neural networks., and single data. But some results are not easy to interpret, e.g., Thm2, Prop 3. Maybe the authors can provide more intuitions.\n\n2) I have a minor concern that in the main contributions, point 4 seems irrelevant to this paper. \n\n-- What's v in Eq.(8)?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NJOlXcf0fqd", "reviewer": "Reviewer_PsE5", "summary": "This work studies the population loss landscape of stochastic (e.g., in the sense of dropout) deep, linear, neural networks under 2-norm weight regularization. The key contribution of this work is the derivation of analytical expressions for the global minima of the aforementioned loss landscape, at least up to a scalar quantity. The implications of this result for training both linear and non-linear neural networks are discussed. In particular, this result illustrates how weight decay and depth can lead to a more challenging optimization problems, as well as the importance of the role of network initialization in avoiding basins of attraction around bad minima.", "questions": "- Lemma 2 if I have understood correctly, it essentially seems to say the output layer weight vector $U$ is of the form $c\\textbf{v}$, where $\\textbf{v} \\in\\{ \\pm 1\\}^d$ vector, likewise the the rows of $W$ are just signs of the same constant row vector. How does this compare to the case with no noise?\n\n- The networks you discuss have no bias parameters, how straightforward would it be for you to extend these results and techniques to also cover this case?\n\n- The remark after proposition 1 is useful. Can you provide examples of cases where a global minimum doesn't exist when you set a single regularizer weight to 0?", "limitations": "I think the authors are reasonably upfront about the limitations of their work, although I think they could perhaps add some suggested avenues for future works. I can't envisage how this work might have a negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: this paper continues the line of work analyzing linear neural networks. I am not a specialist concerning the study of linear networks, but the results appear, at least to the best of my knowledge, novel and interesting.\n\nQuality and clarity: on the whole I think the paper is well organized, well written and clear. A few very minor suggestions in regard to the paper's presentation are as follows.\n\n- I think upfront you could state your network architecture/ forward pass function more clearly / in terms of matrix vector product and give the dimensions of each of your parameter matrices.\n\n- Notation wise both scalars and vectors use lower case characters which can be a bit confusing, perhaps using bold lower case characters for vectors might help.\n\n- It might be helpful for the reader to restate statements of lemmas and theorems in the supplementary so they don't have to flick between.\n\n- Line 475 in the supplementary \"...we see that the left hand side \\textit{is} larger...\"\n\n\nSignificance: I think the extent to which understanding linear networks is important for understand nonlinear networks is not entirely clear. I still think it is important that we understand deep linear networks regardless however, and this work seems like a useful contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MTx5qJIqXl", "reviewer": "Reviewer_SZFK", "summary": "This paper provides a closed-form solution (up to some constant) of the global minima of linear neural networks when trained using square loss and strictly positive weight decay. This result can be extended to the case when the neurons are stochastic and independent. The formulas for the global minima are directly or potentially related to the weight decay, depth, stochasticity of neurons, and signal strength from training data. The authors also used the characterizations of global minima to explain multiple phenomena happening in real neural network training, e.g., deeper networks are harder to optimize. This paper also provided variance analysis in the asymptotic limits of network hyperparameters and did small-scale experiments on synthetic data to validate their theoretical results in non-linear networks.", "questions": "- Could the authors provide more theoretical or empirical evidence connecting the loss landscape of linear networks and those of non-linear ones?\n\n- Is it true that the weights of linear neural networks will almost always converge to the global/local minima characterized by the authors in this paper? Is it possible that the weights of the neural networks could converge to some points that are not captured in this paper, or even diverge?\n\n- Can the proof techniques used in this paper be generalized to other settings? For example, when there are other kinds of regularizations or no regularizations at all.\n\n- (Minor questions) Could the authors address the two questions I mentioned in the \"Minor Comments\" section above? In other words, could the authors provide the detailed settings of their experiments and the definition of the notation \"$v$\" in equation (8)?", "limitations": "The authors stated the assumptions for their theoretical results in the paper and had many discussions about the implications. It might be better if the authors could discuss more explicitly the limitations of this paper in the implication and conclusion sections.\n\nThis paper is mostly theoretical and focuses on a fundamental problem in general neural network training, and I do not see any immediate negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper theoretically gives an analytical formula (up to some constant) for the global minima of linear neural networks trained with weight decay, and this formula works for deep linear networks and could be generalized to independent stochastic neurons. These analytical expressions provide opportunities to study the properties of these global minima of deep linear networks in detail.\n\n- It is an interesting idea to connect the formula of the global minima of neural networks to various common phenomena in this field, e.g., the collapses in deep learning.\n\n- This paper is generally well-written and well-structured. The notations used in this paper are mostly well-defined, and the intuitions and implications of the theoretical results are provided in the main text. These make this paper easy to understand.\n\n- The theoretical proofs in this paper appear to be correct, and the related works are adequately cited.\n\nWeaknesses:\n\n- The theoretical results in this paper are all about linear networks, and the relationship between linear networks are non-linear ones seems somewhat unclear, so most of the conclusions in this paper might not translate directly to non-linear settings. This is my major concern about this paper. The authors claimed on Line 16 that the landscapes of linear networks are believed to well approximate that of non-linear ones, but this claim might be vague and need further explanation. The authors also did experiments with a small two-layer non-linear network on synthetic data, but the scale of this experiment is small, so it is unclear whether this result still holds for more general settings. It would be better if the authors could provide more theoretical or empirical evidence connecting the loss landscape of linear neural networks and those of non-linear ones.\n\n- The characterizations of global minima might not be enough to characterize the training of neural networks, which depends on the properties of the entire loss landscape. For instance, it is possible that the weights of neural networks could diverge and it never reaches a minimum, and it is also possible that the weights converge to a bad local minimum or saddle point. It might be better if the authors could (theoretically or empirically) eliminate these possibilities and show that the network weights will always converge to the points that they characterized, i.e., either the global minima or the bad local minimum at 0.\n\n- The proof techniques used in this paper seem to heavily rely on the existence of weight decay at all layers, making it hard to be generalized to other settings. Without weight decay, relationships like equation (13) will break and the characterizations of the local and global minima could become much more complicated.\n\n- Some arguments made in this paper might be somewhat vague. For example, in Line 268, it might be unclear what the authors mean by \"cannot learn the data\". \n\nMinor Comments:\n\n- The details of the experiments the authors did to produce Figure 2 are missing. These details (e.g., how the data are generated, and what the hyperparameters are) could be important for interpreting the experimental results.\n\n- The notation \"$v$\" on the left-hand side of equation (8) seems undefined. Should it be defined as some term in equation (5)?\n\nTypos:\n\n- Line 243, \"are global and cannot generalize\" -> \"are global cannot generalize\"\n- Line 269, \"two-layer net, and the existence\" -> \"two-layer net, the existence\"\n\n-------------------------------------------------\nUpdate after author response:\n\nI have read all other reviews and the authors' responses, and I decided to increase my score by 2. There are two main reasons why I increase my score:\n1.  The authors have added empirical evidence (e.g., ResNet on CIFAR) to further relate the loss landscape of linear neural networks to non-linear ones.\n2. The proof framework in this paper can be extended to more general settings with similar results, and the authors have provided theoretical results in more general settings, especially when there is no weight decay.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept