Toward Understanding Generalization of Over-parameterized Deep ReLU network trained with SGD in Student-teacher Setting
Authors: Yuandong Tian
Venue:  (2020)
Keywords: deep ReLU network
theoretical analysis
generalization
training dynamics
student teacher setting
interpolation region
over-parameterization

Abstract: To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). Our contributions are two-fold. First, we prove that when the gradient is zero (or bounded above by a small constant) at every data point in training, a situation called  \emph{interpolation setting}, there exists many-to-one \emph{alignment} between student and teacher nodes in the lowest layer under mild conditions. This suggests that generalization in unseen dataset is achievable, even the same condition often leads to zero training error. Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training. As a result, it could take a long time to converge into these small-gradient critical points. Our analysis shows that over-parameterization plays two roles: (1) it is a necessary condition for alignment to happen at the critical points, and (2) in training dynamics, it helps student nodes cover more teacher nodes with fewer iterations. Both improve generalization. Experiments justify our finding.

Under review as a conference paper at ICLR 2020
STUDENT SPECIALIZATION IN DEEP RELU NET- WORKS WITH FINITE WIDTH AND INPUT DIMENSION
Anonymous authors Paper under double-blind review
ABSTRACT
To analyze deep ReLU network, we adopt a student-teacher setting which an wer-paraaeeterized student network learns from the output of a fixed teacher net- work of the same depth, with Stochastic Gradient Descent (SGD), Our contri- butions two-fold. First, we prove that when the gradient is small every training sample, student node specializes to teacher nodes in the lowest layer un- der mild conditions. Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of train- ing. As a result, it could take a long time to converge into these small-gradien critical points. Our analysis shows that over- vve-ppraameeerization dition for specialization to happen at the critical points, and helps student nodes cover more teacher nodes with fewer iterations. Both improve generalization. Dif- ferent from Neural Tangent Kernel (Jacot et al. (0188) and statistical mechanics approach (Goldt et al., 2019). our approach operates on finite width, mild over- parameterization (as long as there more student nodes than teacher) and finite input dimension. Experiments justify our finding.
a necessary con-
INTRODUCTION
Deep Learning has achieved great success in the recent years (Silver al. 2016 He et 2016 Devlin et al.) 2018). Although networks with even one-hidden layer can fit any function Hormik 1989). remains an open question how such networks can generalize to new data. Different from what traditional machine learning theory predicts, empirical evidence (Zhang et al. 2017) shows more parameters in neural network lead to better generalization. How over-parameterizaaion yields strong generalization is important question for understanding how deep learning works.
In this paper, we analyze deep ReLU networks with teacher-student setting: a fixed teacher network provides the output for student to learn via SGD. Both teacher and student are deep ReLU net- works. Similar to (Goldt et al. 2019). the student is over-realized compared to the teacher: at each layer 1, the number n; of student nodes is larger than the number my of teacher (nz >> my). Al- though over-realization is different from pver-parumeterizatio i.e., the total number of parameters in the student model is larger than the training set size N, over-realization directly correlates with the width of networks and is measure of parameterization.
The student-teacher setting has long history (Saad & Solla 1996 1995 Freeman & Saad 1997
Mace & Coolen| 1998) and recently gains increasing interest Goldt et al. 2019 Aubin et al. 2018)
analyzing 2-layered network. While worst-case performance on arbitrary data distributions may not be a good model for real structured dataset and can be hard to analyze, using a teacher network implicitly enforces an inductive bias and could potentially lead better generalization bound.
Specialization, that is. a student node becomes increasingly correlated with teacher node during training (Saad & Solla 1996), is one the important topic in this setup. all student nodes are specialized to the teacher, then student tends to output the same as the teacher and generalization performance can be expected. Empirically, has been observed in 2-layer networks (Saad & Solla, 1996 Goldt al. 2019) and multi-layer networks (Tian et al. 2019 Li et al. 2016), in both synthetic and real dataset. In contrast, theoretical analysis is limited with strong assumptions (e.g.. Gaussian inputs, infinite input dimension, local convergence, 2-layer setting, small number of hidden nodes). In this paper, with arbitrary training distribution and finite input dimension, we show rigorously that when gradient at each training sample is small (i.e., the interpolation setting suggested in (Ma

---

Under review as a conference paper at ICLR 2020
et al. 2017 Liu & Belkin 2018 Bassily et al. 2018), the student node the lowest layer can be proven to specialize to the teacher nodes: each teacher node is aligned with at least one student node in the lowest layer. This explains one-to- to-many mapping between teacher and student nodes and the existence of un-specialized student nodes, as observed empirically in (Saad & Solla 1996). Furthermore, from the proof condition, more over-realization encourages specialization.
Our setting is different from previous works. (1) While statistical mechanics approaches (Saad & Solla| 1996 Goldt al. 2019; Gardner & Derrida 1989, Aubin al. 2018) assume both the training set size N and the input dimension d goes to infinite (i.e., the thermodynamics limits) and assume Gaussian imputs, our analysis allows finite d and impose no parametric constraints on the input data distribution. (2) While Neural Tangent Kernel (Jacot et al. 2018 Du et al., (0188) and mean-field approaches (Mei al. 2018) requires infinite (or very large) width, our setting applies to finite width as long as student is slightly over-realized (nı 2 mi). In this paper we study the infinite training sample case (the training set is a region), and leave finite sample analysis as the future work.
In addition, we further analyze the training dynamics and show that most student nodes converge first towards strong teacher nodes with large fan-out weights in magnitude. While this makes train- ing robust to dataset noise and naturally explains implicit regularization, the same mechanism also leaves weak teacher nodes unexplained until very late stage of training, yiclding high generalization error with finite iterations. In this situation, we show that over-realization plays another important role: once the strong teacher nodes have been covered, there are always spare student nodes ready to switch to weak teacher nodes quickly. Empirically, show more teacher nodes are covered with the same number of iterations, and generalization is also improved.
We verify our findings with numerical experiments. Starting with 2-layer setting, we justify Theo- rem 2 and Theorem 3 with Gaussian inputs, showing one-to-many specialization and existence of un-specialized nodes. For deep ReLU networks. we show specialization happens not only in the lowest layer, as suggested by Theorem 4 but also in other hidden layers, on both Gaussian inputs and CIFARIO. We also perform ablation studies about the effect of student over-realization. For training dynamics, we show the strong/weak teacher effects in 2-layer settings and over-realization could improve specialization and generalization.
RELATED WORKS
Student-feacher setting. This setting has a long history (Figgll & Van den Broeck 2001 Gard- ner & Derrida, 1989). The seminar works (Saad & Solla 1996: 1995) studies 1-hidden layer case from statistical mechanics point of view in which the input dimension goes to infinity, or so-called thermodynamics limits. They study symmetric solutions and locally analyze the symmetric break- ing behavior and onset of specialization of the student nodes towards the teacher. Recent follow-up works (Goldt et al. 2019) makes the analysis rigorous and empirically shows that random initial- ization and training with SGD indeed gives student specialization in 1-hidden layer case, which is consistent with our experiments. With the same assumption, (Aubin et al. (018)) studies phase tran- sition property of specialization in 2-layer networks with small number of hidden nodes using replica formula. In these works, inputs are assumed to be Gaussian and step or Gauss error function is used as nonlinearity. Few works study teacher-student setting with more than two layers. (Allen-Zhu cet al. 2019a) shows the recovery results for 2 and 3 layer networks, with modified SGD, batchsize and heavy over-parametent seee-parameterization.
In comparison, our work shows that specialization happens around the SGD critical points in the lowest layer for deep ReLU networks, without any parametric assumptions of input distribution.
Local minima is Global. While in deep linear network, all local minima are global (Laurent & Brecht 2018, Kawaguchi] (016)), situations are quite complicated with nonlinear activations. While local minima is global when the network has invertible activation function and distinct training sam- ples (Nguyen & Hein| 2017 Yun et al., 2018 or Leaky ReLU with linear separate input data (Lau-) rent & von Brecht 2017), multiple works (Du et al. 2018a Ge et al.) 2017 Safran & Shamir 2017 Yun et al.] 2019) show that in GD case with population or empirical loss, spurious local minima can happen even in two-layered network. Many are specific to two-layer and hard to generalize to multi-layer setting. In contrast, our work brings about generic formulation for deep ReLU network and gives recovery properties in the student-teacher setting.
2

---

Under review as a conference paper at ICLR 2020
(a)
ng
(b)
Layer
f((e))
fiz(k)
fir nodes)
78g
R1
Wi
8((x)
(ta(x)
mg
Layer! (N4 nades)
x
X
Rta(x))
Teacher Network (Ficed perameters)
Studont Network (Leameblee Paramabera)
Figure 1: Problem Setup, (a) Student- teacher setting. The student network learns from the output of a fixed teacher network via stochastic gradient descent (SGD). (b) Notations. All low cases scalar, bolds are column vectors and upper cases are matrices.
Learning wild networks. Recent works on Neural Tangent Kernel (Jacot et al., 2018: Du et al, |018b] Zlhen-Zhu et al., 2019b) show the global convergence of GD for multi-layer networks with infinite width. (Li & Liang 2018) shows the convergence in one-hidden layer ReLU network us- ing GD/SGD solution with good generalization, when the input data are assumed to be clustered into classes. Both lines of work assume heavily over-parameterized network, requiring polynomial growth of number of nodes with respect to the number of samples. (Chizat & Bach (0188) shows global convergence of over-varaman wver-parameterized network with optimal transport. (Tian et al. 2019) as- sumes mild over-realization and gives convergence results for 2-layer network when subset of the student network is close to the teacher. Our work extends it with much weaker assumptions.
Deep Linear networks. For deep linear networks, multiple works (Lampinen & Ganguli 2019 Saxe et al. 2013 Arora et al. [2019] [Advani & Saxe (0117) have shown interesting training dynam- ies. One common assumption that the singular spaces of weights at nearby layers are aligned at initialization, which decouples the training dynamics. Such a nice property would not hold for nonlinear network. (Lampinen & Ganguli 2019) shows that under this assumption, weight compo- nents with large singular value are learned first, while we analyze and observe empirically similar behaviors on the student node level. Generalization property of linear networks can also be analyzed in the limit of infinite input dimension with teacher-student setting (Lampinen & Ganguli 2019). However, deep linear networks lack specialization which plays a crucial role in the nonlinear case. To our knowledge, we are the first analyze specialization rigorously in deep ReLU networks.
3 MATHEMATICAL FRAMEWORK
Notation. Consider student network and its associated teacher network (Fig. aa)). Denote the input as x. We focus on multi-layered networks with (-)) as ReLU nonlinearity. We use the follow- ing equality extensively: (xx) = I[ > Ojæ, where I[.) is the indicator function. For node j. (x),, (x) and 9ز (x) are its activation, gating function and backpropagated gradient after the gating.
Both teacher and student networks have L layers. The input layer is layer 0 and the topmost layer (layer that is closest to the output) is layer L. For layer I, let my be the number of teacher node while ny be the number of student node. The weights Wi E 1gmt-1xm, x refers to the weight matrix that connects layer 1 --1 1 to layer on the student side. Wi = [Wr.1: Wi,2, Wi,n, where each w E R. is the weight vector. Similarly we have teacher weight W E Rm 1 1xm, Denote W = (W,, W2...,,,.WW W.) as the collection of all trainable parameters.
Let f((x)
(x), .-..- ft.m, (x))T E R be the activation vector of layer 1, Dz(x) diag [1 (x), 31,, (x)] E RRnixm be the diagonal matrix of gating function (for ReLU it is either 1), and (x) = (x)........
(x)] " be the backpropated gradient vector. By E Re and my = no. Note that fx),, (x) and (x) are all dependent on W. For brevity, we often use (xx rather than fx(x; W).
definition, the input layer has fu(x)
All notations with superscript are from the teacher, only dependent on the teacher and remains the same throughout the training. Di(x) (x) = Icxc since there no ReLU gating. Note that C' is the dimension of output for both teacher and student. With the notation, gradient descent is:
Wi = Ex (x) (x)]
(1)
In SGD, the expectation Ex [1] is taken over a batch. In GD, it is over the entire dataset.
3

---

Under review as a conference paper at ICLR 2020
Bias term. With the same notation Ria: 1+1)xm WL1 = [w:b] E IR 1+1, fi E Ru (last column is all one), g E R"ttl and Dj E (R((++))((x+)+ (last diagonal element is always 1).
also include the bias tenm. In this case, Wi
Objective. We assume that both the teacher and the student output vector. We the output of teacher as the input of the student and the objective is:
min J(W) Bx [l1 (x) W
We want to ask the following geustion:
Are student nodes specialized to teacher nodes at the same layers after training?
One might wonder this is hard since the student's 's intermediate layer receives no direct supervision from the corresponding teacher layer, but relies only on backpropagated gradient. Surprisingly, the following theorem shows that it is possible for every intermediate layer:
Lemma (Recursive Gradient Rule). At layer I, the backpropagated gi gx() satisfies
gi (x) = Di(x) [A1())) (x) (xxx)((xx)),
(3)
where the mixture coefficient A((x) = Vú (x E R and Bi(x) (x)) E Ruxxn; The matrices Vi(x)= E RCmm and Vi Rxmm are defined in a top-down manner:
Vx- = (x(x), vi=(())=Vi (x'(x)
In particular, Vz(x) (x) lexc.
For convenience, ment of A,, (x) = (x) (x) and element of B₁, (x) = (x)v Note that Lemma D applies to arbitrarily deep ReLU networks and allows different number of nodes for the teacher and student. In particular, student can be over-parameterized (or over-realized).
write Vi(x) (x),, (z.2(x)..... Vi,n, (x)]. then have each ele-
Let Rg = {x: p(x) ) > 0} be the infinite training set, where p(x) is the input data distribution. Let Ri (f((x) x Ro which is the image of the training set at the output of layer I, and also convex polytope. Then the mixture coefficient (x) and (x(x) have the following property:
Corollary (Piecewise constant). Ro can decomposed into a finite (but potentially exponential) set of regions RT-1 = (RI_1, Ri R. A((x) and Bt (x) are constant in R
CRITICAL POINT ANALYSIS
We first show that due to property of ReLU node and subset sampling in SGD, at SGD critical point, under mild condition, the teacher node aligns with at least one student node and the goal (*) can be reached in the lowest layer.
Definition (SGD critical point). W is SGD critical point for any batch, Wi = 0.for 1 < L.
Theorem (Interpolation), Denote D {xx} dataset of N samples. If V is critical point for SGD, then either gi (x;; W) 0 or ( W) 0.
Such critical points exist since over-realized student mimic teacher perfectly. Note that critical points SGD is much stronger than those in GD, where the gradient is always averaged at a fixed data distribution. If f-1 has a bias term (and f * 0 always), then (x;;; W) = 0. For topmost layer, immediately we have (x;; W fx ) fy (x;) = O. which is global optimum with zero training loss. the following, we want check whether this condition leads to specialization, i.e., whether the teacher's 's weights recovered/aligned by the student, i.e., whether for teacher j. there exists a student & at same layer so that Wa = yw, for some > 0.
Note that gi (x;; W) 0 might be a strong assumption since in practice the gradient is small but never zero. A weaker assumption is that (x;; W) S c or even E, (x;; W))l S e. For this, Theorem shows (approximate) alignment/speeciaiizatoo still holds for noisy case.

---

Under review as a conference paper at ICLR 2020
227
2.77
0.66
0.4
6.4
2-4
381
5.4
0.30
2.5
63
60
-02
-32
-04
-04
-44
-34
-05
-3X 42
-22
07
6/
16
60
04
21 -22
02
-66
.F2
02 14 d4
Figure 2: Convergence (2 dimension) for teachers (solid line) and 6 students (dashed line). Legend shows ||v=|| for student node k. ||v=|| - 0 for nodes that are not aligned with teacher.
4.1 ASSUMPTION OF TEACHER NETWORK
Obviously, arbitrary teacher network won't von't be reconstructed. A trivial example is that teacher network always output since all the training samples lie in the inactive halfspace of its ReLU nodes. Therefore, we need to impose condition the teacher network.
Let E, = [x: (x) > 0} be the activation region of node j. Note that the halfspace E, is an open set. Let 8E;- (x [x( 0} be the decision boundary of node
Definition 2 (Observer). Node k is observer of node j if Eş n nOE, + 0. Assumption 1 (Teacher Network). For each layer I, we require that (1) the teacher weights wio are not co-linear. and (2) the boundary of wi.j is visible in the training set: a O Ri-1 * 0.
Assumption I is our assumption of the teacher. The first requirement is trivial. The second one is reasonable since two teacher nodes who behaves linearly in the training set are indistinguishable.
4.2 ALIGNMENT OF TEACHER WITH STUDENT, 2-LAYER CASE
We first start with 2-layer case, in which A₁ (x) and B (x) constant with respect x, since there is no ReLU gating at the top layer | = 2. In this case, from the SGD critical point at ( = 1, g1 (x) (x) (x) (x) O, alignment between teacher and student can be achieved: Theorem 2 (Student-teacher Alignment, 2-layers). With Assumption 7 at SGD critical point, if teacher node j is observed by a student node l and axj + 0, then there exists at least one student node k' aligned with j.
The intuition is that if the input takes sufficiently diverse values, ReLU activations a [w]xx can be proven to be mutually linear independent. On the other hand, the gradient of each student node b, (x) 0, linear combination of teacher and student nodes (note
k when active, is a. a. and 31 are k-thh rows of Ai and B1). Therefore, zero gradient means that the summation of coefficients of co-linear ReLU nodes is zero. Since teachers are not co-linear, any teacher node is co-linear with at least one student node. Alignment with multiple student nodes is also possible. there is nonlinearity (e.g., deep linear models), alignment won't 't happen since linear subspace has many representations.
Note that necessary condition of reconstructed teacher node is that its boundary is in the ac- tive region of student, or is observed (Definition 2 This is intuitive since a teacher node which behaves like a linear node is partly indistinguishable from bias term. This also suggests that over- parameterization (more student nodes) are important. More student nodes mean more observers, and the existence argument in Theorem more likely to happen and more teacher nodes can be covered by student, yielding better generalization.
For student nodes that are not aligned with the teacher, if they are observed by other student nodes, then following a similar logic, we have the following:
Theorem 3 (Prunable Un-specialized Student Nodes). With Assumption i at SGD critical point, if an unaligned student & has C independent observers (concatenating yields full rank matrix), then Ew [kk) Ve | || = O. If node k is not co- co-linear with any other student, then V O.
Corollary 2. With sufficient observers, the contribution of all unaligned student nodes is zero.
5

---

Under review as a conference paper at ICLR 2020
Theorem 3 and Corollary 2 open the way of network pruning (LeCun et al. 1990: Hassibi al.
1993 Hu et al. 2016p. This is consistent with Theorem in (Tian et al., 2019) which also shows the fan-out weights zero up on convergence in 2-layer networks, if the initialization is close. In contrast, Theorem 3
analyzes the critical point rather than the dynamics.
Note that relate theorem (Theorem 6) in (Laurent & von Brecht 2017) studies 2-layer network with scalar output and linear separable input, and discusses characteristics of individual data point contributing loss in a local minima of GD. Here no linear separable condition is imposed.
4.3 MULTI-LAYER CASE
Thanks to Lemma I which holds for deep ReLU networks, we can use similar intuition to analyze the behavior of the lowest layer (1 = 1) in the multiple layer case. The difference here is that A, (x) and B1 (x) are no longer constant over x. Fortunately, using Corollary 1 we know that A1 (x) and B3 (x) piece-wise constant that separate the input region Rg into finite (but potentially exponential) set of constant regions Ry = {RR Ry Rd' } plus a zero-measure set. This suggests that we could check each region separately. If the boundary of a teacher j and a student lies the region, similar logic applies (here ozj is the (k,j) entry of A (x) and is constant in a region R E Ro)..
Theorem 4 (Student-teacher Alignment, Multiple Layers), With Assumption 7 at SGD critical points, for any teacher node at I = 1, if there exists region R R and a student observer l so that dE; n Eg O R. + 0 and oxg (R) + 0, then node aligns with at least one student node K'.
Note that even with random V (x) (e.g., at initialization), Theorem still holds with high probability (when Cgj * 0) and teacher fi (x) can still align with student fx).. This suggests a picture of bottom-up training in backpropagation: After the alignment of activations at layer 1, we just treat layer as the low-level features and the procedure repeats until the student matches with the teacher at all layers. This is consistent with many previous works that empirically show the network is learned in a bottom-up manner (Li et al. 2018).
Note that the alignment may happen concurrently across layers: if the activations of layer 1 start to align, then activations of layer 2, which depends on activations of layer 1, will also start to align since there now exists a W2 that yields strong alignments, and so This creates critical path from important student nodes at the lowest layer all the way to the output, and this critical path accelerates the convergence of that student node. We leave formal analysis to the future work.
Small Gradient Case. In practice, stochastic gradient (or its expectation over time) fluctuates around zero (Igg (x))leo I c, or E, [[]g] [x]] I <), but never zero. In this case, Theorem shows that a rough specialization still follows. The ratio of recovery is also shown for weights/biases separately, as a function of c. Note 0zz is the angle of two weights w, and w.
Theorem 5 (Noisy Recovery). If Assumption I hholds and any two teachers wJ. w satisfy 85y 2 Oo > or 163, - b3] 2 by > O. Suppose [[gı (x, W) S for any x E Rg with e S co. then for any teacher atl = 1, if there exists region R E R and a student observer i so that 0E E, NR + 0.
and onj (R) # 0, then j is roughly aligned with a student k': sin OzK O 22-6 O (1
and
|b} by
28 for any 8% O. The hidden constants depends on 5, co and the size of region aE E
Note that Eg [x)]l=] S e leads Igi (x)] S at least for some iteration Therefore, Theorem 5 still applies since it does not rely on past history of the weight/gradient. Note that Theorem assumes infinite number of data points and leave finite sample case future work.
ANALYSIS ON TRAINING DYNAMICS
Our analysis so far shows student specialization happens at SGD critical points under mild condi- tions. A natural question arises: is running SGD long enough sufficient to achieve these critical points? Some previous works (Ge et al. 2017 Livni et al. 2014) show that empirically SGD does not recover the parameters of a teacher network up to permutation, while other works (Saad & Solla 1996 Goldt al. 2019) show specialization happens. Why there is discrepancy? There several reasons. First, from Theorem 3 there exist un-specialized student nodes, so a simple permutation test student weights might fail. Second, as suggested by Theorem 5, it can take long time to

---

Under review as a conference paper at ICLR 2020
recover teacher node i with small |vvll (since akj Vv v)). In fact, if v 0 then it has contribution to the output and recovery never happens. This is particularly problematic if the output dimension is 1 (scalar output), since single small teacher weight ug would block the recovery of the entire teacher node k. Previous works (Lampinen & Gangulij 2019) shows similar behaviors in the dynamics of singular values in deep linear networks in teacher-student setting, which lack student specialization. Here we study these behaviors in deep ReLU networks.
In the following, we analyze various local dynamic behaviors of 2-layer ReLU network. Due the complexity, we leave a formal characterization of the entire training procedure for future work.
Definition 3. A teacher node j strong (or weak), if [lyvll large (or small).
In this case, the dynamics can be written as the following:
[V (x WE Ex Ex [f az f/ [x,l Ex [
V.G. = Ex [fr (5)
where V1 and Vy are constant, O: = Vi" Vk. 3x V7 and residue rı = V Vifi CRC.
5.1 WEIGHT MAGNITUDE
From Eqn. 5 we know that for both ReLU and linear network (since fa(() - zu =x f(
1 dd dlllwll? w = Ex [fur]va]
(6)
2 di
When there is only single output, r is scalar and Eqn. 6 is simply an inner product between the residue and the activation of node k over the batch. So if the node k has activation which aligns well with the residual, the inner product is larger and || ||wx/|| grows faster.
5.2 ANGLES BETWEEN TEACHER AND STUDENT WEIGHTS
Note that onnly tell that the weight norm would increase, but didn't tell whether wy converges any teacher node w3. It could be the case that [/wlll goes up but doesn't 't move towards the teacher. To see that, let's check the quantity:
5x [
Ex [f-134226[[ wj =Gk,W]
where Gky = Fex = [[ Putting in another way, we want to check the spectrum property of the PSD matrix Gky. Intuitively, the direction of Ex [f-14*f]] should lie between wi and wf. and the magnitude large when Wa and wf are close to each other. This means that if is dominated by a teacher (i.e., ||v]]] || large), then Wa would push Wj towards wj. This also shows that SGD will first try fitting strong teacher nodes, then weak teacher nodes.
Theorem confirms this intuition f follows spherical symmetric distribution (e,...((O, I)).
Theorem
6. If f follows spherical symmetric distribution, then Ex [f]]1= x [w] ||l]wall [ 0)w + sin Owa], where 0 is the angle between wi and Wa-
2
As result, for all 0 E [0, TT), Ex [fi-1 is always between w and wa since - 0 and sin 0 are always non-negative. Without such symmetry, we assume the following holds:
Assumption 2. Ex [f (0,x)) + (((0,k)w W where ((T)) 0.
Note that critical point analysis is applicable to any batch size, including 1. On the other hand, Assumption 2 holls when moderately large batchsize leads to a decent estimation of the terms.
With this assumption, we can write the dynamics W |/ where the time-varying residue ra of node & defined as the following (v is scalar related to v():
rk Σ agnb(0))))
Σ 3enD(Okx) DW6
(8)
5.3 SYMMETRIC BREAKING, WINNERS-TAKE-ALL AND FOCUS SHIFTING We could show that for two nodes ie # K. regardless of the form ry. we have (note that w the length-normalized version of w):
7

---

Under review as a conference paper at ICLR 2020
00m for out weghts
1x, lo6sm0.00
2x, loss=0.00
5x, loss=0.000
norm of tor out wogghts
10x, loss=0.00
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
100
0.0
0.2
0.4
0.6
0.8
1.0
Figure 3: Student specialization of 2- 2-layered network with 10 teacher nodes and 1x/2x/5x/10x student nodes, p teacher polarity factor (Eqn. For a student node k2 we plot its normalized correlation (in terms of activation vector evaluated in separate evaluation set) to its best correlated teacher as the coordinate and the fan-out weight norm |v&|l as the coordinate. We plot results from 32 random seed. Student nodes of different seeds are in different color. An un-specialized student node has low fan-out weight norm (Theorem
Theorem 7. For dynamics Wy ||wl||Es, we have
In Jw de w
W WITK W.EK.
We consider a special (and symmetric) case: rs = r-= w" Ex ag W. with all os >> O. where w is a joint contribution of all teacher nodes. In this case, we could show that when wrrs > w d (w[rs W r < O and vice versa. So the system provides negative feedback until wi = We de and according to Eqn. 7 the ratio between |/w&&] and ||W remains constant, after initial transition. We can also show that Wa will align with w" and every student node goes to w
However, due to Theorem 6 the net effect w can be different for different students and thus ra are different. This opens the door for complicated dynamic behavior of neural network training.
Symmetry breaking. As one example, if we add a very small delta to some node, say ri ew', Then to make d
so that to Theorem 7 [|wall/|| dt (w -w.rx = 0, we have wrrx > Wirr and thus according grows exponentially. This symmetric breaking behavior provides a potential winners-take-all mechanism, since according to Theorem 6 the coefficient of w" depends critically on the initial angle between Wa and w".
Strong teacher nodes are learned first. If ||vī|l ll is the largest among teacher nodes, then the joint w" heavily biases towards teacher j and all student nodes move towards teacher j. As a result, strong teacher learns first and is often covered by multiple co-linear students (Fig. 6 teacher-O).
Focus shifting to weak teacher nodes. The process above continues until residual along the direc- tion of w quickly shrinks and residual corresponding other teacher node (e.g., w for j' # j) becomes dominant. Since each F&k is different, student node i whose direction is closer to wj. (j' # j) will shift their focus towards wi,, as shown in the green (shift to teacher-2) and magenta (shift to teacher-5) curves in Fig. 6
Possible slow convergence to weak teacher nodes. While expected angle between two weights from initialization is TT n//2, shifting student node wa from chasing after a strong teacher node w to weaker one w could yield a large initial angle (e.g., close 7) between Wa and wy-. For example, all student nodes have been attracted to the opposite direction of a weak teacher node. In this case, the convergence can be arbitrarily slow. fact, if there is only one teacher node and the angle between teacher and student, then from Eqn. g arrive at Ó -0) (i0))inO. 0. Since 5- +
(0) sin 0 ~ (π 8)² around 0 = π. the time spent from 0 = π - c to some fu is to +oo when + 0. In this case, over-realization helps by having more student nodes that are possibly ready for shifting towards weaker teachers, and thus accelerate convergence (Fig.) 77. Alternatively, we could reinitialize those student nodes (Prakash et al. 2019).
6 EXPERIMENTS
We first verify our theoretical finding synthetic dataset. We generate the input using ( o with a = 10 and we sample 10k as training and another 10k as evaluation. For deep ReLU net- works, we regenerate the dataset after every epoch mimic infinite sample setting. The details of teacher/student construction in Appendix (Sec. 1.16 The normalized correlation between nodes is computed in terms of activation vectors evaluated
separate evaluation set.
Two layer networks. First we verify Theorem 2 and Theorem 3 in the 2-layer setting. Fig. shows student nodes correlate with different teacher nodes over time. Fig. shows for different degrees of
8

---

Under review as a conference paper at ICLR 2020
Layer
Layera
Layer
64
08
24
12
36
01
92
M
1
10
11
14
3475
Layer
Figure 4: The strength of student specialization versus their fan-out coefficients in 4 layer ReLU network. Number of hidden teacher nodes is 12-75-000-125. Student is 10x over-realized. The dataset is regenerated with the input distribution after every epoch. For node k, y-axis is / Ex (Bx((x), equivalent to the fan-out weight norm [ in 2- 2-layer case, and x-axis is its max correlation to the teachers. The lower layer learns first,
Successu Recovery Rart
p
0.75
0.50
0.25
10x
Figure 5: Success rate (over 32 trials with different random seeds) of recovery of 20 teacher nodes on 2-layer network different teacher polarity (Equ. (9) and different over-rcalization. Dotted line: successful rate after 5 epochs. Solid line: successful rate after 100 epochs.
over-realization x1x/2x/5x/00x) for nodes with weak specialization (i.e., its normalized correla- tion the most correlated teacher is low), their magnitudes of fan-out weights are small. Otherwise the nodes with strong specialization have high fan-out weights.
Deep Networks. For deep ReLU networks, observe specialization not only the lowest layer, as suggested by Theorem 4 but also at multiple hidden layers. This is shown in Fig. 4 For each student node k; the x-axis is its best normalized correlation to teacher nodes, and y-axis is / /Ex [8xx(x)) which is equivalent ||vx/]| 2-layer case. In plot, we can also see the lowest layer learns first (the "L--shape" curve was established at epoch 10), then the top layers follow.
Ablation on the effect of over-realization. To further understand the role of over-realization, we plot the average rate of teacher node that is matched with least one student node successfully (i.e., correlation > 0.95). Fig. shows that stronger teacher nodes are more likely to be matched, while weaker ones may not be explained well, in particular when the strength of the teacher nodes are polarized (p is large). Over-realized student can explain more teacher nodes, while a student with xx nodes has sufficient capacity to fit the teacher perfectly, it gets stuck despite long training.
In addition, the evaluation loss (Appendix Fig. 11 shows that over-realization yields better general- ization, in particular with large teacher node polarity (p is large), where weak teacher nodes are hard to capture. For good performance on real datasets, getting weak teacher nodes can be important.
Training Dynamics. We set up a diverse strength of teacher node by constructing the fanout weights of teacher node j as follows:
| 1/j".
where p is the teacher polarity factor that controls how strong the energy decays across different teacher nodes. = 0 means all teacher nodes are symmetric, and large p means that the strength of teacher nodes more polarized.

---

Under review as a conference paper at ICLR 2020
04
L4-
8002
Figure 6: Student specialization with teacher polarity = (Eqn. 9 Same students are represented by the same color
plots. Three rows represent three different random seeds. We see more students nodes specialize to teacher-1 first. contrast, teacher-5 was not specialized until later by a node (e.g.- magenta in the first row) that first chases after teacher-1 then shifts its focus.
Condatin of bot Swdeen tuchee
p- 0.5, logloss 6.45
p= p=1. logloss = 6.50
p=1.5. logloss = 6.50
p=2. 5.09
2.5, logless = 5.30
98-
9.8
0.8
0.8
0.0
3.5
0.5
3.5
0.5
0.4
3.4
04-
0.4
0.4
02-
5.2
0.2
300
30
150
55
300
150
30
100
Fnach
Foorh
Fouch
Foorh
Froh
Figure 7: Evolution of best student correlation to teacher over iterations. Each rainbow color represents one of the 20 teachers (blue: strongest, red: weakest). 5x over-parameteriiation on 2-layer network.
Fig. o and Fig. 7 show that many student nodes specialize to strong teacher node first. Once the strong teacher node was covered well, weaker teacher nodes are covered after many epochs.
CIFAR-10. We also experiment CIFAR-10. We first pre-train a teacher network with 64-64-64- 64 ConvNet (64 are channel sizes the hidden layers, = 5) on CIFAR-10 training set. Then the teacher network is pruned in structured manner to keep strong teacher nodes. The student is over-realized based on teacher's remaining channels.
The convergence and specialization behaviors of student network is shown in Fig. Spe- cialization happens at all layers for different degree of over-realization. Over-realization boosts student specialization, measured by mean of maximal normalized correlation Pmean meanje teacher maxj'e student f.. at each layer (f, is the normalized activation of node j over N evaluation samples), and improves generalization, evaluated on CIFAR-10 evaluation set.
CONCLUSION AND FUTURE WORK
In this paper, we student-teacher setting to analyze how (over- paraacterimmtee) deep ReLU student network trained with SGD learns from the output a teacher. When the magnitude of gra- dient per sample is small (student weights near the critical points), the teacher can be proven to be covered by (possibly multiple) students and thus the teacher network is recovered in the lowest layer. By analyzing training dynamics, also show that strong teacher node with large ||v"|| is reconstructed first, while weak teacher node is reconstructed slowly. This reveals one important rea- son why the training takes long to reconstruct all teacher weights and why generalization improves with more training. As the next step, we would like to extend our analysis to finite sample case, and analyze the training dynamics more formal way. Verifying the insights from theoretical analysis on a large dataset (e.g., ImageNet) is also the next step.
Figure 8: Mcan of the max teacher correlation Pwean with student nodes over epochs in CIFARIO. More over-realization gives better student specialization across all layers and achieves strong generalization (higher evaluation accuracy on CIFAR- evaluation set).
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXi::1710.0366; 2017.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. NewrlPS, 2019a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 242 252, Long Beach, California, USA, 09-10 Jun 2019b. PMLR. URL mir pwwss wtt.ss//v7/aaaenn-- zhul 1tm.mmml.mmmmmm
http://proceedin/
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In ICLR, 2019. URL [tLLitttps://peeeeviiee net/
Eorum??ddSSK0043C55
Benjamin Aubin, Antoine Maillard, Florent Krzakala, Nicolas Macris, Lenka Zdeborová, et al. The committee machine: Computational to statistical gaps in learning a two-layers neural network. In Advances in Neural Information Processing Systems, pp. 3223-3234, 2018.
Raef Bassily, Mikhail Belkin, and Siyuan Ma. On exponential convergence of sgd in non-convex over-parametrized learning. arXiv preprint 18/ 2018.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Advances in neural information processing systems, pp. 3036-3046, 2018.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.044: 2018.
Simon S Du, Jason D Lee, Yuandong Tian, Barnabas Poczos, and Aarti Singh. Gradient descent learns one-hidden-layer cnn: Don't be afraid of spurious local minima. ICML, 2018a.
Simon Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 2018b.
Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge Uni-
versity Press, 2001.
Jason AS Freeman and David Saad. Online learning in radial basis function networks. Neural Computation, 9()):1601-162) 1997.
E. Gardner and B Derrida. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 22(12) jun 1989. doi: 10.
1088/0305-447002211100044
URL https: org/ 10. 11w.00000011220000000000 047033000544000000
2F 2F12$2F004
Rong Ge, Jason Lee, and Tengyu Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint arXiv:1711.00500 2017.
Sebastian Goldt, Madhu S Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-studen setup. NeurIPS, 2019.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In JEEE international conference on neural networks, pp. 293-299 IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural nefworks, 2(5):359-366, 1989.

---

Under review as a conference paper at ICLR 2020
Hengyuan Hu, Rui Peng, Yu- Yu-Wing Tai, and Chi- Reu-Ke Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.032550 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in neural information processing systems, pp. 8571- 1- 8580, 2018.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.
Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. ICLR, 2019.
Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International Conference on Machine Learning, pp. 2908-2913, 2018.
Thomas Laurent and James von Brecht. The multilinear structure of relu networks. arXiv preprint 1711111712.10132. 2017.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-599860 088,605,15 1990.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. ICLR, 2018.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hoperoft. Convergent learning: Do different neural networks learn the same representations? In ICLR, 2016.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157- 8166, 2018
Chaoyue Liu and Mikhail Belkin. Mass: an accelerated stochastic method for over-parametrized learning. arXiv preprint arXiv:1810.133: 13395, 2018.
Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. On the computational efficiency of training neural networks. In Advances in neural information processing systems, pp- 855 863, 2014.
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern weepptraeetrize learning. arXiv preprint arXiv: 171.059,,,,06559, 2017.
CWH Mace and ACC Coolen. Statistical mechanical analysis of the dynamics of learning in per- ceptrons. Statistics and Computing, 8(1):55-88, 1998.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33)E77665-7777) 2018
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Pro- ceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2603-2612. JMLR. org, 2017.
Aaditya Prakash, James Storer, Dinei Florencio, and Cha Zhang. Repr: Improved training of convo- lutional filters. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recog- nition, pp. 10666-10675. 2019.
David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.
David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. In Advances in neural information processing systems, pp. 302-308, 1996.
12

---

Under review as a conference paper at ICLR 2020
Itay Safran and Ohad Shamir. Spurious local minima common in two-layer relu neural networks. arXiv preprint 1r2iv:1712.08968, 2017.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 2013.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari Morcos. Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint arXiv: /ttp//955..3400 2019.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Global optimality conditions for deep neural net- works. In International Conference on Learning Representations, 2018. URL https:// openreview wwwtforumm
7GE-CZ
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. In International Conference on Learning Representations, 2019. URL http:ttss///ppeeeeiieeee net httes//formmmidddokk YiRct?
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
13

---

Under review as a conference paper at ICLR 2020
8 APPENDIX
8.1 LEMMA
Proof. We prove by induction. When / L know that gr.(x) [x()) (x), by setting (x) (x) = Iexc and the fact that Dr(x) = -exx (no ReLU gating in the last layer), the condition holds.
Now suppose for layer 1, we have:
gx() (x) B(X))))))
(x)))((())
(10)
= Di(X)V? (x) (x)
(11)
Using
fx() = f-xx))
(12)
(x)) = -
(13)
gx)
-Di()())Wi((())
(14)
we have:
gx-1
(x
(15)
vI())((())
(16)
(x
Di-1(x)V(((xx)
(x(x) (Z (x) (x
(17)
vx1())
V1-((x)
Di-1(x)V[,,xx [x]=(()((i - (x-1(()((-
(18)
D (A)=()) (x-1(())(-1(x))
(19)
8.2
THEOREM I
Proof. By definition of SGD critical point, we know that for any batch By, Eqn.
vanishes:
W, = Ex (X)
Σ (x:: WJJ = Σ U4=0 iEB
(20)
EB
where U. = (x;; V W). Note that B, can be any subset of samples from the data distribution. Therefore, for dataset of size N. Eqn. 20 for all (18) batches, but there are only N data samples. With simple Gaussian elimination we know that for any in * iz, U, U, U.
Plug that into Eqn. 20 we know U 0 and thus for any i, U, = 0. Since U, is an outer product, the theorem follows.
Note that if Wi S which is EEEE Up|bo S then with simple Gaussian elimination for two batches B, and B2 with only sample difference, we will have for any i * i2. |0, 1b Σie,, U; Eies, Uslloe I EiER, Ui + E,es, Uillo = 2c. Plug things back in and we have Il s [2( - 1) + Ije, which is |Uslloo I 2e. f- (x; W) has the bias term, then immediately we have (x; W) Se
14

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
0E;
W
0E,
Figure 9: Proof illustration for (a) Lemma 2 (b) Lemma and (c) Theorem
8.3 COROLLARY
(x) lexc, which is constant (and thus piece-wise constant) over the entire input space. If for layer 1, Vi(x) and V/ are piece-wiss constant, then by Eqn. (rewrite here):
Proof. The base case is that (x)
Vx- (x) V
Vj (x)
(21)
since (x(x) and Dj (x) are wisce-wise constant and W.T and W are constant, we know that for layer - 1, (x) and V (x) are piece-wise wise constant. Therefore, for all 1.... L. Vi(x) and (x) are piece-wise constant.
Therefore, A:(x) and Bi(x) are piece-wise constant with respect to input x. They separate the region Rg into constant regions with boundary points in zero-measured
8.4 LEMMA
Lemma 2. Consider K ReLU activation functions fx) = o(w]x) j 1f # 0 and weights co-linear, then Ey Cgr Jy (x) 0. all Rd+11 suggests that all ej 0.
Proof. Suppose there exists some Cj * 0 so that Σ, (z]s(x) = for x. Pick a point x E 3 so that w]xx0 = 0 but all J.x00 * for j' + j. which is possible due to the distinct weight conditions. Consider e-ball Bx (x || xoll I <). We pick so that sign(w].x) for all j' # j remains the same within Bx (Fig. 9a)) Denote [j+]] the indices of activated ReLU functions in Bxy. j.
Then for all Bxg. Fgj. have:
h(x) Σ (x) Σ
(22)
r'E++
Since Bx d-dimensional object rather than a subspace, for Xy and Xo + eek B(xg. e). have
h(xo + (og) h(xx) cle +
Σ Cy(tyk)
(23)
JEB'
where e₁ is axis-aligned unit vector (1 S S d). This yields
cjW; + Σ
W = 0₂2
(24)
Plug it back to Eqn. 22 yields
b,b, + Σ byby =0
(25)
5'El++
where means that for the (augmented) d + dimensional weight:
Cj W + Σ
W = 0
(26)
J
15

---

Under review as a conference paper at ICLR 2020
However, if we pick x' xo
Wy CI
E Bxo,e n Ej. then fy(x) 0 but (
-cy = co and thus
Ex(t((x') =co; +
(27)
which is a contradiction.
8.5 LEMMA 3
Lemma 3 (Local ReLU Independence). Let be an open set. Consider K ReLU nodes fy(x) (w]x), j = -,.,..... l...., K. W * O, W * y for j # j' with any y > O.
If there exists C1, ck2 e. so that the following is true:
+ 0, VX R
(28)
and for node j. 0E, n R 0, then Cj = O.
Proof. We can apply the same logic Lemma 2 to the region R (Fig. Ib).. For any node j. since
its boundary θE, is R. we can find a similar Xp so that X₂ E dE, n R and xo & &E, for any
j' * j. We construct Bx Since R is open set, we can always find > 0 so that Bx C R
and no other boundary is in this e-ball. Following similar logic of Lemma
=
8.6 LEMMA 4
Lemma (Relation between Hyperplanes). Let w and Wy two distinct hyperplanes with [[w [ 1. Denote 03 as the angle between the two vectors w, and wy. Then there exists ūp w and uy = sin 0z...
Proof. Note that the projection of w onto w, is:
üy sin 0,3
P W
(29)
It is easy to verify that ||ا and w.u. sin 0zg.
8.7 LEMMA 5
Lemma 5 (Evidence of Data points on Misalignment). Let C. Rd be an open set. Consider K ReLU nodes (x) a dw]xx) = K. [w = 1, w, not co-linear. Then for a node with OE n R + 0 and I eo, either of the conditions holds:
(1) There exists node j' + so that sin oj S &MKe2-6/lcy /lcg/l and (by- - byl S Mac -////chh
(2) There exists E &dE O R so that for any j' + j. [w]x, > 5e/lcgl-
where Ozz is the angle between w, and wy. 8 > 0, r is the radius of a d - 1 dimensional ball contained in 0E, R, M
10ch d Mo maxxe0E,n8 ||x| and M2 2MM 5c
Proof. Define 9 3e/1cyl. For each j' * j. define Ly fx: /wJxx/ S x 0E,}. We prove by contradiction. Suppose for any j' + j. sin Oj > KM (51c- or 1by byl > M2e 26 /lapl- Otherwise the theorem already holds.
Case 1. When sin Uzz > KM -8 - holds.
From Lemma 4 we know that for any x E ÖE, if wJx = - with ay < 2
10c T0e5/M /MK, K, have x' uy 0F, and wJ.xx' = +
16

---

Under review as a conference paper at ICLR 2020
Consider d - 1-dimensional sphere B S 22 and its intersection of I3. B for j' * j. Suppose the sphere has radius r. For each 1y n B, its 1-dimensional volume is upper bounded by:
10
Vl nB) S ay Va-2(r) S MR Va-2(r)
(30)
where (r- is the d 2-dimensional volume of sphere of radius r. Intuitively, the intersection between w = 4j and B is at most a d - 2-dimensional sphere of radius r, and the "height" is most ayr
Case 2. When sin O2z S KMee /1yl but byy bbyl= > M2c1-26/ //c/ holds.
In this case, we want to show that for any x E 22. |w]x| > Øj and thus Ly n = 0. If this is not the case, then there exists x E S2 so that ]w] Oj- Then since x E aEj, we have:
[ [ 1(wy- w,) 1(wy ww) + (1/ b1)l S a
(31)
Therefore, from Cauchy inequality and triangle inequality, we have:
w ||| > ( w - |bا byb - |
(32)
From the condition, we have ||wr - w,ll 2in Pgg' $ 2snn B3 S 2K MC 2KMcc=0/l00 //cy// Then
2M MMMMKK+--//6 /ls/ 2 1(wy -w,) ) ž 1by bزو 9j > M2c--28/105 /l//1 5c/ 5c/lcyl which is equivalent to:
1-2
(33)
2MaKKe > M2 -562
(34)
which means that
M2 < 2M.MKe + 5e 25 < 2MgMKe & + 5e
(35)
for S 60- This is a contradiction. Therefore, In- = and thus V(I, n B) = Volume argument. Therefore, from the definition of M, we have V(B) Va-1(r) 2
22 a ( TV
10 V--2(() then for e S ED. we have:
10 v(B) M (r)
>
Σ
V(
(36)
8'455' in case 1
This means that there exists Xj E B BCQ, so that xj & Ty n B for any j # and j' in case 1. That
is,
wJXx|| > qi
(37)
On the other hand, for j' in case 2, the above condition holds for entire Qy and thus hold for the chosen Xj.
8.8 LEMMA
Lemma 6 (Local ReLU Independence, Noisy case). Let R. be an open set. Consider K ReLU nodes (x) = [w]] x), = 1,..., K. ||w,l| = 1, w, are not co-linear. If there exists C₁ ..... CK2 C. and
S en so that the following is true:
+alix) Se. VXER
(38)
and for node j. OEE n R + O. Then there exists node jr # j so that sin Ojf S <MKe2 and 1by- - bu < 2-2 /lcsb. where r, 5, M, M2 are defined in Lemma 5 but with r' = 5e/lcyl
Proof. Let = 5c/ley and Q xx OE, n RR, B(x, q:) S R). If situation (1) in Lemma happens then the theorem holds. Otherwise, applying Lemma 5 with I²' (x:x E R, B(x, q;) R} and there exists x, E 2g so that
[ 2 Gj 5k/lol
(39)
17

---

Under review as a conference paper at ICLR 2020
(a)
(b)
R
0Ey
RnâE,
x]
a5,
aE,
Iw x| S
8Ep
Figure 10: (a) Lemma
(b) Lemma
6
Let two points x xj = g, E R. In the following show that the three points x, and x are on the same side OE, for any j' # j. This can be achieved by checking whether (w])x))wWJxX) 0 (Fig. 10:
(w (wIx))(w3xx5) = (w]ixz) [. + 23 w)) (wJ,x)) )2 q3 [ [w
(40)
(41)
[ - w,
(42)
Since |w]. wyl < 1, it is clear that (w) x xx) (,) (w)xx 2 0. Therefore the three points x, and x are on the same side of θE for any j' + j.
Let (xx) = E, (x)(x) + c.w,x, then [x(x)] < for R. Since x +x, 2x,, we know that all terms related to w. and wy with j A will cancel out (they are in the same side of the boundary ) and thus:
4c 2 (M(x)) + M(5)) 2k(x,) 16,g, w. 1cyla; 5c
(43)
which is contradiction.
8.9 THEOREM 2
Proof. In this situation, because (x) D2 I. according to Eqn. Vi(x WT and Vx*(x) = W are independent of input x. Therefore, both A₁ and B3 are independent of input x.
From Assumption 1 since p(x) > 0 in Ro. from Theorem. [ we know that the SGD critical points gives (x) xx) [A B,f (x)) = 0. Picking node the following holds for every
node k and every Ro n Ex:
a.f (x) -]] =
(44)
Here a. is the k-th row of A A1 = la "n.,.......,' and similarly for B Note here layer index 1 1= is omitted for brevity.
For teacher j. suppose it is observed by student k. i.e., ОE3 n Eg * 0. Given all teacher and student nodes, note that co-linearity is equivalent relation, we could partition these nodes into disjoint groups. Suppose node is in group In Eqn. 44 if we combine all coefficients in group together into one term e w (with |w]]l 1), we have:
Okj
|| 11,8ek
(45)
k ('coolinnaa())
18

---

Under review as a conference paper at ICLR 2020
"At most" because from Assumption all teacher weights not co-linear. Note that co-linear() might be an empty set.
By Assumption dEj n Ro # 9 and by observation property, 2E3 n Ex * 0, we know that for R Rs n Eg, dEj R 0. Applying Lemma 3 we know that C2 0. Since okj # 0, know co-linear (j) # and there exists at least one student k that is aligned with the teacher j.
8.10 THEOREM
Proof. We basically apply the same logic as in Theorem 2 co-linear(k)). If for all k E co-linear(k), BEw || 1|2 complete.
Consider the colinear group then V&² and the proof is
Otherwise, there exists some student so that ve * O. By the condition, it is observed by some student node ko, then with the same logic we will have
Σ
BR l/ =0
(46)
('coo-lin)a(())
which is
Σ
Ve v
(47)
1/ ('Cro-ineaarkk)
Since k is observed by C students kb, h2
kol then we have:
Σ
vi || 0
(48)
M'Eco-llnear(k)
v By the condition, all the C' vectors
RC are linear independent, then we know that
Σ
Vi / =0
(49)
k ((Eco-linear(k)
8.11 COROLLARY 2
Proof. We can write contribution student nodes which are not aligned with any teacher nodes as follows:
Σ
Σ
vafx(xx)
E Σ
vxllwa|lo(w(T>)
vall
(50)
kEco-linear(s)
kEco-linear(s)
Co(w," Tx)
Σ
vel|wx/
(51)
kéco-linear(s)
where w' the unit vector that represents the common direction of the co-linear group 8. From
Theorem 3 for group that is not aligned with any teacher, Exeoo-linear()) Vk w 0 and thus the net contribution is zero.
8.12 THEOREM
Proof. In multi-layer case, Ax(x) and (x)x) are no longer constant over input x. Fortunately, thanks
the recursive definition (Eqn.) 4 which only contains inppt-indeeendentt terms (weights) and gating function, (x) and (x) are piece-wise constant function over the input Ro-
Note that Ro can be partitioned into R {R] R3 ... Rdd } and a zero-measure set. Each of them constant region for Ax(x) and B, (x). Since R2 an intersection of finite open half-planes (from k's parent nodes), RJ is still an open set.
From the condition, there exists open set R E R and student observer node k that OE n Eg R 0 (Figg (c). Let Hi and similarly Hi be the student and teacher nodes whose boundary
19

---

Under review as a conference paper at ICLR 2020
intersects with R. Therefore j fi- For other teacher/student nodes, they are linear functions within R and thus can be combined together into w.x. For all weights in H. H and w., applying Lemma 3 R. n Eg, we know that the SGD critical point a h (x) hh, (x) alignment between H and R. Let group be the one that contains all weights that are co-linear to teacher node j (note that no other teacher nodes are involved), and Ca its coefficient. Since j E Hğ-
leads to
Cs 0. Since onj R)) # O, there exists at least one student node k that is co-linear to teacher node
j.
8.13 THEOREM 5
Proof. We follow the logic of Theorem 4 Instead of applying Lemma for gradient that is not zero but bounded within we pick the student observer l and we have for P.g 0 R:
- Se
(52)
we use Lemma 6 and know that there exists node k'' * j so that sin 022 Oc and bby - 6i = ( 22-22/11 /lcyl) for any > 0. Under the observation student k. the teacher has coefficient cj Ouj-- Since all teacher weights distant each other with positive constant bu > and 0g > 0, with sufficiently small co and c I co, this node has to be a student node and the bound follows.
8.14 THEOREM
Proof. From the expression we can see that it is positive homogeneous with respect to [[w || and [|w&]]- ||. So we can assume |/w]ll ||wxlll Without loss of generality, we set up the coordinate system so that w = [ and We = [cos f, sin T... Then
Ex [f-12f]]] = Ex f1 wj
Σ
f f-IfLI w
(53)
W,,,22 FL w420
/=
cos 0' sin
pp((r)
cos costf'((0')))ee (r)d0' +
(54)
-3+0
where e is the term reflecting the asymmetry of the data distribution p(f(-1) with respect to the plane spanned by the vectors Wx and wj.
If the data distribution p(f--1) is scale invariant (rescaling the data point won't change the angular distribution), then p(0' (r) p(8") and we only need to check the angular integral:
{
cos O'
I(0)
cos (''p(('))dd
(55)
-
sin 0'
Note that cos2 = (1 + cos 20) and sin 0 cos 0 = 1/2 sin 20, so we have:
21(()
/+=
p(0') )do'
w
cos 20'
p(0')o0'
(56)
sin 20'
700
+
4
1
2n
cos 8M
8"
-
p(6' )do'
w³ +
2
28
sin 0"
2
d
(57)
= (0)w; + Io
(()
(58)
20

---

Under review as a conference paper at ICLR 2020
where 8" 20' + and
S 4+0
(0)
p(0')a )d0'
(59)
Io f.
2x
ODS 0
8"
sin 0"
p
2
do"
(60)
20
COS 8"
(2) =
d
sin 0"
2
2
0'
0'
)
+ P
0
cos O'd0'
Wk
2
2
2
2
0'
0'
}] sin d'd
7
0
w
(61)
2
2
2
2
where w is the unit vector that is perpendieular We but still in the plane spanned by W and w3. Note In is the fixed integral of unit vectors weighted by angular distribution of input data on activated half-plane Di of teacher node j.
1() rotational symmetric, then e O, p(0') = 1 then we can compute these terms analytically: Io 0, (0) = 27 (T - 0) and I2(0) = sin OwR-
2
8.15
THEOREM
Proof. Note that have:
d
2w Wk w
Wxl di
[wx 12
w wwwx w.rk
(62)
|
Therefore, we have
d
|kll = -..r
(63)
dt
and
d
||
d
In
(In
||
- In llw work w..r
(64)
dt
[|
dt
Note that we have:
d
Ws
W (1 (.-waw)rrg P. rk
de
Wk
di
|[
ra
W
w
(65)
Let h4 W.rk. We assume all h2 (positive correlation), then we have:
hy = r P rs + w.ix
||rx|1 h + wiike
(66)
dt
If = w Ex Wk. then we have:
d h4 ||r||2 h Shh
(67)
dt
where S = (E anll wxll is independent of k. So
(h2 - hw)) = ( h2) + S(w - hx) (hw hw) + hy + S)
(68)
hy by 0, then 4 (14-1w) - system enters the stable region. On the other hand, if ||r l12 ||r + e (e.g., I& has stronger teacher component), then we have:
and vice versa. This means that Eqn. 64 is zero when the
di (h+ hw/ = (hw hw) + h2 + S) +e
(69)
which is only zero when hg hy. This yields exponential growth of ||we]| compared to [[eel] n
21

---

Under review as a conference paper at ICLR 2020
p
p
p-1
1x
1255
Log
1.
40
60
80
100
40
60
80
100
80
100
1.5
p
2
p
1033
20
40
60
80
100
20
40
60
80
300
50
80
100
Iteration
Ran
Figure 11: Evaluation loss convergence curve.
8.16 DETAILS IN TEACHER/STUDEN CONSTRUCTION AND TRAINING
We construct teacher networks in the following manner. For two-layered network, the output mension C 50 and input dimension d = my no 100. For multi-layered network, 5, = my = Thp = 100
50-75-100- 125 (i.e, mi 50, m2 = 75, my = 100, my = 125, and C my n2 50). The teacher network constructed to satisfy Assumption [ each
layer, teacher filters are distinct from each other and their bias is set so that ~ 50% of the input data activate the nodes. This makes their boundary (maximally) visible in the dataset.
To train the model, we use vanilla SGD with learning rate 0.01 and batchsize 16.
8.17 ADDITIONAL FIGURES
Fig. 11 shows how the loss changes over iterations. With high teacher polarity (Eqn. 9). becomes harder to learn the weak teacher nodes and over-realization helps in getting low evaluation loss (in particular for p = 2.5).
Besides Gaussian distribution we also test on uniform distribution x U[-15, 15). For training, we sample 100k data points each epoch. Fig. 12 shows that the results on layer ReLU network 550-75-100-125 similar. Note that in multi-layer setting, Theorem 3 might not hold since it is for 2-layer so there could be un-specialized student nodes with large (x)
22

---

Under review as a conference paper at ICLR 2020
Spody
Esoch 25
Spaci 50
Epech
Bauch 100
106
88
1.0
86
02
64
05
06
15
00
02
84
10
5.
22
1/
02
Figure 12: Strength of student specialization for layer network (50-75-100-15 when each entry of the input dimension is uniform distributed in U[-15, 15). For all teacher nodes, the normalized correlations are all close to 1.0 (Pmean 2 0.998 at all layers).
23


Reviews:
[{"review_id": "HJgl0PlpFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the role of over-parametrization in the student-teacher multilayer ReLU networks. It presents a theoretical part about properties of SGD critical points for the teacher-student setting. And a heuristic and empirical part on dynamics of the SDG algorithm as a function of properties of the teacher networks. Overall, given previous literature, I do not find the presented results novel nor fundamentally very interesting and some parts are hard to understand due to missing details. I tend to vote for rejection at this point. More detailed questions, comments follow.\n\n\nIn related works:\n\n** Paragraph on \"Teacher-student/realizable setting\": The recent line of works is interesting, but the authors should be clearer about this being a very classical setting dating back several decades. The first paper I know where the teacher student setting appeared is by Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf). In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a very detailed account of many results on the setting from 80s and 90s.  \n\n** \"A line of works (Saad & Solla, 1996; 1995; Goldt et al., 2019; Freeman & Saad, 1997; Mace & Coolen, 1998) studied the dynamics from a statistical mechanics point of view, focusing on local analysis near to some critical points.\" and \"(Goldt et al., 2019) assumes Gaussian input and symmetric parameterization to analyze local structure around critical points,\" The statements that these works focus on local analysis is not correct. While some formal analysis in these works required an infinitesimally informed start toward the teacher the experiments (in particular all those in Goldt et al., 2019) are run from random initialization and these works show empirically that randomly initialized training converges exactly to the fixed points described in the analysis.\n\n** \"Local minima is Global\" paragraph: This paragraph seems to neglect the empirically observed fact (e.g. https://arxiv.org/pdf/1906.02613.pdf) that there can be global minima that generalize bad. Hence being global does not ensure good generalization. \n\n\nBody of the paper:\n\n** The authors cite: \"Previous works (Ge et al., 2017; Livni et al., 2014) show that empirically SGD does not recover the parameters of a teacher network up to permutation.\" but they fail to mention that separate line of work, e.g.  (Saad & Solla, 1996; 1995; Goldt et al., 2019) observed empirically the opposite.The different exiting works have to be reconciles and understood and that may be beyond the scope of the present work. But presenting only one side of the results is not helping.\n\n** The part on the dynamics with strong and weak directions reminds me on the results on so called \"INCREMENTAL LEARNING\" e.g. in the work:\nAndrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.\nalso later: https://arxiv.org/pdf/1809.10374.pdf and others. \nIt would be useful to understand what is the relation in more detail and comment on it. \n\n** The experimental part of the paper has numerous flaws that make it hard to be understood. For instance the authors do not specify the distribution of the input data. Some experiments are run with CIFAR and others with \"random\" data, but random in which sense? While generalization is the main focus of the paper the experimental results focus on the alignments of the teacher and students without really being clear how specifically the speed or the generalization error improves when neural networks are overparametrised. I found this information only in Fig. 8 for the test error. In Fig. 11 I do not know what are the different panels. What is the parameter p? So I do not know what to conclude from this figure .... in the first pannel the non-overparametrized loss (blue) decreases fastest. In the last pannel all curves are comparable. But this would suggest that over-parametrizatoin is not really helping which seems to go agains the rest of the conclusion in the paper.\n\n** A side remark: I note that the paper is on 10 pages and hence according to the paper call higher standards should be applied in the review process. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeVITt6KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the learning of over-parameterized neural networks in the student-teacher setting. More specifically, this paper assumes that there is a fixed teacher network providing the output for student network to learn, where the student network is typically over-parameterized (i.e., wider than teacher network). \n\nThis paper first investigates the properties of critical points of student networks in the ideal case, i.e., assuming we have infinite number of training examples. Then the results have been generalized to a practical case (the gradient is smaller than some small quantity). Moreover, this paper further studies the training dynamics via gradient flow, and proves some convergence results of GD.\n\nOverall, this paper is somewhat difficult to follow and understand. The notation system is kind of complicated and some assumptions seem to be unrealistic.  Detailed comments are as follows:\n\nIt is a little bit difficult to get insightful understandings towards the critical points of deep neural networks from the theorems provided in this paper. I would like to see clearer properties of the critical points learned by student network rather than some intermediate results. \n\nThe title is not consistent with the content of the paper. From the title of this paper looks like a characterization on the student network trained by SGD. However, throughout the paper, the authors somehow investigate the critical points under a stronger condition, i.e., all stochastic gradient is zero, rather than the widely used one, the expectation of stochastic gradient is zero. I don’t think the critical points considered in this paper can be guaranteed to be found by SGD. Besides, when analyzing the training dynamics, as provided in Section 5, the authors resort to gradient descent, because in (5) the dynamics of $W_k$ rely on the expectation of stochastic gradients.\n\nMany statements should be elaborated in detail. For example, in the paragraph before Corollary 1, why $R_l$ is a convex polytope? In Theorem 2, what’s $\\alpha_{kj}$? What’s the meaning of alignment? In the paragraph after Theorem 4, why Theorem 4 suggests a picture of bottom-up training? I believe the authors should provide a more detailed explanation.\n\nThis paper studies the over-parameterized student network, is there any condition on its width?\n\nIn Theorem 5, the assumption $\\|g_1\\|_\\infty<\\epsilon$ seems rather unrealistic, typically this bound can only hold in expectation or with high probability. Besides, why there is no condition on the sample size n in Theorem 5? It looks like Theorem 5 aims to tackle the case of finite number of training samples.\n\n-----------------------------------\nThanks for your response and revision.  The current title is clearer and the definition of SGD critical points is more accurate. The observations regarding the alignment between teacher and student networks are indeed interesting. However, I still feel that this result is somehow difficult to parse, as I am not clear why this can be interpreted as the learning of the teacher network. Therefore I would like to keep my score.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkekyRs2YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written, but I am not entirely sure of the interest of the results. \nI might accept, but would not be too disappointed if it didn't pass.\n\nA first comment is that the  alignment between student and teacher nodes is a very old problem, discussed at length in, for instance Saad&Solla, under the name \"specialisation\". Since the phenomenon is known, and already has a name, it should at least be also refereed to as such.\n\nThe result on the overlap and the \"specialisation\" of the teacher to the student presented in the paper is rigorous (though I did not completely checked the proof), and seems general enough, but it seems a bit trivial: of course if I have no or little error on all my data-points, I have overlap with the teacher, and since I'm over-parameterised and it's a ReLU network, then the alignment will be many-to-one.  More interesting would be to study alignment in the deeper case, but the authors prove it only for the lowest layer of a deep network.\n\nThe paper is mainly mathematical, and they are number of things I would find more interesting than the proof (though of course, this is a personal bias):\n- plot the overlaps layer-wise (i.e. student layer 1 vs teacher layer 1, student layer 2 vs teacher layer 2, etc.) What do they look like? That's something I would actually quite like to know!\n- the result on larger nodes being learnt first is known for online learning already in the 1990s (This is a celebrated results of Saad&Solla, though not an entirely rigorous one, and only for model data), so here the contribution is to show this for ReLU networks in particular. \n- Since ReLU networks are somewhat linear, it would be interesting to compare the results on the dynamics to plain linear networks, as in Saxe et al (e.g. https://arxiv.org/abs/1710.03667 ). Discuss similarities / differences?\n- The absence of \"specialisation\" in linear model is also a well known feature, see for instance https://arxiv.org/abs/1312.6120 https://arxiv.org/abs/1710.03667.\n\nFinally, I am a bit confused by the experiments : I did not understand which experiment is done for which data in fig. 5.6.7 (8 is for CIFAR of course) and 11. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The article studies a student-teacher setting with over-realised student ReLU networks, with results on the types of solutions and dynamics. The reviewers found the line of work interesting, but they also raised concerns about the novelty of the presented results, the description of previous works, settings and claims, and experiments. The revision clarified some of the definitions, the nature of the observations, experiments, and related works, including a change of the title. However, the reviewers still were not convinced, in particular with the interpretation of the results, and keep their original ratings. With many points that were raised in the original reviews, the article would benefit from a more thorough revision. 
Decision: Reject