Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies the phenomenon of incremental learning in several deep models. It starts with analyzing the optimization dynamics of a toy model, and showing that it follows incremental learning, a notion defined clearly in the paper. In particular, it shows that depth affects the strength of incremental learning in the sense that when the depth of the model is increased (especially when going from N=2 to N=3), the maximal initialization value with which incremental learning can occur is increased. In this sense, deeper models experience incremental learning more easily. The paper then moves on to other “deep linear“ models, including matrix sensing, one-hidden-layer quadratic neural networks and diagonal/convolutional linear neural networks, derives ODEs for the evolution of the singular values in the learned models, which is argued to also lead to incremental learning.

I would recommend a “weak accept” for this paper. The nice contributions include a clear definition of incremental learning, results showing the depth’s effect on incremental learning as well as extensions to several other models. My main question is regarding the relevance of the toy model to more realistic models, as I will discuss below, and I’d love to hear more about the authors’ thoughts on this.

Besides being linear, another important simplification of the toy model is that there is no interaction among the hidden units, which is rather crucial for ordinary neural networks. I am curious to what extent the authors think this simplification matters for incremental learning. It’s nice that similar analysis can be extended to other settings including matrix sensing, quadratic NNs and linear diagonal/convolutional NNs. But it seems that there is no theorem analogous to theorems 2 and 3 for those models, and I am curious why.

The qualitative transition from N=2 to N=3 in the toy model is interesting. Is there a more intuitive explanation for it? Also, from Figure 2, it seems hard to say whether there really is a qualitative change between N=2 and N=3.

Some other suggestions for improvement:
1. It may be helpful to somehow visualize the bounds obtained in theorems 2 and 3.
2. Some typos: 
1) “it’s” should be “its” in the last paragraph of section 2.
2) ”effect” should be “affect” in the first paragraph of section 3.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper studies the role of depth on incremental learning in several toy models for neural networks. In particular, they show that in these models, deep models require polynomially small initializations to exhibit incremental learning than shallow models. The paper is well written, and I think there are several interesting contributions.

The authors contribute analysis for non-asymptotically small initializations, and study an interesting role of depth in how small this initialization must be. Furthermore, they extend their results to several other models including matrix sensing, linear convnets, and classification.

I think nonetheless the paper suffers from a few issues. Some very important ones.

1) The authors study the role of depth on incremental learning, and exhibit how several models theoretically have this property. However, they do not study how incremental learning drives generalization. In the entire paper, the gradient flow is with respect to the *expected* loss, rather than the empirical loss. The research program of incremental learning for deep neural networks would show something like "Incremental learning exists when minimizing empirical loss", "Incremental learning and early stopping imply certain properties (like low capacity) on the resulting neural network" and "These properties imply low generalization error". However, the fact that the authors' models minimize the expected loss altogether a priori rules out the direct applicability of this result to explaining generalization. That is OK, in the sense that one could aim for these results to be modified and applied with empirical losses, and then a separate line of research could study how incremental learning bounds generalization error.
In this sense, I think the authors should take out the "how incremental learning drives generalization" since there is no study on generalization whatsoever, just how depth plays a role in incremental learning. An alternative title could be "How depth drives incremental learning." or something like that.

2) Another point is that all these models are very toy and mostly linear. That is OK again, but the introduction overclaims in this respect. The sentences "we characterize the effect of the model's depth [...] showing how deeper models allow for incremental learning in larger (realistic) initialization scales." and "Once incremental learning has been defined and characterized for the toy model, we generalize our results theoretically and empirically for larger models". This makes it seem that results apply to realistic settings, which is really far from true. I'm not expecting realistic results, this is a nascent theory, but I am expected the claims made to be validated and not misleading.

3) In section 2.2, sigma(t) for N-> \infty is undefined, and the proof for this result is missing (only for finite N appears). In particular, it is not clear if sigma(t) for N -> \infty is obtained by a) taking limit N -> \infty in the ODE of equation (8), and then finding the solution of this limiting ODE, or b) finding \sigma(t) for equation (8) on finite N and then taking limit N -> \infty of the solution. I.e. there are two potentially different ways to define \sigma(t) for N -> \infty which are solving the ODE and then taking limit or taking limit and then solving the limiting ODE. The definition of \sigma(t) for N -> \infty is completely missing so I have no way to assess the validity of this result.

A small pet peeve: when writing math, try to avoid using symbols like \forall and \exists unless you're writing a logic paper. Instead, try to write equation 2 like $$\sigma_i = w_i^N \quad \text{for all i = 1, \dots, d} $$, which reads a lot nicer. Also, avoid assigning equation numbers to equations you never reference.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: *Contributions*
This paper deals with the theoretical study of the gradient dynamics in deep neural networks. More precisely, this paper define a notion of incremental learning for a particular learning dynamics and study how the depth of the network influence it. Then, the authors show two cases where it applies: matrix sensing, quadratic neural networks and provide intuitions on how it could also apply to linear convolutional networks. 
This work leverage the framework and the proofs of Gidel et al. 2019 and Saxe et al. 2014 [1] to study the impact of depth in that sequential learning (that is the novelty of this work).

I really like the idea of studying the impact of the depth in the training dynamics. And the results (Thm2 and 3) are really interesting (but a bit hard to interpret in my opinion, see my questions)
Also, this work should make clearer that the results stated in Thm1 were already substantially presented in Saxe et al. 2013 ( eq. 17 and eq. 12 in [1])

Note that this paper is borderline regarding anonymity since it contains an acknowledgement section revealing the fundings of the authors (that could give enough information to identify the authors of this paper). 
 
*Decision*
Weak accept: The key results in this work is Theorem 2 and it’s extension to discrete case Theorem 3. they seems really interesting: when $N > 2$ in order to observe sequential learning, the dependence in the eigengap for the initialization goes from polynomial ($N=2$) to polynomial. 
Showing that result is interesting (even in this limited setting) because it theoretically shows (at least for these simple classes of problem) that deeper network perform a notion of incremental learning of components with non-prohibitively small initialization. 
However, these results are very hard to read. They could be interpreted and simplified in that purpose. I think it would greatly improve the quality of this work. 
I develop these points in the *questions* section of my review.
 
*Questions*
- The definition 1 is hard to interpret. For instance why do you need $t_i$ and $t_j$ since the functions $\sigma_i$ are increasing ? (note: the fact that these function are increasing is never mentioned in the paper but is key to talk about “incremental learning” and for your Definition 1 to make sense, since otherwise $\sigma_i$ could be “forgotten” without violating Definition 1) thus for any $t \in [t_i,t_j] $ we have $\sigma_i(t) \geq f \sigma_i^*$. Using only one time would make the definition easier to understand. 
- The result presented in theorem 2, (and 3) are hard to interpret because of the many parameters that distract the reader to the main point. The dependence in s and f would be interesting if we would like to compute these bound in practice but I think that the interest of this work is in the distinction exponential versus polynomial (in $r$). Thus even though I think that a version with s and f is worth being in the appendix, a version with $s = f = ½$ would make the result statement and the discussion way clearer. (other question why restrict yourself to $s \in (0,½)$ and $f \in (¾,1)$?)
- In the theorem 3 who is $c$ ? who is $\sigma_1$ (the largest eigenvalue?) ?
- In theorem 3, I am very surprised that there is no notion of eigengap that restrict the size of $\eta$. for instance let us consider the three eigenvalues $\{2,2-\epsilon,1\}$ with epsilon very small. I think that this condition is implicitly appearing in A and B. Actually for a fixed $c$, if we do $\sigma_j^* \to \sigma_1^* = \sigma_i^*$ then we got $A = 1/B $ thus one of them is smaller than 1. 
- You restrict yourself to a uniform initialization. Could you extend your results (and definitions) to non-uniform initialization (particularly initialization where  
- Very small initialization is a big issue in practice because it induces very small gradient at the beginning of the training 
 
- Figure 2, why is the time to learn those components increasing ? (i guess it is because of the $w^(2-2/N)$ that get smaller as $N$ increases. Isn’t it an issue in practice ? What is the sensitivity to the step size? in the discrete size (i.e. can we increase the step size to compensate the slower learning)? 
 
*Minor remark*
- Saxe et al. 2013 as been accepted to ICLR in 2014 (would be better to cite the ICLR proceedings) see [1]
 
[1] Saxe et al. 2014 in ICLR url: https://openreview.net/forum?id=_wzZwKpTDF_9C


=== After rebuttal === 
I have read the authors' response.
I think the authors could have discussed more the interpretation of Theorem 3. in the revision. 
But, I really like the main takeaway which is that there is a huge discrepancy between 2 and 3 layers in terms of dynamics.  

I maintain my weak accept

review_title: Official Blind Review #2

Meta-review: The paper studies the role of depth on incremental learning, defined as a favorable learning regime in which one searches through the hypothesis space in increasing order of complexity. Specifically, it establishes a dynamical depth separation result, whereby shallow models require exponetially smaller initializations than deep ones in order to operate in the incremental learning regime. 

Despite some concerns shared amongst reviewers about the significance of these results to explain realistic deep models (that exhibit nonlinear behavior as well as interactions between neurons) and some remarks about the precision of some claims, the overall consensus -- also shared by the AC -- is that this paper puts forward an interesting phenomenon that will likely spark future research in this important direction. The AC thus recommends acceptance. 
Decision: Accept (Poster)