Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the impact of initialization settings on gradient based learning and describes a mathematical toolkit for two layer linear networks for understanding the impact of prior knowledge on learning dynamics. The paper derives the exact solutions to these dynamics in the two layer linear network regime as a function of network initialization, utilizing the Matrix Riccati solution.
strengths_and_weaknesses: The paper is overall well written and a lot of effort has been placed in making the presentation of the topic very accessible. I specifically want to point out the high quality and informativeness of the included figures.

The paper continues a longer research direction that tries to understand learning dynamics in (linear) networks. By taking into account so called "rich learning regimes" the paper covers a larger class of settings (including pretraining settings hereby), which are also practically more relevant. A core contribution my understanding is picking up a older idea (Matrix Riccati formulation for learning dynamics) and re-establishing it in the deep learning setting.
questions: None.
limitations: As you point out in your summary, "While our results broaden the class of two-layer linear network problems that can be described analytically, they remain limited and rely on a set of assumptions", the paper by itself may come across as too restricted. However, I applaud stating this fact, as it makes it clear that the paper should be better considered as a first step in a direction that potentially gives tooling to analyze more generic network types.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the training dynamics of deep (2-layer) linear networks. It derives the full gradient flow dynamics under mild conditions, in a form that is suitable for numerical evaluation. It then applies these results to study several deep learning phenomena including alignment, catastrophic forgetting, and knowledge revision.
strengths_and_weaknesses: [Score updated after authors addressed all of my questions]

The paper improves on previous works in that is presents a closed-form expression for the gradient flow of deep linear networks that works under fairly mild assumptions, and that can be readily evaluated numerically. In particular, equations (7-8) do not include any non-trivial matrix exponentials (i.e. exponentials that include non-diagonal matrices), and only involve decaying exponents (which are numerically stable).

The numerical experiments confirm the results presented in the paper. The applications of the theoretical results to alignment, catastrophic forgetting, and other effects show the promise that these results hold in shedding some light on interesting deep learning phenomena. The main limitation of this work is that it applies to deep linear networks, and it is not clear to what extent these results will extend to non-linear networks.

questions: My main comment is that I do not understand parts of the proof of Theorem 3.1, and have listed several questions below. Looking at the code, these issues seem to be resolved (in particular, SVD computes both $\tilde{U}$ and $\tilde{U}_\perp$ etc.) which makes me think that they have more to do with how the theorem and proof are presented and do not indicate an actual flaw with the proof. I will be happy to revise my score if the authors can answer these questions and clarify the presentation in the paper.

1. Line 123: "We note that if $N_i < N_o$, the left and right singular vectors are of shape $N_o \times N_i$ and $N_i \times N_i$". If $A$ is a matrix with dimensions $N_o \times N_i$, its singular value decomposition is given by $A = U S V^T$ where $U$ has dimensions $N_o \times N_o$, and $V$ has dimensions $N_i \times N_i$. In other words, the left and right singular vector matrices are square, and so I don't understand the statement in the paper. Also, the matrix $S$ has the same dimensions as $A$ and is not square. Much of the proof revolves around completing partial bases to full ones, but it is not clear to me what this means, given that the singular vectors already span their respective spaces.

2. In the proof (Appendix B.1), equation (26) implies that $e^{F t/\tau}$ depends on $V_\perp$ and $U_\perp$ through $M$. However, according to (20), $F$ itself does not depend on these matrices, and therefore its exponent cannot depend on them either. This is clearly true in eq. (24), since if we take the limit we see that this dependence on $M$ disappears. However, in going to (25) it seems the $\epsilon$ limit was exchanged with the limit in the exponential series, leading to this result. (I believe the issue here is related to the previous one, as $F$ should already depend on complete bases obtained through SVD.)
limitations: This work has no obvious societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This contribution introduces a reformulation of an existing matrix Riccati formulation of learning dynamics in linear networks.
This reformulation is numerically more stable and can be used to explain various known observations.
The analysis of the dynamics is used to enable rich learning with fast alignment using high variance weights.
Finally, the insights provided by the reformulated dynamics are used to predict the learning dynamics and amount of forgetting in the context of continual learning.
strengths_and_weaknesses: ### originality

This paper introduces a novel point of view on known dynamics.
Also, most contributions are framed in the context of relevant/related work.
However, I noticed a few sections where references could be improved:

  - The experiment with hierarchical data looks very similar to the one used in [17],
    which was introduced in (Saxe et al., 2013).
  - I believe that theorem 5.1 is more similar to the setting in [17] than the paper seems to acknowledge.
    After all, in [17,&nbsp;section 1.3] it is clearly stated that $\mathbf{W}_1 = \mathbf{R} \operatorname{diag}(\mathbf{b}) \mathbf{V}^T$ and $\mathbf{W}_2 = \mathbf{U} \operatorname{diag}(\mathbf{a}) \mathbf{R}^T$ for some orthogonal matrix $\mathbf{R}$.
    In this sense, theorem 5.1 has very similar assumptions as [17], 
    in contrast to what is claimed in section 5 of the submission.

Saxe, A., McClelland, J., & Ganguli, S. Learning hierarchical category structure in deep neural networks. In Proceedings of the 35th Annual Conference of the Cognitive Science Society, 2013.

### quality

Although I like the idea, the mathematical aspects of this paper appear to be sloppy.
There is also one critical flaw: $\mathbf{B}$ might not be invertible.
Assuming that reasonable, sufficient conditions for the invertibility of $\mathbf{B}$ can be formulated, I believe most results would be correct.
However, often the proofs are not well formulated:

  - The proof for theorem 3.1 starts from equation 5.
    Therefore, theorem 3.1 would still require assumption 2.1.
  - The proof for theorem 3.2 has an unexplained gap in it.
    After all, there are no explicit expressions for $\mathbf{W}_1(t)$ and $\mathbf{W}_2$.
  - Both theorem 3.1 and a proper proof for theorem 3.2 depend on the invertibility of $\mathbf{B}$,
    which is only guaranteed for $N_i = N_o$ according to the appendix.
  - Is Assumption 3.1 an actual assumption? To me, this implies that the results only hold if $N_i \neq N_o$, which should not be necessary if I understand it correctly.
    I think it would be better to simply state that assumption 2.1 is not needed anymore.
  - I would argue that assumption 3.3 is not necessary, because it is implied by assumption 3.2.
    After all, if $N_h < \min(N_i, N_o)$, then both $\operatorname{rank}(\mathbf{W}_1(0)) \leq N_h$ and $\operatorname{rank}(\mathbf{W}_2(0)) \leq N_h$ and therefore, $\operatorname{rank}(\mathbf{W}_2(0) \mathbf{W}_1(0)) \leq \min(\operatorname{rank}(\mathbf{W}_2(0)), \operatorname{rank}(\mathbf{W}_1(0))) \leq N_h$, which leads to a contradiction, given assumption 3.2.
  - Also, _sufficient rank_ is a term I am not familiar with.
    I believe rectangular matrices are still said to be of _full rank_ if their rank is the minimum of its dimensions.
    In this sense, Assumptions 2.4 and 3.2 are equivalent.
  - The proof of theorem 3.1 introduces a limit, that (I believe) is unnecessary.
    This leads to additional terms that should always be zero, making the entire formulation harder to understand.
  

### clarity

Overall, the paper is a nice read and figures are clearly annotated.
However, I think there are a few concepts that would benefit from further explanation:

  - I do not understand what the RSA is.
    The reference to [49] mentions multiple steps involved in the analysis.
    However, in the text it is used as something that should be straightforward to compute.
    Figure 4A seems to indicate it should be related to the covariance,
    but I do not know if this intuition would be correct.
  - I fail to understand what it means for a solution to be _rich_ or _lazy_.
    After all, _rich_ and _lazy_ are used to describe dynamics.
  - Figure 5C introduces target matrices, which are not explained in the text.
    I would have assumed that these are the targets for $\mathbf{A}^T \mathbf{A}(t)$, but then the off-diagonal elements should converge to non-zero values.

### significance

The results of this paper could be used to establish possibly interesting connections between different types of analyses that already exist.
Moreover, the expression for the dynamics might make it possible to enable rich learning when using high variance for the initial weights.
Although it can be argued that the proposed contribution is incremental, I believe that it could be valuable for the community.
On the other hand, this approach seems to be limited to networks with a single hidden layer.

---

### Minor comments
 - on line 11: the slow dynamics are described to be _step-like_, in contrast to _exponential_.
   However, the dynamics in figure 4E are still exponential (albeit after a plateau) and I would describe the dynamics in figure 5A as _sigmoidal_.
 - on line 49-50: I tend to read _This work_ as a reference to the manuscript,
   rather than to the (variety of) work listed in the previous sentence.
 - line 63: the _but_ seems to be out of place.
 - possible typo on line 77: I would have expected $\mathbf{W}_2 \in \mathbb{R}^{N_o \times N_h}$.
 - on line 120, it might be useful to explicitly mention that this is actually the _reduced_ SVD.
 - In equation 7, I believe $\mathbf{A}^{-1}$ should be $\mathbf{S}^{-1}$.
 - on line 224, the term _troughs_ is used, but I would argue that it is more common to talk about _valleys_.
 - on lines 35 and 247, the term _catastrophic interference_ is used. 
   To the best of my knowledge, this is actually called _catastrophic forgetting_, which can be considered a form of _destructive interference_.
 - on line 238, there is a reference to D, which might not belong there.
 - it would be nice to cite the conference papers rather than their pre-prints
   (e.g. [17] and [26] were presented at ICLR 2014 and 2022, respectively).

---

**update**: 
Considering the revision, the answers to my questions and further discussion with the authors significantly, my original evaluation is no longer accurate. 
Therefore, I decided to raise my score from 3 to 6 (as well as the soundness score from 1 to 4).

My main motivation for increasing the score:
 - A proper proof for the main theorem has been provided in the revision
 - My misunderstandings concerning the generalisation of the Fukuzima results were resolved in the discussion.
 - The main presentation issues were tackled in the revision or were promised in the discussion.

My main motivation for not increasing the score even further:
 - The analysis is inherently limited to (linear) networks with a single hidden layer.
questions: - Would it be possible to generalise this analysis to deeper networks?
- Can you provide sufficient conditions for the invertibility of $\mathbf{B}$?
- Could you verify that equations 7 and 8 could be simplified by using $$\mathbf{Q}\mathbf{Q}^T(t) = \mathbf{Z} \bigg[\mathbf{S}^{-1} + \mathbf{B}^T \Big(e^{\tilde{\mathbf{S}}} - \mathbf{I}\Big)\tilde{\mathbf{S}}^{-1} \mathbf{B} + \mathbf{C}^T \Big(e^{\tilde{\mathbf{S}}} - \mathbf{I}\Big)\tilde{\mathbf{S}}^{-1} \mathbf{C}\bigg]^{-1} \mathbf{Z}^T,$$ where $\mathbf{Z} = \begin{bmatrix}\tilde{\mathbf{V}} \Big(e^{\frac{1}{2} \tilde{\mathbf{S}}} \mathbf{B} + e^{-\frac{1}{2} \tilde{\mathbf{S}}} \mathbf{C} \Big) \\\\ \tilde{\mathbf{U}} \Big(e^{\frac{1}{2} \tilde{\mathbf{S}}} \mathbf{B} - e^{-\frac{1}{2} \tilde{\mathbf{S}}} \mathbf{C} \Big) \end{bmatrix},$ $\mathbf{B} = \tilde{\mathbf{V}}^T\mathbf{V} + \tilde{\mathbf{U}}^T\mathbf{U}$ and $\mathbf{C} = \tilde{\mathbf{V}}^T\mathbf{V} - \tilde{\mathbf{U}}^T\mathbf{U}$?
limitations: I think it is great that the authors added a section to the appendix to warn that they discovered a mistake in their proof for the invertibility of $\mathbf{B}$.
Apart from that, I do not think that this paper would have a direct impact on society
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper, entitled "Exact Learning Dynamics of Deep Linear Networks With Prior Knowledge", "derives exact solutions to the dynamics of learning" for deep linear networks. The authors leverage these solutions to understand how different regimes 
strengths_and_weaknesses: STRENGTHS:
1. The paper is very clearly written and motivated. This is true both of the conceptual and technical discussion. The authors should be commended on their use of clear figures and their commentary on how their expressions should be interpreted. 

2. The goal of the paper - to provide theoretical understanding and tools for studying how the specific initialization of networks affects their learning dynamics - is a worthy one. The authors make a substantiative step in this direction by overcoming a difficulty that had previously limited the development of theory. 

3. The framework the authors develop provides novel insight. For instance, it was from the theory that weight matrices initialized with large values, that are in the rich regime and have exponential loss, was discovered. Similarly, the connection between input-output correlation and forgetting in deep linear networks was made. These illustrate the potential of the framework. 

WEAKNESSES:
1. It is not clear to me why the task shown in Fig. 4A has the input and output correlation that is shown. Looking at the hierarchy, the fish are equally similar to each other as the flowers, but looking at the input correlation matrix, the fish are more similar to each other than the flowers. Similarly, I did not understand why the flowers have lower output correlation than the fish. This made understanding the results a little difficult. 

2. That the alignment dynamics are different for different sized weights is very interesting (Fig. 5). But there was not explanation as to why some tasks should see transient increases in the alignment part way through learning. Is there some intuition/explanation that can be gleaned from the framework?

3. The forgetting metric used in Fig. 6 should be defined. What does negative forgetting correspond to? Additionally, in Fig. 6B, tanh networks look like they have considerable difference in the analytical and numerical forgetting, but in Fig. 6C, this differences look quite small. Indeed, it seems like ReLU networks have larger discrepancy when looking at Fig. 6C, but that they have smaller discrepancy when looking at Fig. 6B. Is this a matter of the color scheme? Or are the two subpanels computing different things?

4. I found it a little hard to keep track of all the nice findings the authors had (possibly because there framework "bounced around" as applied to multiple different ideas). The impact of the paper would be increased if a table (or clear discussion in Sec. 7) was included that really spelled out exactly the theoretical contributions that were made. 

MINOR POINTS:
1. The referencing of figures, equations, and assumptions varies throughout the paper (eq., Eq., equation / Fig 5D, Fig. 5D, figure 5D, 5D / assumption 2.3, 2.3). These should be made consistent 

2. There is a typo in Eq. 8. It should be $S^{-1}$ and not $A^{-1}$ (this is correct in the appendix just not in the main text). 

3. The curves in Fig. 3C should be color coded/marked by their learning rate. As is, it is difficult to determine whether there is any pattern in the error of the approximation of the loss. 

4. Is the convention $-T$ for the transpose of the inverse a common one? To me it looks like the quantity is being raised to the negative $T$ power. 

5. It should be made clear in the main text that Fig. 4B-D shows the RSA computed once the network has converged. 

6. Is there a typo in Fig. 5C? Is $S_{top}$ supposed to be the same as $S_{centre}$? 

7. The example of learning a different hierarchy in the subsection of Sec. 6 "Revising structured knowledge" did not seem to fit in with the theme of the subsection (although I did think the result was interesting). I would suggest either dropping it or making it its own subsection. 
questions: I have no additional questions, beyond those asked in the Strengths and Weaknesses section above, for the authors. 
limitations: The authors do a great job making clear what the assumptions they rely on are and how they are limited.  The featuring of nonlinear networks (Fig. 6) was helpful to this regard and makes this point clear.
review_title: null

Meta-review: metareview: This paper derives the exact gradient flow dynamics in linear networks studying the impact of initialization under mild conditions so as to incorporate prior knowledge as initialization. Fukumizu's matrix Riccati formulation for learning dynamics is utilized for understanding deep linear networks. Under very mild conditions, the solution can be obtained numerically. The paper uses this solution to understand deep learning phenomena such as alignment, catastrophic forgetting and knowledge revision. In effect the authors provide the first step to provide mathematical tools for characterizing prior knowledge in neural networks. 

Reviewers agree that while the formalism in the paper is limited to "linear" networks, the insights drawn from the analysis and research direction the paper opens up will be valuable for the deep learning community and bear interesting connections to different analyses. While initial evaluations were mixed, after the author's responses, all reviewers recommended acceptance for the paper. 

confidence: Certain
Decision: Accept