reviews: 
Comments: This paper reveals that critical learning periods are not limited to biological learners
but are fundamental to the learning itself. Through well-written experiments, the
authors show how deep linear networks display critical learning periods. They
conducted two experiments: one focused on competition in a multi-path model
(similar to our eyes), and the other focused on the effect of critical periods on transfer
learning.

The first experiment reveals that a deprivation deficit in one pathway will result in the
other pathway to winning the competition and learning all the corresponding features.
The deprivation only has an effect if applied in the early epochs. This reinforces the
critical learning period claim. The authors also show, by permanently lesioning the
dominant pathway, that the deprived pathway isn’t dead and will pick up features that
the dominant pathway has missed.

The second experiment highlights the effect of depth and data distribution on the
impact that deprivation during the critical period has. It reveals that deeper networks
are more sensitive to deprivations in the critical period, and the structure of the data
distribution also dictates the impact of the deprivation.
Strengths: 1. The paper provides a mathematical understanding of the psychological concept of the “critical learning period”.
2. The paper reveals that critical learning is not restricted to biological learners but is inherent to learning itself!
3. Theoretically strong.
4. Well-thought-out and well-delivered experiments.
5. The paper neatly highlights the pros and cons of competition between a model’s branches.
6. By introducing a permanent lesion in the dominant branch, the authors show that the deprived branch isn’t dead for the rest of the training.
7. The authors draw convincing insights from the transfer learning experiments, both low-to-high rank and high-to-low rank. In the appendix, the authors have also covered (both theoretically and experimentally) the case where two different matrices are involved.
Weaknesses: > …our analysis shows that critical periods only depend on two main factors: the depth of the model and the structure of the data
distribution, as opposed to details of the architecture and optimization problem.
1. The authors prove the dependence on depth and data distribution. But how can
they claim independence from other factors, like architecture and non-linearities,
especially when considering models other than deep linear networks?

2. Figure 2’s markers should be changed. It is difficult to differentiate between a
circle and a circle-triangle overlap. Try plus (+) and cross (x) as markers or a
straight line (|) and an oblique line (/).

3. Codebase not provided.

4. Typos:
* Page 3: had minimum the → had the minimum
* Page 6: training has only affects → training only affects

------------------------------

Overall, I am appreciative of the work done. I would like the authors to release the codebase for reproducibility, and clarify some of the concerns raised. I would be happy to revisit the rating if these concerns are addressed.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good