reviews: 
Comments: Noting the anecdotal and empirical evidence of critical periods in both biological and artificial learning agents, the authors hypothesize that critical learning periods are a general feature of learning, and not an accident of biology or training, They investigate this hypothesis analytically via deep linear networks, which retain many characteristics of typical neural networks while being amenable to tractable analysis. They perform a variety of analyses and experiments, largely inspired by work on critical periods in biological systems, demonstrating that depth and data distribution provide information processing constraints that give rise to the critical periods.
Strengths: Well written, interesting avenue of research. Results well explained early in the text, with clear examples and clear explanations of experimental settings and technical details. Particularly nice clarity given the cross-disciplinary nature of the work.
Related work is well written, reasonably thorough (to my knowledge), and well-explained.
Nice phase portraits.
Captions are mostly well-explained and self contained! (I don't know why this is so rare, but good job)
Weaknesses: It's a bit of a tradeoff with having a clear hypothesis, but I find the case oversimplified/misframed  in the beginning and conclusion. Sure the literal biochemical explanation doesn't hold in an artificial setting, but analogies of this (reduced plasticity) could well occur. I don't think these are 'competing' hypotheses as they are framed; plasticity, imperfect optimization, etc. are *mechanisms* by which critical periods could arise in both artificial and biological systems; it doesn't tell us the reason  (inherent to learning, quirks of substrate, something else, etc.). In the conclusion "natural information processing constraints" are contrasted with biochemical processes like plasticity as an alternative hypothesis, but what is (decreased) plasticity buyt a natural information processing constraint? Demonstrating presence in artificial systems doesn't estabilsh it as a general feature of learning, except insofar as we've defined learning narrowly to be done by biological and ANNs (does naive bayes learn? does it exhibit critical periods?). It becomes sort of a circular argument about what does "learning" even mean. This commonality is actually supported by your experiments in Fig 3 recreating Guillery's. Anyway, I think the key sentences/claims (e.g. last of the abs) and results stand and are extremely interesting,  I would just like to see this framed around mechanisms or instantiations of information processing constraints, or something, rather than competing hypotheses. and maybe acknowledge more of the ambiguity of what "learning systems in general" means/ (without compromising clarity and narrative, which I believe are strong points of the paper).

Terms used in equations are not explained when introduced  in 3.1 -- I know it's from a citation and you might be short on space, but you will lose a lot of readers here and it's a shame not to continue the clear prose you've had thus far.

Around eqn 9, connection of matric completion to what you're trying to do is not clear -- be explicit about how imputing missing values is the same as / allows us to specify  task relationships with flexibility. 

While the relationship of experiments to showing dependence on depth is very clear and well-explained, the insights and relationship of experiments to data distribution is much less clear.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent