Critical Learning Periods Emerge Even in Deep Linear Networks
Authors: Michael Kleinman
Alessandro Achille
Stefano Soatto
Venue: ICLR 2024 spotlight (2024)
Keywords: critical learning periods
deep neural networks
gradient descent
linear networks
Primary area: applications to neuroscience & cognitive science

Abstract: Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. 
Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology.
Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.

Published as a conference paper at ICLR 2024
CRITICAL LEARNING PERIODS EMERGE EVEN IN DEEP LINEAR NETWORKS
Michael Kleinman'
Alessandro Achille
Stefano Soatto
Stanford University "Caltech
SUCLA
mk www.enmannsttaneeee aachille@caltech.cm soatto@ucla. edu
ABSTRACT
Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems, This suggests that critical periods may be fundamental learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks
extend our analysis to multi-task learning to show
INTRODUCTION
Critical learning periods are time periods early development where temporary sensory deficits can permanently damage the outcome of learning. In biology, critical periods have been studied systematically since Hubel and Wiesel analyzed visual development in kittens (Wiesel & Hubel [963). Critical learning periods have since been shown to exist across different learning skills (vision and language), different species (kittens, dogs, and humans) and different sensory modalities (visual and auditory) (Kandel et al. 2013).
The most widely accepted explanation for the existence of critical learning period phenomena has to do with the characteristics of biological hardware: As the brain ages, biochemical processes decrease neural plasticity, making it increasingly difficult to form new synaptic connections and to learn new skills (Hensch) (004). Surprisingly, phenomena analogous to critical learning periods have been empirically observed for artificial deep neural networks (DNNs) (Achille et al. 2019 Kleinman et al., 2023), suggesting critical learning periods may be a more general feature of agents learning, and not caused directly by biologically ascribed factors like changing plasticity or inhibition.
Since the biochemical explanation does not hold for artificial systems, it is unclear what may cause them: One possibility is that critical periods in artificial DNNs could be due particularities of the optimization (e.g., an annealing learning rate); alternatively it could arise from defects in the artificial implementation and training (e.g., ReLUU units becoming frozen or gradients vanishing). If that were the case, it would be difficult to argue the connection with biological systems, which would not have these issues.
In this paper, we establish for the first time that critical periods can exist in a minimal analytical model of deep networks: deep linear networks (which do not suffer from any of the above). Deep linear
"ooee available at: https::/githuu..eee
Careaan/rriiii Peri odweepll nea nearreettttttt

---

Published as a conference paper at ICLR 2024
networks (Saxe al. 2013) are deep networks without non-linearitie between layers. We consider two related, but distinct cases, of overparametrized deep linear networks that capture distinct critical periods phenomena of interest. In Sect. 3 we consider multipathway deep linear network (Shi et al., 2022) to study how competition between pathways affect how features get learned. In Sect. we consider the setting of matrix completion using deep linear network parameterization (Gunasekar ict al. 2017; [Arora et al. 2019)) to study how generalization is impacted by initially training using different data distribution. We show that both settings are analytically tractable, with differential equations that characterize learning dynamics underlying such phenomena.
The multi- multi- model parameterization allows us simulate different competing ways to explain the data. Biological systems often exhibit critical periods that depend on a complex interaction between sensors. For example, in classical experiment, Hubel and Wiesel showed that occluding one eye early in development leads to permanent loss of vision in that eye (Wiesel & Hubel| 1963). However, if a part of the retina of the uncovered eye damaged, the other eye still learns to exploit the limited information (Guillery 1972). In our model, we are able to reproduce similar critical between sensors as observed in biological models. We show analytically and in simulation that the learning of features is tied to competition between sources. The learning of the singular values occurs in a "race", similar to a winner sinner-take--ll structure, with both pathways competing produce the output and the competition becoming more pronounced as the depth of the network is increased.
periods and competition/imiibite
The matrix completion setting allows natural notion of generalization, well well defined notion of tasks, their complexity, and their relationships. We show that pre-training on certain tasks can damage the transfer performance on new tasks. This occurs if brittle features from the initial task can sufficiently explain the data on the final task, and this effect becomes more pronounced in deeper networks. Such complex interaction is, again, not function of the complex architecture, or the complex optimization or implementation, but is manifest even in tractable deep linear networks.
Overall, our analysis shows that critical periods in deep networks depend primarily on two main factors: the depth of the model and the structure of the data distribution, opposed to details of the architecture and optimization problem. This level of abstraction allows us to establish a strong correspondence with biological systems. From a neuroscience perspective, our analysis provides an alternative explanation of critical periods that does not hinge on biochemical changes in plasticity, but is rather fundamental to learning, observed in radically different embodiments. Our analysis and the empirical evidence we uncover in the tractable deep linear network setting may also provide tools for deep learning practitioners better understand transfer learning and multimodal learning.
2 RELATED WORK
Critical learning periods in humans and other animals. Critical learning periods are time windows early in development where temporary sensory deficits can permanently impair behavior and alter learned representations (Wiesel 1982; Kandel et al. 2013 Knudsen 2004). In biology, critical periods have been studied systematically since Hubel and Wiesel analyzed visual development in kittens (Wiesel & Hubell 1963) Hubel & Wiesel 1970) and have since been shown to exist across different learning skills (vision and language), different species (kittens, dogs, and humans) and many different sensory modalities (visual, auditory, and motor) (Kral 20133 Kandel et al.) 2013). While critical learning periods are typically studied by altering the sensory information an animal is exposed to early during development, these learning periods also provides system with the ability to flexibly adapt to their particular environment (Kandel et al. 2013).
Critical periods in artificial networks. Achille et al. (2019) found that deep neural networks exposed to blurred images early during training exhibited phenomena analogous to animals exposed to similar deficit. More recently, [Kliinann et al.] (2023) found that DNNs also had critical learning periods for multisensory integration, with deficits early in training affecting both the learned representations and behaviour. Also related, Golatkar et al. (2019) found that regularization applied during this early period of training had the most significant effect on generalization performance.
Learning dynamics in deep linear networks. A deep linear network is a deep neural network, with identity for the activation function. Although the onput- map is linear, the learning dynamics are nonlinear (Saxe et al., 2013). Moreover, it is possible to obtain exact learning dynamics based on how the network will learn task structure (Saxe al. 2013), and such networks have provided insight
2

---

Published as a conference paper at ICLR 2024
into semantic development in humans (Saxe al. 2019). Recently, such models have been extended to the multi-pathway setting, where it was shown that deeper networks are increasingly likely to learn features of either pathway (but not share on both) (Shi et al. 2022). It is also possible to add gating to deep linear networks to alter the flow of information, and this can allow nonlinear computation. (Saxe al., 2022). Here, we incorporate both gating and multi-pathway extension to study critical learning periods in deep linear networks.
Matrix completion, deep matrix factorization, and the implicit regularization of SGD. Matrix completion is general problem of imputing missing values in a matrix, given some observed entries. This setting is common, and often seen recommendation systems where only fraction of recommendation (or ratings) are known. If the ground-trutie matrix is low-rank (often the case for real-world data), missing values can recovered given sufficient number of observed entries by minimizing over matrices that match the observed entries and have minimum nuclear norm (Candes & Recht 2012). Gunasekar et al. (2017) took different perspective on the matrix completion problem, and empirically found that parametrizing the target matrix using two layer linear neural network, that W WW2W1, and optimizing over the weights of this factorization using gradient descent with small initialization lead to matrices that had minimum nuclear norm, suggesting that the implicit bias of gradient descent was to find low nuclear norm solutions. Afterwards, Arora et al. (2019) refined the results, finding that gradient descent starting from small initialization implicitly minimizing the rank of the matrix, as opposed to the nuclear norm, and this effect is increasingly more pronounced in deeper networks. We use the matrix completion setup to study how generalization is affected if network did not start from small initializations, but rather was initially trained on another task.
IMPACT OF DEPTH AND TEMPORARY DEPRIVATION ON FEATURE LEARNING IN MULTI-PATH MODEL
Inputs are rarely processed by a single processing stream. For example, humans and other animals have two eyes that process visual information coming from a scene. In a series of systematic biological experiments, researchers discovered that kittens with single eye occluded early during development were more affected than kittens with both eyes occluded during the same period. This suggests complex dynamics are mediating learning from multiple sensory modalities. Here, we analyze how features get learned a minimal multi- multi-patwhway model, stripped of nonlinearities. In particular, we study what happens if pathway becomes temporarily blocked, such as from suturing a eye. Surprisingly, as we will see, this minimal and analytically tractable model captures much of the learning dynamics in biological experiments.
3.1 LINEAR
FRAMEWORK
We consider a multipath linear network (Shi et al. 2022) where the output y is produced by propagat-
ing an input through multiple pathways P (a, b} follows:
WD W wh ... w
wD pEP
w x= s2x.
(2)
where p denotes the th weight matrix along pathway p. We will focus on the case where |P| = Dr
pathways. As in Shi al. 2022 we have defined s2p = III=r
so Wd ps
Dp
Ω Q.
Σ S2 PEP
II[
w pEPd=1
The input x thus gets propagated through multiple deep pathways of depth Dp. with each pathway consisting of a series of linear transformations T2e W We consider the case of minimizing the squared error loss
N
L = 2
lly
(1)
112
(4)
with training set of i.i.d samples D ((((0 , To simplify even further, assume that the inputs have been whitened, that the input correlation matrix Σ N x000 =I
3

---

Published as a conference paper at ICLR 2024
10
2=2
0-5
10
D=8 D-113
Deficit
10
Pot. Siigeat volee Mal
Late: Deficitt
Futh A Serptar Valeo Med
Figure 1: Deeper networks are more affected by temporary early deficit. Phase portrait of pathway specific singular value Kan and Kbn where Kan + Kiv Oa. where da 10 is the singular value and denoted by the diagonal blue line in the plots. Black traces indicate simulation for a particular initial condition (100 shown). Flow fields (purple arrows) are shown for systems without deficit. (Top row) An early deficit to one pathway ("B") leads to the other pathway learning more of the feature, and this effect becomes increasingly more pronounced in deeper networks (shown for depths ranging from 2 to 13). (Bottom row) contrast, late deficits have a negligible effect on the final solution, where neither pathway dominates (on average) how feature is learned.
Let EEE = Ei= x00 x()T be the cross-correlation matrix between the inputs and the target vector y and let g USVT be its singular-valus decomposition (SVD). The loss will be minimized when y = Ω and hence when Ω U RV = S, the network has learned the tapk-approe singular values. We define K, = U Ω. as the pathway specific contribution to the singular values for path (the contribution of both pathways K, + K,, sum to S at convergence).
Using the continuous time limit of the SGD update equation leads the following differential equation for pathway (and analogously for pathway 6):
D. W:
W
II i=d+1
T ENE
PE
w.:)
T
(5)
dt
Assuming Wd Rd+1 Rd where an orthogonal matrix and R = and R. U
get
d
( III w:) (S -2)) d-1 [I::.
Da
W dt
(6)
i=d+1
and note that if all weight matrices W are diagonal initialization, then since S, Z are diagonal, we can arrive at system of scalar differential equations, where for each singular value S.
d D. Poa IS. Stxl
dt qua qua
Da-1 1 S.a - -1a] d Vaa
di Pan
where qaa reflects the scale of diagonal entries of the intermediary matrices (d < Da) while Poo reflects the scale of diagonal entries of the final matrix (d = D.). Note further that the ratio of Eq. will be constant such that:
dgan
Pao
(8)
dpaa
qaa
80 9² 1--p" p2 ==(0²² -p(0)2 and this difference will stay constant during training.
4

---

Published as a conference paper at ICLR 2024
Early Deficit
Middle Deficit
Late Deficit
2.0
2.0
50E
00E
5GD Kny
SGD Ke.
500 Kec
3
SGD Kau
1.5
5GD
1.5
500
Path Singuar
2-
1.0
1.0
1
0.5
0.5
0.0
0.0
200
400
600
800
1000
0
200
400
600
800
1000
0
200
400 600
800
1000
Epoch
Epoch
Epoch
Figure 2: Early deficits affect learned representations in multi-path model, while late deficits do not. (Left) Early gating deficit (denoted by gray period; epoch O to 150) to pathway B leads to all features being learned in pathway A. When training with SGD, we indicate the singular values for pathway A with crosses ('X'), and pathway B with plus signs ('+',, with the different colors identifying different singular values. In dashed lines, we show the results of integrating the ODE (Fq. []) for each singular value for both pathways. We observe a match between the differential equation and the learning dynamics obtained with SGD training. We observe sigmoidal learning trajectories of the singular values, with larger singular values learned earlier in training. (Middle) Deficits applied the middle of training only affects previously unlearned features (bottom four singular modes), whereas other singular values are learned equally in both pathways (crosses and plus signs overlapping for blue, orange, red, green singular modes). (Right) Late deficits (epoch 750 to 900) has negligible effect on how features are learned, and results in features being learned equally in both pathways (crosses and plus signs overlapping for all singular values).
3.2 LEARNING DYNAMICS IN REDUCED SCALAR DIFFERENT EQUATION HIGHLIGHT EFFECT OF COMPETITION
To better understand how competition between processing pathways mediate learning, we introduce deprivation deficits during different windows of training where we temporarily prevent learning (parameter updates) in a deprived pathway. This deficit can also be interpreted as blocking input information from being processed by the deprived pathway (see Appendix A.I for details), and refer to the deficit as gating deficit.
Using the mullt--pathwy setup described above, we integrate the differential equation of Eq. 7 and plot the corresponding flow fields in Fig. [ To integrate the differential equation, we use a step size of A = + 0.001 discrete time for 1000 epochs ensure convergence. To better understand how temporary deficits impact how features get learned, we applied a gating deficit early (first 15 epochs) and late (epoch 100 to 115) in training to one pathway ("B"). In this way, parameter values along the deprived pathway remain constant during the deficit.
We initialize each pathway independently with p(0) (o,00.001)) and -0)))~p2(0)
in Fig. and also observe similar trends if p(0) = and (() = corresponding to a small initialization in Appendix Fig- 7 even though the deficit is only applied during the initial phase of a sigmoidal learning trajectory. We find that competition becomes more pronounced at greater depths; in this manner temporary deficits (of a fixed number of epochs) alters the learning of features more in deeper networks (Fig. [T. Further, only gating deficits early during training affect how features get learned; gating deficits applied late in training do not alter the features that get learned by each pathway (Fig. bottom row).
3.3 DEEP
LINEAR NEURAL NETWORK SIMULATIONS
We next verified that the results from integrating the differential equation corresponded to real learning
motifs by training a deep multi- matht-p linear network on a previously studied hierarchical task (Shi
al.) 2022) (Sec Appendix I. I. We trained depth D, D, = 4 network, with 100 units per layer of each pathway using SGD with a constant learning rate of 0.01 using the squared error loss (Eq. 4).
Again, we find that early deficits affect learned representations, while late deficits do not (Fig.
In particular, an early gating deficit (epoch 0 to 150; left) to pathway "B" leads to all features being learned in the other pathway (*A").
This can be better understood by examining the learned singular values for both pathways, which are only learned by the normal pathway (denoted by crosses for SGD simualtions) when the deficit is
5

---

Published as a conference paper at ICLR 2024
Gating and feature lesioning deficit
out
Kg
Path Singuer Value
3
2-
-2
2000
400
600
800
1000
Epoch
Figure 3: We can reproduce and better understand classical experiments showing competition between eyes using our multi-path framework. Applying a deprivation deficit to one pathway (analogous to eye suture; in this case pathway to "B") for first 200 epochs, while permanently lesioning a feature in the other pathway (second singular mode of pathway "A") leads to the initially sutured pathway only learning the corresponding lesioned feature (orange plus signs, and corresponding singular value in Ks). Integration of the ODE (Eq. 7 for each singular value matches simulations with SGD training before, during, and after the deficit (shown in dashed lines). Singular value dimensions 1 to 5 shown for improved readability. This highlights how the pathways compete to learn the input features, and shows that a deprivation deficit to one pathway will result in the other pathway "winning the competion" and learning the corresponding feature This experiment recapitulates classical experiments by Guillery that lesioned a local region of the normal eye during monocular deprivation experiment, which was intended for studying how competition between eyes/pathways affect how visual features get learned (Guillery 1972).
applied early during training. This early deficit led to the normal pathway "winning the competition" to learn the singular values and explain the output. In contrast, deficits during the middle of training (Fig. [ center) only affects singular values that were not previously learned (bottom singular modes). The late deficit (Fig. 2 right) does not affect the previously learned features. The learning of the singular values has a particular sigmoidal learning trajectories, where the network learns the singular value in order of their magnitude (higher singular values learned earlier training), as we will discuss more in the next section, and in line with prior work (Saxe et al. 20133 Arora et al. 2019). We also observe a match between the differential equation and the learning dynamics obtained with SGD training and integrating the ODE of Eq. 7 for both pathways before, during, and after the deficit period (Fig. 2). We observe similar learning dynamics and effect of a temporary gating deficit in nonlinear networks with Tanh or ReLU activation function (Fig. 11 and Fig. 12 respectively).
To better understand how competition and inhibition between sensors affects how features are learned, applied a gating deficit one pathway (analogous eye suture), while permanently lesioning feature in the other pathway (a singular mode). Without the lesioning, the normal pathway learned all the features (Fig. 2). and in this case it resulted in the initially deprived pathway only learning the corresponding lesioned feature (here, the second singular mode, orange plus signs in Fig. 3). This experiment recapitulates classical experiments by Guillery which a local lesion was made in the normal eye during monocular deprivation experiment, and found that the initially deprived eye would only learn visual information corresponding to the lesioned area. (Guillery 1972). In particular, this experiment highlights that multi-pan patti-p deep linear network captures the competitive learning dynamics that have been empirically observed in many animal studies.
4 CRITICAL LEARNING PERIODS FOR MATRIX COMPLETION: GENERALIZATION IN DEEP LINEAR NETWORKS
In the previous section, we explored how competition between pathways affects how features are learned, finding that deficits early during training affect the learning of all features, while deficits later training only affect features that have not been learned earlier training. However, in the deep multi- multi-pathway network, the network always learned the same global input-output mapping, though the network learned to represent features differently across the different pathways depending on the onset of the deficit during training. The previous setup did not allow for a natural notion of generalization and an understanding of how generalization depends on the relationship between the data distribution during and after the deficit period.
To better understand generalization using minimal and tractable models, we turn to a matrix comple- tion framework. Matrix completion is general problem of imputing missing values in matrix, given

---

Published as a conference paper at ICLR 2024
Effect of Depth (Subset Obs.)
Effect of Depth (Rank 10 Rank 5)
0.00
Rel. Recon. Emror
0.0
Rel. Recon Error
-0.25
-02
D-
-0.4
D= 2
-0.75
D-3 3
-06
-1.00
25000 50000 75000 100000
25000 50000 75000 100000
Deficit Removal (Epoch)
Deficit Removal (Epoch)
Figure 4: Deeper networks have more pronounced critical periods. (Left) Effect of variable depth parameterization
reconstruction error when observing partial observation of the final task (1000 out of NV entries) during the deficit period before the task switches and training continues by training all N observed entries. N 1500. We computed the relative error by comparing the reconstruction error with a network that started training from the random initialization. depth-1 parameterization is not affected (blue line), whereas deeper architectures sufficiently long initial deficit. Absolute reconstruction errоr was 0.864, 0.320 and 0.118 for the depth 1, network trained from random initialization. (Right) Effect of variable depth parameterization when going from rank 10 matrix completion task to rank 5 matrix completion task. In this case, for a depth-1 parametrization, no effect is observed, whereas deeper networks are increasingly sensitive to perturbations early during training. Absolute reconstruction error was 0.795. 0.053 and 0.0001 for the depth 1, 2, 3 network trained from random initialization.
more affected by
some number of observed entries of the ground-truth matrix. The matrix completion setup, as we will see, is useful for studying critical learning period phenomena because allows to explicitly specify the relationship between tasks (during and after the deficit period) and allows flexibility for the type of deficit that can be applied. For matrix completion, given observed entries (Mr, (i,j) E Ω of unknown ground-truth matrix M, the challenge is to optimize loss over a training set
1 Σ (Mrs - W L(W) = 2
(n))
and generalize to the entries unobserved locations. Typically the ground-truth matrix M is assumed to be low rank to make the problem tractable.
Similar to the previous section, we parametrize the matrix W using a deep linear neural network so what W = W WDWp-i ... Wi, where D refers to the depth of the parametrization and run gradient descent over this loverrparametrization. This setup has been used to study the implicit bias of SGD in tractable setting (Gunasekar et al. 2017 Arora et al. 2019).
we also obtain exact differential equations that charac- terize the evolution of the singular values (Eq.
As we elaborate in Sec. 4.2 and Appendix B
10) and singular vectors during and after the deficit period (Eqs 13 14p for matrix completion using a deep linear network parameterization.
4.1 EXPERIMENTAL DETAILS
In line with previous work (Arora et al. 2019), we initialize components by setting the standard deviation for each parameter in the deep matrix factorization to be - b where g sets the initial scale, and N refers to the number of columns (or rows) of the square matrices W.. This allows the Frobenius norm of the overall product matrix to be independent of the depth of the factorization. We consider ground- -ruth matrices of size 100 x 100, and matrices of the same size for all [W.),2 in the deep linear network parametrization. We set the number of observed entries to be 2000 and sample the same observations during the pre-training and final matrix completion task, unless otherwise stated for experiment. We trained with SGD with constant learning rate of 0.2 by using batch gradient descent and minimizing the loss in Eq. averaged over observed entries.
Whereas previous work have studied matrix completion using SGD from small initializations (Gu- nasekar et al. 2017 Arora et al.] 2019), we explore the case of transfer learning, where prior analyze how pre-training
knowledge is embedded in the learned parameters. In particular one task (which we refer to as a deficit period) will affect generalization new task. We trained networks for variable duration on the first task, and then subsequently trained the network for
7

---

Published as a conference paper at ICLR 2024
Effect of Rank (Task 1)
Num. Observations (Final Task)
Num Observations
1.6
Rocon. Erar
0.5
R- 2
Re.
R- 15
Recon, Eitor
0.25
0.75
1,00
N- 1250
NF 1530
1750
2000
Recon Emor
N- 4000
N- 1500
2000
1000
25000 50000 75000 100000
50030
100003
25000 50000
75000 1000000
Daficit Removal (Epoch)
Deficit Removal (Epach)
Deficit Removal (Epoch)
Figure 5: Dependency on matrix rank and number of observed matrix entries. (Left) In the matrix completion setup, sensitivity to an initial deficit occurs when the initial matrix completion task higher rank than the final matrix completion task (rank 5). Pretraining on a rank 2 or rank task does not adversely affect reconstruction error for the final matrix completion task (overlapping in the plot). (Center) Effect of partial observation of the final task (1000 out of N) during the deficit period before the task switches, and all N observations as present. Generalization error as a function of N in the final task. We observe a deficit when the number of samples in the final task are small, and "close" to the initial training task (NV = 1250 and N = 1500). (Right) Sensitivity to initial training task as a function of total number of random observations N for both tasks. The initial task is a rank 10 matrix completion task, and the final task is rank matrix completion task. The sensitivity change in task is most prevalent in low-sample regimes, where there are a variety of ways for fitting the observations, and is less impacted by the initialization coming from the initial learning. When the observations are large (N = 3000, or N = 4000), we do not observe sensitivity to the initial task (overlapping in plot).
the
fixed number of epochs of the final task (30000 additional epochs). On experiments where we vary the rank of the task during the initial training period, we set the rank of the first task to be 10, and the rank of the final task to be 5 by construction, unless otherwise stated.
4.2 RESULTS: IMPACT OF PRE-TRAINING ON GENERALIZATION FOR MATRIX COMPLETION
We first examine how the depth of the deep linear network parametrization in the matrix completion setup alters generalization on the final task, as a function of pre-trainin on an initial different task (Fig. 4.. First, find that deeper architectures subset of the entries during an initial deficit period (Fig. left). We also find that deeper networks are more affected by the deficit of pre-training on different higher rank initial task (Fig. 4 right). We do not observe any sensitivity in depth D = 1 parametrization, as only components corresponding to observed entries get updated during both the pre-training and final task, and hence generalization to the unobserved entries will not be affected (being equivalently poor). Our finding that deeper architectures are more affected by an initially training on a related, corrupted dataset, consistent with previous empirical results AfcCclle et al. (2019) and Kleinman et al. (2023) who empirically found that deeper convolutional architectures (with nonlinearities) were increasingly affected by initially training on blurred images before training on regular images.
increasingly affected by training on partial
Next we fixed the depth of the network to be D = 3, and varied the rank of the matrix during the pre-training task. We find critical learning periods when going from higher rank task lower rank final task (Fig. 4 right), but not when going from a lower to higher rank tasks. We can better understand the learning dynamics by examining the singular values during the deficit periods and normal training. Arora et al. (2019) showed that starting from small initializations, the singular values will evolve as:
dt = - (t) u, (I)(
(10)
where u, and v, and the th singular vector of the product matrix W through training. In this manner, depth D makes larger singular values increase faster, and makes smaller singular values evolve slower, and as in the previous section, the network will undergo sigmoidal learning trajectories for each singular value. In particular, we find that the ability to learn low-rank solutions relates to how well the network will generalize to unobserved entries (Fig. 6 top row), and deficits that lasted late into training prevents the learning of the low-rank solution for the final task. We also obtain closed form equations that describe how the singular vectors evolve during and after the deficit period (Eqs B [4. further described in Appendix[B] Simulating these differential equations matches gradient descent learning dynamics before and after the task switches (Fig. 9
8

---

Published as a conference paper at ICLR 2024
50-
50
52-
40-
20
10000
20000
30000
10000
20000
30000
40000
20020
40000
23000 50000
75000 100000 Epoch
20
40
12-
22
12000 33200
20:000
45000
25000 75000 100000
tpsch
Figure 6: Evolution of singular values before and after deficit as a function of the number of observed matrix entries. The dashed gray lines indicate the epoch where the task changed from a rank 10 task to a rank final task. (Top row: 2000 observations) If the deficit is applied early in training (up epoch 10000) before the network has learned many singular modes, the network will converge to a low rank solution (and generalize well). Deficits late in training after many singular modes have been learned for the initial task lead to solution that is not low rank and has generalization. (Bottom row: 4000 observations) With a large number of observations the network can eventually learn the correct final rank 5 task regardless of the initialization coming from training on the initial task. Regardless of the initialization, the network converges to a solution with exactly 5 singular values. Experiments are shown for a depth 3 network.
In the case of matrix completion, the sensitivity to the initial phase of training is most prevalent in low-sample regimes, as there are variety of ways for fitting the observed entries. When the observations are large (NV 3000, or N = 4000), we do not observe sensitivity to the initial task (Fig. [) right). With a large number of observations, the network can eventually learn the correct final rank task, evidenced by the final singular values shown in Fig. 6 bottom. We also explore how the reconstruction eтror depends on the number of observations for the final task when initially observing a subset of the observed entries in Fig. (centtr), and we observe critical learning periods in the low sample regime, where increasing duration of pre-training on partial task information increasingly impairs generalization performance for the final task.
5 DISCUSSION
The explanation most often ascribed to critical learning periods is that they emerge from factors unique to biology, such as biochemical processes that alter neural plasticity as animals age (Hensch) 2004). In this work, using minimal models that are analytically tractable, we argue that critical learning periods may be fundamental to deep learning systems and emerge as result of information processing constraints, they are even present in deep lincar networks devoid of biochemical processes (and nonlinearities).
We find that multi-patio multi-pathway deep linear networks capture the competitive dynamics between multisen- sory inputs, such as visual input coming into two eyes, and display maximum sensitivity to deficits to pathway early during learning, as such defcits affect the learning of all features. Further, the effect of such deficits are increasingly pronounced in deeper networks. We also find that generalization in the matrix completion setting is increasingly altered by longer pre-training on different task, and this effect too depends critically on the depth of the parametrizatiion. Together, our analysis highlights that critical periods depend on two main factors: the depth of the model and the structure of the data distribution, which allows us to make strong correspondence with biological systems that share these details.
Our models are minimal and analytically tractable, and yet surprisingly are still sufficient to recapit- ulate much of the critical period phenomena seen biological systems. Our work thus makes an important step towards developing mathematical understanding of critical learning periods as fundamental phenomenon common to artificial and biological learning systems that depends primarily on the data distribution and the depth of the network learning to process such information.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference Learning Representations, 2019. URL hww.hhttps//eecmm /oppas:///perrreee peppnrevvin ew
net bet/Eooum?iin Bket
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp. 244-253. PMLR, 2018.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimization. Commu- nications of the ACM, 55(6): (5(6):111-11) 11-119, 2012.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems 32, pp. 10677-10687. Curran Associates, Inc., 2019.
RW Guillery. Binocular competition the control of geniculate cell growth. Journal of Comparative Neurology. 144(1):117-12,, 1972.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
Takao K Hensch. Critical period regulation. Annu. Rev. Neurosci., 27:549-579, 2004.
David H Hubel and Torsten N Wiesel. The period of susceptibility to the physiological effects of unilateral eye closure in kittens. The Journal of physiology, 206(2):41--43,, 1970.
Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven A Siegelbaum, and A J Hudspeth. Principles of neural science. McGraw-Hill, New York, fifth edition, 2013.
Michael Kleinman, Alessandro Achille, and Stefano Soatto. Critical learning periods for multisensory integration in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24296-24305, June 2023.
Eric I Knudsen. Sensitive periods in the development of the brain and behavior. Journal of cognitive neuroscience, 16(8):1412-1425, 2004.
A Kral. Auditory critical periods: review from system's perspective. Neuroscience, 247:117-133, 2013.
Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: dynamics of abstraction in gated networks. In International Conference on Machine Learning, pp. 19287-19309.
PMLR, 2022.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 1312.613 2013.
Andrew M Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23): 11537-11546, 2019.
Jianghong Shi, Eric Todd SheaBrown, and Michael A Buice. Learning dynamies of deep linear networks with multiple pathways. In Advances in Neural Information Processing Systems, 2022.
Torsten N Wiesel. Postnatal development of the visual cortex and the influence of environment. Nature, 29955884):583-591, 1982.
Torsten N Wiesel and David H Hubel. Single-cell responses in striate cortex of kittens deprived of vision in one eye. Journal of neurophysiology, 1606):1003-1017, 1963.
10

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 MULTIPATHIWAY EXPERIMENTS
In the multi-pathway experiments, when we applied the deficit to pathway, which we refer to as "blocking" or "gating" an imput pathway in the paper, the desired target output was also shifted by a baseline amount corresponding the deprived pathway's 's output. This was to ensure that the normal pathway was only required learn the unexplained component of the output, and not the entire output. In practice we implemented this using the detach () method in Py PyTorch applied on the output of the deprived pathway during the deficit period, while having the total output the sum of the contribution of both pathways (while leaving the desired target unchanged), In particular, this deficit corresponds to a deficit where the gradient was not supplied to the deprived pathway during the deficit window.
The task we consider in Fig. and Fig. 3 is the following, with Σ- = I. Each input 8 dimensional and the output is 15 dimensional, with the input encoded one-hot vector, and the output corresponding to the columns of y (rows of ")
0
1
0
0
0
0
0
0
T
0
1
0
0
1
0
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
1
0
0
0
1
(11)
0
1
0
0
0
0
0
0
0
1
0
0 0
1
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
0
0
ADDITIONAL DETAILS FOR FIGURE 2.
We used fixed length deficit of 150 epochs starting at epochs {0, 400, 750} (early, middle, and late deficit respectively). We use batch gradient descent, with learning rate of 0.01 and squared error loss. We did not use biases in the linear networks. We are able to exactly trace the trajectories of the singular values for the task in Fig. 2 provided the following initialization from (aax et al. 2013). We initialize weights matrices such that W RDVT, W RDR" and W!.. UDR (and analogously for pathway B). We set R to be a 100 8 orthogonal matrix, D, to be a diagonal matrix, and U, V to be the singular vectors of E The scale of the entries in diagonal matrices D at initialization was set 1/D1 and used the same diagonal matrix in both pathways. We added a small amount of Gaussian noise to the diagonal entries so that equivalent singular values were learned at slightly different times. This initialization ensures that the weights are balanced across layers, and the differential equations then characterizes the learning dynamics before, during, and after the deficit period.
ADDITIONAL DETAILS FOR FIGURE 3.
We used a fixed length deficit of 200 epochs starting at epoch 0 for the gated pathway. We also lesioned the second singular mode from the otherwise normal pathway by setting the second row and column of K₂ to zero during training. We also use batch gradient descent, with a learning rate of 0.01 and squared error loss. We did not use biases in the linear networks. We used a multi-pathwe network with D2 = D3 = 3. The scale of the entries in diagonal matrices D at initialization was set to I/D and we used the same diagonal matrix in both pathways. We added a small amount of Gaussian noise to the diagonal entries so that equivalent singular values were learned at slightly different times. We initialize weights matrices such that W RDVT, W RDR"" and D.. UDR (and analogously for pathway B). We set R to be 100 x 8 orthogonal matrix.
A.2 MATRIX COMPLETION EXPERIMENTS
We constructed an x matrix M of particular rank R by creating two R x N matrix L and L' each with entries sampled from zero-mean Gaussian distribution with standard deviation 1, and then
11

---

Published as a conference paper at ICLR 2024
D=2
D=3
D=4
10
10
10
Early Deficit Singular
Path Singular (K.)
Late Deficit Singular
Singular (K.)
Figure 7: Deeper networks are more affected by a temporary early deficit, even for small initial conditions. Phase portrait of pathway specific singular value Kea and Kba where Kaa Ktcx oa where aa = 10 is the singular value and denoted by the diagonal blue line in the plots. Black traces indicate simulation for particular initial condition where (0) p(0) c, and 0.005. Flow fields (purple arrows) are shown for systems without a deficit. (Top row) An early deficit to pathway (for only 5 epochs) leads the other pathway learning more of the feature, and this effect becomes increasingly more pronounced in deeper networks (shown for depths ranging from 2 to 4). (Bottom row) In contrast, late deficits (even of 10000 epochs. but starting at epoch 500000) have a negligible effect on the final solution, where neither pathway dominates (on average) how feature is learned. Step size of 0.01 was used and simulation was run for T 1000000 epochs.
12

---

Published as a conference paper at ICLR 2024
Effect of Init scale
Dependency on random seed
1.5
1.00
0.0001
Recon. Error
0.75
Seede 1
1.0
Init= 0.001
Recon,
Init= 0.01
0.50
Seed= 3
0.5
Init= 0.1
Seedm 4
0.25
50000
25000
Deficit Removal (Epoch)
Deficit Removal (Epoch)
Figure 8: Effect of initialization scale and robustness to random initialization. (Left) We used the same parameter settings in the paper, and varied the scale of the initialization. For all initializations, we see higher reconstruction with longer pre-training on the initial task. Smaller initializations have a longer period (number of epochs) where the deficit will not impact the final solution, but have more marked final effect (e.g. blue trace). (Right) Results are robust to different random initialization seed (for initialization scale of 0.001).
|| denotes the Frobenius norm. In Fig. weee plot the top 10 singular values computed from the product matrix W Wp W p-1 ... W1 during training. In our experiments, rather than optimize the sum of all the losses as in Eq. 9 we computed the loss as the average. Reconstruction error was measured by II W (average squared error per entry). We used an initial scale = 0.01 in our experiments unless otherwise stated, where entries were drawn from a normal distribution with standard deviation = VN gb We vary this initialization in Fig. 8 All other experimental details to reproduce experiments are provided in main text.
taking M = L L. This matrix was normalized such that M M π² where ||
A.2.1 ADDITIONAL EXPERIMENTS: INITIALIZATION SCALE AND ROBUSTNESS TO RANDOM INITIALIZATION
We vary the scale of the initialization in Fig. g (left), and observe critical learning periods across different initialization shapes. Notably, smaller initializations have longer period where an initial deficit will not impact the final solution, but have a more marked final effect with long deficit. In these experiments, once the task switched, we continued training for 50, 000 epochs, which was sufficient training duration for the small initializations to converge without a deficit.
We also find that our learning dynamics are robust to different initializations (Fig. g right).
A.3 COMPUTE TIME
All experiments in the paper can be reproduced on a local computer in around 7 hours. We used 2017 Macbook Pro (3.1 GHz Quad-Core Intel Core i7).
B EXACT DIFFERENTIAL EQUATIONS CHARACTERIZE MATRIX COMPLETION LEARNING DYNAMICS BEFORE AND AFTER TASK TRANSFER
We consider the singular value decomposition of the product matrix W(!) WD(t) WD-1(t)... Wt( ((tV The following equations describe how the effective singular values A(t) and the singular vectors U(t) and V(t) evolve over time, under certain conditions (Arora et al. 2019), which we will elaborate on:
((,(1) = DD B (003BB..[((((WW0))....
(12)
rū(1) = -U((I) (F(1) @ (UT(V((()))(())))) TO) (O U(I-U(IU (1) V(W
(13)
13

---

Published as a conference paper at ICLR 2024
Analytical Evelution:
Evelution.
Analydical Evelution:
Evelutio: 1750
2000
1502
13000
Analytical Evelation: Num dbs: 5000
Analytical Evelution. Num dbs: 5000
Analydicall Evelution: Num obs: 5000
Analytical Eveluton: Num abs. 5000
2300 1000
1507
10003
3000 10300 15200 25007
20000 20000 50000
23030
42000
20007
2010
Figure 9: Correspondence between learning dynamics through differential equations and neural network training. Simulating dynamics from the differential equations from Eq. [2 14 (dashed lines) matches simulations obtained with gradient descent training (colored lines). The blue dashed lines indicates the epoch where the task switches. (Top Row:) Given small number of observations (1750). the network converges towards a low rank solution when the deficit is removed early, while if the deficit is removed late, the network retains previously learned information (singular modes). (Bottom row:) Given large number of observations (5000), the network converges a low-rank solution regardless of when the deficit is removed. For these simulations, the initial ground truth matrix Mj was a rank matrix, and the final target matrix was a rank matrix M2. After changing tasks (indicated by vertical blue dashed line), we continue training for 10000 epochs. Additional parameters used: = 3, = 0.25, M1, M2 E NNN with = 100.
(t)) -t)) (F(1) @ [A(U)U (V) VT(VV (W(O)U(()A(I))
-I- Ve (W(O)UT (()(A)
(14)
where @ indicates element-wise product, indicates the depth of the network, F(I) skew- symmetric matrix with (1) (1) in entry (r,r") where r', and I refers to an identity matrix.
These equations describe the evolution of weights starting from balanced initialization (W3+100) (0)W j+1 (0) = W WO) (0) A j), and assuming that singular values are distinct and different from zero (Arora et al., 2019). The balanced initialization is approximately satisfied when starting from small initial conditions (Arora et al. 2019).
The gradient of loss can be computed in closed form for matrix completion with squared error loss.
(((((()) = - (Mr Wi3) if (i.j) E Ω otherwise.
(15)
Importantly, the conditions hold even when the task switches, as in our critical learning period experiments. This is because the difference of WT ( of training for all j (Arora et al. 2018). We show in Fig. 9 the simulations obtained through the differential equations match the simulations obtained by training a neural network SGD, both for the
W (T))) is constant
periods before and after changing tasks. (In practice we initialized each matrix as diagonal matrix with distinct and small values to ensure the conditions were satisfied.)
C COMPARISON AGAINST CLOSED-FORM LEARNING DYNAMICS FOR COMPLETE MATRIX OBSERVATIONS
Let product matrix Wtt) = ((t)A(())*(())) Arora et al. 2019 Theorem 3) showed that its singular values evolve as:
(1,(1) D -D - (()))
(16)
14

---

Published as a conference paper at ICLR 2024
Note that we added time constant
Using the loss L(W) = 2 Eayeen (M. W) from matrix completion, where M refers to the ground-truth matrix, the gradient is:
VL(We(t) [M w(t)ks
(17)
in the observed entries (i,j) S and 0 otherwise in unobserved locations.
Consider the case when all entries are observed and M USV and W = (t) then:
VL(W(()) U(S - (
(18)
and substituting Eq. 18 into the first equation Eq. 10 we will get:
(t) D a, (t)² - b. (s, (t))
(19)
This corresponds to the differential equation from (Saxe et al. 2013 Eq. 15). For depth D = 2 network (Saxe et al. 2013) found that the effective singular value of the product matrix
W(t) (t) (t) = UA(() Σ da(t)
(20)
would evolve in the following manner where
8 25at//T
a, (t) This equation precisely specifies the dynamics if the weights are initialized to lie in the SVD basis where W(t = 0) eUVT.
(21)
22000/7 + 8a/a2
C.I EXTENSION TO TRANSFER BETWEEN TASKS.
Consider a setting where we first are given subset of entries from matrix M. and train first the first epochs, and then switch or transfer to different matrix Mg which we continue training on for subsequent epochs. This setting corresponds to one where have two ground truth matrices M, and MB, corresponding to two tasks A and B. For the first task, task A, training will proceed as described in the previous section.
If the singular vectors are shared between tasks, MA USA V and M = V (but the ordering changes e.g. corresponding singular value), we can use the exact same dynamics from above to solve for transfer between tasks.
V
C.2 EXPERIMENTS
we show the predictions from Eq. 21 and the results obtained from simulating a neural network in the case of full observations and partial observations. In particular, we simulate neural network of depth D = 2 for matrix completion task of matrix dimension N x N, where N 100. For the first 10000 epochs, the ground truth matrix was rank matrix M.. and for the subsequent epochs, the desired task was rank 5 matrix M,. The matrices are constructed so that M, M, + s, where s, 20. We trained networks with a learning rate X 0.5, initialization scale Vap- 0.01, and time constant used the average matrix completion loss in our implementation (as opposed to the sum in Eq. (9) The closed-fon solution from Eq. 21 matches the simulation when all matrix entries are observed, and also approximately matches simulations when the number of entries is relatively large (Fig. I0 middle). When the number of entries is small, the network learns a different solution (Fig. 10 right)
In Fig- 10
N² where the A22 factor is because we
15

---

Published as a conference paper at ICLR 2024
Observations:
Observations:
Snguiar Vals
40
Snguiar Voll
40
40
30
30
Snngllr
30
20
20
20
10
10
5000
10000 15000
20000
50000
10000
15000
20000
5000
10000
15000
20000
Epoch
Epoch
Epoch
Figure 10: Comparison against closed-fore learning dynamics for complete matrix observations. When all entries of matrix are observed, can obtain an exact closed-form solution for the evolution of singular values (shown in gray dashed lines). The simulations of training a neural network are shown in colored lines. For the first 10000 epochs, the ground truth matrix was rank 6 matrix M., and for the subsequent epochs, the desired task was a rank 5 matrix M,. The change in task is denoted by the blue dashed lines. The matrices are constructed so that M, M, + 8,, v, where s, 20. (Left) If all entries are observed, the analytical prediction matches simulation and do not observe a critical period, as observed in Fig. 6 (Middle) Even with a fraction of observations (5000 entries which is 50% of the total entries), the analytical predictions assuming all entries observed for both tasks closely match simulation. (Right). Given a small number of entries (1500) the network does not eventually learn minimum rank solution, and as shown in Fig. [ and Fig. 6 is associated with worse generalization and critical learning periods.
Early Deficit
Middle Deficit
Late Deficit
2.0
2.0
5G0 Kay
5GD Kao
5G0 Kar
SGD
SG0 Ka
5GD K.
Singuiar
1.0
1.0
0
0.0
0.0
200
400
600
800
1000
0
200
400
600
800
1000
0
200
400
600
800
1000
Epoch
Epoch
Epoch
Figure 11: Early deficits affect learned representations in multi-pathway model using the Tanh nonlinearity, while late deficits do not. We consider a network with D. = D, = 3, with otherwise the same settings in Fig. (Left) Early gating deficit (denoted by gray period; epoch 0 to 150) to pathway leads to all features being learned in pathway A. When training with SGD, we indicate the singular values for pathway with crosses ('X'), and pathway B with plus sign ('+'), with the different colors identifying different singular values. We observe sigmoidal learning trajectories the singular values, with larger singular values learned earlier in training. (Middle) Deficits applied in the middle of training only affects previously unlearned features (bottom four singular modes), whereas other singular values are learned equally both pathways (crosses and plus signs overlapping for blue, orange, red, green singular modes). (Right) Late deficits (epoch 750 to 900) has negligible effect on how features are learned, and results in features being learned equally in both pathways (crosses and plus signs overlapping).
16

---

Published as a conference paper at ICLR 2024
Early Deficit
Middle Deficit
Late Deficit
2.0
2.0
1.5
Path Singular
2
1.0
1-
0.5
200
400
6.00
800
1000
200
400
600
800
1000
200
600
800
1000
Epoch
Epech
Epoch
Figure 12: Early deficits affect learned representations in multi-pathway model with the Relu nonlinearity, while late deficits do not. We consider network with D, D, = with otherwise the same settings as in Fig. 2 (Left) Early gating deficit (denoted by gray period: epoch 0 to 150) to pathway leads to all features being learned in pathway
When training with SGD, we indicate the singular values for pathway with crosses ( ('X"), and pathway B with plus sign ('+'), with the different colors identifying different singular values. We observe sigmoidal learning trajectories of the singular values, with larger singular values learned earlier in training. (Middle) Deficits applied the middle of training affects previously unlearned features (bottom four singular modes). Features that are partially learned (blue, orange, red, green singular modes) are slightly affected, in contrast to the setting without nonlinearity (Fig. 2). (Right) Late deficits (epoch 750 to 900) has a negligible effect how features learned, and results in features being learned equally in both pathways (crosses and plus signs overlapping).
17


Reviews:
[{"review_id": "a2Qi4GrQGD", "reviewer": "Reviewer_SR9U", "summary": "This paper is a continuation of the study of critical learning periods in deep learning. This work focuses on why critical learning periods emerge in deep linear network models. The paper investigates the critical learning period that depends on the depth of the model and structure of the data distribution, with supporting experiments. Meanwhile, they also show the learning of features is tied to competition between sources. They analyze the impact of pre-training on some tasks, and there are some simple experiments used to prove the “critical learning periods”. This work provides analytical understanding of critical periods in deep linear networks and draws connections between artificial and biological learning.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper continues previous work (on the deep network), the experiments correspond to research on the depth of the network, data distribution, competition between sources, and pre-training.\n2. This work on studying the competition of different data sources is solid, and it seems to be a relatively good job through the linear multi-pathway network.\n3. Analytical and minimal models provide fundamental insight. Intuition and empirical observations match well.", "weaknesses": "1. The paper is less readable and requires a higher theoretical foundation. The description of the “linear multi-pathway framework” (Sec.3.1.) is not clear enough and lacks corresponding details.\n2. Only studying deep linear networks seems not enough to clearly understand why deep networks have critical periods, and network depth and data sources cannot guarantee sufficient persuasiveness.\n3. There are fewer categories of experiments, and it would be better if the authors could provide more types of experiments to prove their claims.\n4. For the experiments related to pre-training, I cannot guarantee sufficient correlation with key learning practices. I hope I can explain the motivation for doing this part of the experiment. By the way, this is not to say that this is an obvious weakness.\n5. I don’t find any support in the paper for what the abstract said: “Why critical learning periods emerge in biological and artificial networks”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F4DSu7txW7", "reviewer": "Reviewer_PVne", "summary": "Noting the anecdotal and empirical evidence of critical periods in both biological and artificial learning agents, the authors hypothesize that critical learning periods are a general feature of learning, and not an accident of biology or training, They investigate this hypothesis analytically via deep linear networks, which retain many characteristics of typical neural networks while being amenable to tractable analysis. They perform a variety of analyses and experiments, largely inspired by work on critical periods in biological systems, demonstrating that depth and data distribution provide information processing constraints that give rise to the critical periods.", "questions": "See \"weaknesses\" for some less concrete suggestions (feel free to tell me what you plan to write if you address them and I'm happy to provide feedback!)\n\nThe main other thing I'd like to see is connections made/discussed with nonlinear networks -- what should we expect to change vs. hold? maybe according to the sources you cite (e.g. Saxe), or ideally, a suite of experiments with e.g. MLPs of increasing depth. \nAnother specific link could be made in Fig. 6 to scaling laws.\n\nsmall things:\n - first sentence shouldnt repeat abstract\n - 'competition between pathways affect' -> affects\n- inconsistent use of \"artificial neural network\", \"deep net\", \"neural net\", et al. For me it's fine, but could be confusing for someone outside the field; better to be consistent \n- title for plot should be removed (conflicts/causes potential confusion vs caption and X axis label, nonstandard for technical works to have an embedded title)\n - remind us what \"Path Singular Value\" is in the caption/change caption e.g .singular value dimensions (confusing that its referred to as 1-5 but labelled 0-4 on plot and called something different). All this stuff explaining the axis and basics of the plots should go first, before the interpretation. - I didn't understand the comment \"and do not affect performance on the final gask\" in fig $. Does it mean they have equal performance?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well written, interesting avenue of research. Results well explained early in the text, with clear examples and clear explanations of experimental settings and technical details. Particularly nice clarity given the cross-disciplinary nature of the work.\nRelated work is well written, reasonably thorough (to my knowledge), and well-explained.\nNice phase portraits.\nCaptions are mostly well-explained and self contained! (I don't know why this is so rare, but good job)", "weaknesses": "It's a bit of a tradeoff with having a clear hypothesis, but I find the case oversimplified/misframed  in the beginning and conclusion. Sure the literal biochemical explanation doesn't hold in an artificial setting, but analogies of this (reduced plasticity) could well occur. I don't think these are 'competing' hypotheses as they are framed; plasticity, imperfect optimization, etc. are *mechanisms* by which critical periods could arise in both artificial and biological systems; it doesn't tell us the reason  (inherent to learning, quirks of substrate, something else, etc.). In the conclusion \"natural information processing constraints\" are contrasted with biochemical processes like plasticity as an alternative hypothesis, but what is (decreased) plasticity buyt a natural information processing constraint? Demonstrating presence in artificial systems doesn't estabilsh it as a general feature of learning, except insofar as we've defined learning narrowly to be done by biological and ANNs (does naive bayes learn? does it exhibit critical periods?). It becomes sort of a circular argument about what does \"learning\" even mean. This commonality is actually supported by your experiments in Fig 3 recreating Guillery's. Anyway, I think the key sentences/claims (e.g. last of the abs) and results stand and are extremely interesting,  I would just like to see this framed around mechanisms or instantiations of information processing constraints, or something, rather than competing hypotheses. and maybe acknowledge more of the ambiguity of what \"learning systems in general\" means/ (without compromising clarity and narrative, which I believe are strong points of the paper).\n\nTerms used in equations are not explained when introduced  in 3.1 -- I know it's from a citation and you might be short on space, but you will lose a lot of readers here and it's a shame not to continue the clear prose you've had thus far.\n\nAround eqn 9, connection of matric completion to what you're trying to do is not clear -- be explicit about how imputing missing values is the same as / allows us to specify  task relationships with flexibility. \n\nWhile the relationship of experiments to showing dependence on depth is very clear and well-explained, the insights and relationship of experiments to data distribution is much less clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znMwbOp5rw", "reviewer": "Reviewer_uHcm", "summary": "This paper demonstrates that critical learning periods in deep learning models depend on the depth of the model and the structure of the data distribution, offering an analytical perspective on the learning dynamics. This study provides both analytical and simulation evidence that the learning of features is closely tied to the competition between different sources of information, highlighting the competitive dynamics within the learning process.", "questions": "Please consider the things listed in the “Weaknesses” section.\nAlso please consider providing information regarding the limitation and future work of this paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-written, technically solid and has a concrete flow.\n2. This work offers insights into the reasons behind the emergence of critical learning periods in both biological and artificial networks.", "weaknesses": "1. I think the assumption of the deep linear network is a bit strong. Since in real-world applications, most neural networks require non-linear activations. A deep linear network can be actually approximated by a single-layer linear network.\n2. Critical periods in artificial deep neural networks (DNNs) may be due to specificities of the optimization process, such as an annealing learning rate, or from defects in the artificial implementation and training, like ReLU units becoming frozen or gradients vanishing.\n3. What's the model's generalization ability on more sparse data settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqgaobkeTw", "reviewer": "Reviewer_qaj6", "summary": "This paper reveals that critical learning periods are not limited to biological learners\nbut are fundamental to the learning itself. Through well-written experiments, the\nauthors show how deep linear networks display critical learning periods. They\nconducted two experiments: one focused on competition in a multi-path model\n(similar to our eyes), and the other focused on the effect of critical periods on transfer\nlearning.\n\nThe first experiment reveals that a deprivation deficit in one pathway will result in the\nother pathway to winning the competition and learning all the corresponding features.\nThe deprivation only has an effect if applied in the early epochs. This reinforces the\ncritical learning period claim. The authors also show, by permanently lesioning the\ndominant pathway, that the deprived pathway isn’t dead and will pick up features that\nthe dominant pathway has missed.\n\nThe second experiment highlights the effect of depth and data distribution on the\nimpact that deprivation during the critical period has. It reveals that deeper networks\nare more sensitive to deprivations in the critical period, and the structure of the data\ndistribution also dictates the impact of the deprivation.", "questions": "> In the multi-pathway experiments, when we applied the deficit to\na pathway, which we refer to as “blocking” or “gating” an input to\na pathway in the paper, the desired target output was also shifted\nby a baseline amount corresponding the deprived pathway’s\noutput. This was to ensure that the normal pathway was only\nrequired to learn the unexplained component of the output, and\nnot the entire output.\n\nI understood the use of PyTorch’s “.detch()” for blocking the flow of gradients in the\ndeprived branch. However, shifting the desired target output to ensure learning the\nunexplained component did not make sense to me. Please elaborate.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a mathematical understanding of the psychological concept of the “critical learning period”.\n2. The paper reveals that critical learning is not restricted to biological learners but is inherent to learning itself!\n3. Theoretically strong.\n4. Well-thought-out and well-delivered experiments.\n5. The paper neatly highlights the pros and cons of competition between a model’s branches.\n6. By introducing a permanent lesion in the dominant branch, the authors show that the deprived branch isn’t dead for the rest of the training.\n7. The authors draw convincing insights from the transfer learning experiments, both low-to-high rank and high-to-low rank. In the appendix, the authors have also covered (both theoretically and experimentally) the case where two different matrices are involved.", "weaknesses": "> …our analysis shows that critical periods only depend on two main factors: the depth of the model and the structure of the data\ndistribution, as opposed to details of the architecture and optimization problem.\n1. The authors prove the dependence on depth and data distribution. But how can\nthey claim independence from other factors, like architecture and non-linearities,\nespecially when considering models other than deep linear networks?\n\n2. Figure 2’s markers should be changed. It is difficult to differentiate between a\ncircle and a circle-triangle overlap. Try plus (+) and cross (x) as markers or a\nstraight line (|) and an oblique line (/).\n\n3. Codebase not provided.\n\n4. Typos:\n* Page 3: had minimum the → had the minimum\n* Page 6: training has only affects → training only affects\n\n------------------------------\n\nOverall, I am appreciative of the work done. I would like the authors to release the codebase for reproducibility, and clarify some of the concerns raised. I would be happy to revisit the rating if these concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)