Contribution of task-irrelevant stimuli to drift of neural representations
Authors: Farhad Pashakhanloo
Venue: NeurIPS 2025 poster ()
Keywords: Neuroscience
computational neuroscience
representational drift
continual learning
linear neural networks
representation learning
stochastic gradient descent
Hebbian learning
learning rule
Primary area: neuroscience_and_cognitive_science
TL;DR: Using theory and simulations, we show that task-irrelevant stimuli could cause long-term drift of neural representations

Abstract: Biological and artificial learners are inherently exposed to a stream of data and experience throughout their lifetimes and must constantly adapt to, learn from, or selectively ignore the ongoing input. Recent findings reveal that, even when the performance remains stable, the underlying neural representations can change gradually over time, a phenomenon known as representational drift. Studying the different sources of data and noise that may contribute to drift is essential for understanding lifelong learning in neural systems. However, a systematic study of drift across architectures and learning rules, and the connection to task, are missing. Here, in an online learning setup, we characterize drift as a function of data distribution, and specifically show that the learning noise induced by task-irrelevant stimuli, which the agent learns to ignore in a given context, can create long-term drift in the representation of task-relevant stimuli. Using theory and simulations, we demonstrate this phenomenon both in Hebbian-based learning---Oja's rule and Similarity Matching---and in stochastic gradient descent applied to autoencoders and a supervised two-layer network. We consistently observe that the drift rate increases with the variance and the dimension of the data in the task-irrelevant subspace. We further show that this yields different qualitative predictions for the geometry and dimension-dependency of drift than those arising from Gaussian synaptic noise. Overall, our study links the structure of stimuli, task, and learning rule to representational drift and could pave the way for using drift as a signal for uncovering underlying computation in the brain.

change Contribution of task-irrelevant stimuli to drift of neural representations
Farhad Pashakhanloo
Center for Brain Science Harvard University, Cambridge, MA fpashaufas.. hww.aard.aa... edu
Abstract
Biological and artificial learners are inherently exposed to a stream of data and experience throughout their lifetimes and must constantly adapt to, learn from, or selectively ignore the ongoing input. Recent findings reveal that, even when the performance remains stable, the underlying neural representations gradually over time, a phenomenon known as representational drift. Studying the different sources of data and noise that may contribute drift is essential for understanding lifelong learning in neural systems. However, systematic study of drift across architectures and learning rules, and the connection to task, missing. Here, in an online learning setup. we characterize drift as function of data distribution, and specifically show that the learning noise induced by task- irrelevant stimuli, which the agent learns to ignore in a given context, can create long-term drift in the representation of task-relevant stimuli. Using theory and simulations, we demonstrate this phenomenon both in Hebbian-based learning Oja's rule and Similarity Matching- and stochastic gradient descent applied to autoencoders and supervised two-layer network. We consistently observe that the drift rate increases with the variance the dimension of the data the task-irrelevant subspace. We further show that this yields different qualitative predictions for the geometry and dimension-dependene of drift than those arising from Gaussian synaptic noise. Overall, our study links the structure of stimuli, task, and learning rule to representational drift and could pave the way for using drift as signal for uncovering underlying computation in the brain.
Introduction
Continual lifelong learning requires intelligent agents to learn continuously and adaptively from stream of data and experience M. this process, the internal representations of data may themselves shift or evolve over time. Understanding flexibility and stability of neural representations, as well as the learning algorithms that allow for efficient continual learning is fundamental both neuroscience and artificial intelligence (2). In neuroscience, recent advances have allowed for tracking neurons over several weeks or months. Such studies have revealed that representations at the single neuron level might not be as stable previously thought 3 4 This is specifically observed in the context
of stable performance and is referred to as representational drift
[5-8).
This phenomenon has also been recapitulated in computational models from different perspectives (116). Under one of postulates, long-term changes in representations are result of noisy learning [17. However, it is not clear which sources of noise drives this process. Broadly speaking, this could include biological sources, such as intrinsically noisy synaptic turnover [TX or those related
We take representational drift to be any long-term changes in the internal representations that occur after the loss or behavior reaches a staad This applics in both neuroscience and machine learning contexts.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to learning from experience [19], such as sampling stochasticity in online learning. Understanding the contributions of different sources noise could help reverse-engineer mechanisms of learning. especially if each renders distinct sets of predictions.
In machine learning, a primary source of noise during learning stems from stochastic gradient descent (SGD). There is large body of work on characterization of SGD noise with focus on how it can benefit generalization by driving the network toward flatter areas of the loss landscape (20 The noise in SGD has also been shown to drive long-term drift in the network parameters and representations after minimal loss is achieved 22 25).
is unclear if drift due to learning noise can also be observed across different architectures and learning rules (including bio pio-plausible rules), and what commonalities and differences among these setups. Here, we use theory and simulations to systematically characterize drift as a function of data distribution for different networks and learning rules. We show that certain data-dependency features of drift robustly hold across networks, and carry different predictions than drift caused by other sources of noise.
Contributions
We show in an online learning setup that task iusk-irrelevant data can act source of noise, changing the representations of task-relevant data over time despite maintained task perfor- mance.
We analytically study drift in a of canonical architectures and learning rules, including networks with SGD and Hebbian- based learning. Despite the individual differences. they all exhibit dependency of the drift on task-irrelevant stimuli.
Using both synthetic and real datasets (MNIST), we show that learning-induced drift leads to different predictions for the geometry and dimension-depeedennyy of the drift than those caused by Gaussian synaptic noise.
motivating example: drift under task-irrelevant noise
We start with simple example demonstrate the drift induced by lask-irrelevant stimuli in network trained with a Hebbian- -aaed learning. Specifically, we consider one-layer network trained with multi-dimensional Oja's rule (26 27]. This is canonical unsupervised learning method, which learns to represent the principal subspace of data at its output layer. The input to the network is R", and the output is given by y Wa æ E R where W mmx is trainable weight matrix (m < n). The online update rule is:
AW my( w"y)² OOja's 's learning rule),
where 7j is the learning rate. After convergence, the solution weight W aligns with the m-principal subspace of data (27). More concretely, if Σg VAVT is the singular value decomposition of the input covariance. then W QIm,n VT. where Q E m is an orthonormal matrix, and I, m a rectangular identity matrix with [Im, 8[. We see that the rotational symmetry at the output layer (associated with Q) creates a degeneracy for the solution. Here, we demonstrate how online learning, and specifically the statistics of stimuli, could lead to drifting weight matrices within this degenerate space over time. To do so, we consider Gaussian stimuli (0, Ex)) with covariance y, that has singular values of:
A diag [1,
A1)
(1 < 1). Here, the first m eigenvalues correspond to the m-principal subspace of the data X1 that the network learns to represent at its output. Conversely, the complementary (n -imdimmnsional subspace X1 associated with eigenvalue ty < is repressed at the output (i.e. y 0 for X1). Hence, we call the latter the rask-irrelevant subspace.
Figure I demonstrates simulations of continued online learning in this network after convergence. During this time, the principal subspace is already learned (as measured by a subspace distance), and norms of representations are stationary (Fig. [...). However, the autocorrelation of the representation

---

of a task-relevant stimulus decays over time, which is compatible with rotational drift. Importantly, we observe that the rate of the autocorrelation decay increases with the variance of the task vireleaatl stimuli (X1.) as well as the corresponding dimension (n m. (Fig. [.d.. This is interesting, because suggests that even though the task-irrelevant stimuli are suppressed at the output, their presence still leads to perturbation and drift of task-relevant stimuli representations over time. To get further
b) 1.2
C)
d)
Pitsiie Salapox Datuas
0.6
aod / 3n.)
0.8
0.0
Tosk rolevort Teak-imttawar
0.
0.4
0.2
VA
0.23
1L-311
5
10
10
10"
Figure 1: Demonstration of the effect of irreevatt stimuli on representational drift. A one-layer network is continuously presented with input samples and updated using Oja's learning rule long after convergence (n = 50, m = 30, 1) = 0.025). a) Grassmann distance between the m-principal subspace of the data and row space of W. b) Norms of representations for task-relevant (I1 E X|-- blue), and a task-irrelevant stimuli (æ1 E K| orange). Note that, y: 's for Tu fluctuate around zero which leads non-zero steady-state norm. c) Decay of average cosine similarity for task-relevant stimuli representations, under two values of rr Each curve is averaged for 30 stimuli; shaded regions indicate the standard error the mean). d) Same as c) but for n = 50 and n = 80.
insight into this, let's consider the update equation evaluated at a solution point W. By replacing W = W and y We x in Eq. the instantaneous update after seeing sample x becomes:
AW </ a w
In the above, ii and T are projections of æ onto the m -n-piiccipl subspace and the n m dimensional space orthogonal to that, respectively. This multiplicative term is particularly interesting, as it suggests both components need to be present in given sample for drive deviation from the solution. In the special case where æ E Xi, the stimulus already lies within the principal subspace, and no adjustment by the network is necessary. Similarly, when æ E XL, the stimulus is "filtered out" (y- 0), and update occurs. This nonlinear dependence of learning update on stimuli causes task-irrelevant stimuli to act as a source of noise, contributing to long-term dynamics and drift. We will next study this more systematically in the next section.
Theory and Models
3.1 Theoretical approach
We consider continual (lifelong) learning scenario in which the agent (neural network) experiences stimuli in the environment while performing a particular task. Since drift is experimentally associated with stable task performance, we assume that the agent has reached its optimal performance and the data are sampled in an online fashion from a stationary distribution, i.c. æ ~ P(x). (In the supervised learning case, includes the input-output pair). Let 0 denote the network parameters. After observing sample æ. a discrete learning update modifies the parameters according to:
AO -1/ 0),
where 17 is the learning rate, and g()) represents the learning rule. When learning follows gradient descent on an explicit objective, g corresponds to the sample gradient. However, in general, g may represent other update fields, such as Hebbian-based learning. We will consider both cases in the paper. We define the manifold of solutions as a set parameters that are stable fixed points of the above dynamical system. Previous work has shown that under certain conditions, such as small learning rate, the dynamics of learning with SGD can be approximated using a continuous-time stochastic differential equation (see 24
25
28)). We will adopt that approach, and in particular,
3

---

similar to [25], decompose the dynamics near point on the solution manifold into local normal (N) and tangential (T)) spaces the manifold:
dd H(O 0) (O)xB d0 (O)DB
(5)
Here, H = 0())=/00) denotes the Hessian (or in general the Jacobian of the average dynamics) evaluated 0, (0) and (0) are projections of C(0) cov(g)1/2 onto the normal and tangent spaces respectively, and B, and Bi denote independent standard Brownian motion. (For clarity, omit the explicit dependence of ! 0). The above decomposition assumes that H can be block-diagonalizn into the normal and tangent spaces. When the learning is gradient descent, this automatically the case as there exists an explicit loss function, and the projection of H into the langent space is zero. Among the Hebbian-based learning fields, we found this to hold for Oja's rule [26] and not the Similarity Matching network [29], for which the dynamics are not curl-free. However, we performed a similar decomposition of dynamics using non-orthon -othoroooaan projections (see Appendix C))
The first equation above describes fast fluctuations orthogonal to the solution manifold. For small perturbations, these dynamics can be approximated by Ornstein-Uhlenbed process, from which the corresponding covariance scales as x r). The second equation, governs a diffusion on the manifold itself, which, over long timescales, leads to drift of parameters and representations. Importantly, we focus on a late phase of learning where there is no effective tangential flow along the manifold so the dynamics in the tangential space purely diffusive. If (0) does not vanish at
II 224,, solution, i.c. (0) # O, then the leading-order contribution to diffusion is Do x 17². Otherwise, one must account for contributions from C0( at 0 + 0. this case, the effective diffusion also depends on the fluctuations, resulting in scaling of Da x 1y (see Appendix A for additional details).
3.2 Drift across architectures and learning rules
We now apply the above framework to analytically solve for drift in different networks (Figure 2). The choice of networks made to cover both Hebbian-basee and gradient descent learning rules in supervised and unsupervised setups. Importantly, the three unsupervised networks are intrinsically performing the same task of principal subspace tracking but achieving it in different ways. This fact allows us explore the role of learning rule and architecture in a more controlled way, when the task is the same. Additionally, the two-layer supervised learning setup more general, allowing for learning arbitrary linear mapping between the input and the output. However, for comparison other network, and as special case, the teacher in that network can also be designed to perform principal subspace tracking. Below, present the specific models and key results, with a more detailed discussion of Oja's case, as certain aspects its dynamics are shared with those observed in other networks. Complete derivations are provided in the Appendix. Throughout this section, we assume input data, æ E R", has a covariance with eigenvalues in descending order Xn d & A
b)
c)
d)
O00
000
LOOD
W
W
M
U
W
U
W
O
E R
y Rm
y yER
h E R
&ER"
3 E I""
2 E GER
ER
E R"
hERP
Figure Neural network models studied in this work. a) Similarity Matching network, c) autoencoder with bottleneck, and d) two-layer network. The first three networks are trained in an unsupervised way, while the twoel network is trained with supervised learning.
Oja network, b)
Forr the networks that perform principal subspace tracking, we assume that the m--principal subspace of data unique. This necessitates spectral gap between Am and Am+1.

---

Multidimensional Oja's network As mentioned in Section 2 Oja's network uses a Hebbian-basee
rule to perform principal subspace tracking at the output 26 27. The solution after convergence satisfies W WWT geometry has been characterized previously [30]. An arbitrary deviation from a point W on this manifold can be decomposed as:
m. for W R= This is known as Stiefel manifold and its differential
JW N1 N +T
(6)
N1 = KUOW WI, N₂ = Z WW T = SWWW,
for special matrices K Rmx(n-m) Z E Rmxm skew-symmetric D2 E R=x" and W1 ((n-m)xn whose row space is orthogonal to that of W (i, E [m], v [n - m]. see Appendix B). It can be shown that, in the above coordinate system, H becomes diagonal, with positive eigenvalues for N1 and N₂ (normal spaces), and zero eigenvalues for T (tangent space).
Further calculations show that the bulk of the fluctuations are subspace N1 and indeed are induced by task-irrelevant stimuli. This can be seen from the fact that projection of AW* from Eq. 3 non- zero along N1, while vanishes along N₂ and T. These fluctuations subsequently induce diffusion into the tangent space (see Appendix B A similar mechanism is observed in other networks.
The relation between the rotational symmetry of representations and the tangent space becomes evi- dent by observing that movement along the Ω W corresponds to rotations between orthogonal representations, y/ Wv, and y. W.,, where vs and U, are principal axes of the input space. Hence, the diffusion of the entire representation space can be described by m(m 1)/2 pairs of angular diffusion coefficients D.r for r, s E [m], r > s (see " In Appendix B we show that:
72
A
DOja
16
m+t
Amcton Dr
Av
In the above, the summation goes through the (n -immennniooaal task-irrelevant space, with the associated variances along different dimensions in that space (Am+u's contributing the sum, Total diffusion for the representation of a given stimulus, y. becomes: D. x-=)
D 22rllrr D.r-
This network optimizes a similarity matching objective using a bio- plausible Hebbian/anti-Hebbia learning rule in a one-layer network with feedforward and recurrent weights (W E _mx", .M E Rmxm respectively, Figure(bb) [ (2). The neuronal dynamics follow the update equation -ÿ- W My, where T is much smaller than the time constant of synaptic updates. The synaptic update rules are:
Similarity Matching network
AW- "yy W), AM = -=nn"""" - M) (Similarity Matching learning rule)
(8)
The linear version of this network essentially achieves the same principal subspace tracking as Oja's network. However, its update rules are local and hence biologically plausible. The rotational symmetry at the output layer is analogous to that in Oja's 's case and similarly, it corresponds to infinitesimal changes of the form oF -2 where F = M M-'Win is the filter weight from the input to the output. Consequently, the rotational diffusion can be characterized by the following pairwise coefficients Dar, for r, E [m], > (see Appendix ():
1
DSM
2 A
(9)
16A.Ar
Am
Am
We again see that the summation is over the (n m) dimenssiol task-irrelevant space; however, the specific dependence on the spectrum differs from that in Oja's case.
Autoencoder Here we consider two-layer linear autoencoder with bottleneck. The encoder and the decoder weights are U E Px and W "xxp respectively. with p < n (Figure 2b). The network is trained with SGD with batch size of one. At the solution, the bottleneck layer represents
'Because of the diffusive process that underlies representational drift in this work, we will measure diffuusion rates a proxy for drift. Further, as long as drift is associated with rotational transformations of the representation space, it sufficient to measure pairwise angular diffusion rates Drr. This holds for all networks studied here.
5

---

the o-principal subspace of data B1. We also assume that the network operates near a balanced weight solution, where the solutions weights U and W satisfy U W The rotational symmetry, in this case, corresponds to coordinated weight changes of the form ow = $2 W and OU = SWT. Similar to Oja and Similarity Matching cases, pairwise angular diffusion coefficients Der can be used to characterize the diffusion, which in this case applies to the hidden layer representation h = Ux. In Appendix D
we show that for r, 8 E [p], r r>r> 8:
T A R
D
Aou (der Aptu) + f(Ar, Aptu)),
(10)
64
where f(Ap, Apto)= Exc Q(a)=0 (I+a")(1-a) and Q(a) Apture + (A Apto)a Apta- A common feature of this result with previous cases is the dependence of diffusion on the variance in the task-irrelevant subspace which in the autoencoder case has dimension n p (note that the diffusion vanishes if all Aptt are zero for E [n p)). The exact dependence on the spectrum determined by the solutions of the quadratic equation ((a) = 0.
Two-layer network (supervised) Finally, we study drift in the hidden layer representation of an expansive two-layer network trained under a linear regression task (Figure 2d). Specifically, the input-output relationship determined by the teacher signal y Px, where P E Tx (m < n). This is more general task than the principal subspace tracking studied for other networks. In this case, the task-irrelevant stimuli lie in the null-space of the mapping P (and for which y = 0). We train the network with online SGD and weight decay coefficient 7. This setup also exhibits a similar rotational symmetry to that observed in the above networks (with most resemblance to the autoencoder network), resulting in rotational diffusion within the hidden layer. We leave the derivation and the results to Appendix and only present special results in the next section.
4 Numerical results
In this section, we study drift in different experimental setups and datasets using theory and simula- tions. To measure the drift rate in simulations, we run multiple instances of continued learning, all starting from the same trained network, but undergoing different random data sampling seeds. We then compute the slope of the average autocorrelation curve as an estimate of the drift rate.
4.1 Gaussian data
To specifically study the effect of task-irrelevant stimuli to drift, we apply our theoretical predictions to the Gaussian toy data introduced in Eq, 2 To recall, the stimuli covariance eigenvalues are a = I in the task-relevant subspace (principal subspace for the unsupervised networks), and are equal to Y in the task-inrelevant. This is a simplified dataset and allows for controlling the effect of task-irrelevant stimuli. Replacing this spectrum in the theoretical predictions for drift in Section 3.2 and assuming T
<! leads to the following results for the total diffusion rates:
Dy
(m -1)(n m)
(Oja and Similarity Matching)
(11)
Dn ~ 2222 32
(p 2 (p- p) (Autoencoder)
(12)
We see that the rate of drift increases with the variance and the dimension in the task aask-irrelevant space. The dimension of this space for Oja and Similarity Matching is determined by the difference between the input and the output layer dimensions, i.e. x (n - (n-m). while for the autoencoder depends on the dimension of the input and the bottleneck layer, i.e. D x (n p). The theoretical results show excellent match to drift rates measured in simulation (FigureB3) 33..
For the supervised network, we see a similar dependency of the drift on the task-irrelevant subspace. Specifically, if rank of the input-output mapping is denoted by & m, and its non-zero singular values are equal to one, total drift rate for the representation simplifies to (Appendix F)
2 (k (k + 2 22++6 (n b) d+) 2
D ~
(Supervised Two-layer)
(13)
16

---

The above consists of a baseline drift as well as an additional drift term that depends on the dimension and variance of the the task-irrele iise-irteeevant subspace. However. in contrast to previous principal subspace task, the task-irrelevant subspace is determined by the null-space of P and has dimension n k. The principal subspace task can be considered as a special case of the above where the right singular vectors of P align with the principal subspace. Additional simulation results are shown in Figure ST
8)
b)
10
2.5
10
10
O
Dosa Dast
1.5
D (raff
10 15
v/l
10
-47,8 Thoory
0.000
0.001
0.01
0.002
0.00
0,005
0.5
10
0.001
0.002
0.006
0.01
0.02
10
2.5e-06
0.0001
0.0004
0.0025
0.01
0
71 712
Figure 3: Drift rate for different architectures as a function of: a) learning rate n, b) mask-irrelvaan variance λ1, and the dimension of the task-irrelevant space, n mm. Note that m m for Oja and Similarity Matching (SM), and m = P for the autoencoder (AE). For panels a) and b), 5, m 3; for panel c), 17 0.001 and vλ1 0.01.
4.2 Nonlinear network
We also demonstrate the contribution of task-irrelevant stimuli to drift in nonlinear network with localized receptive fields. We consider version of the two-layer network where the network reconstructs the task-relevant position variables on a ring at its output. Specifically, data at the input x E R". consists of telk-velevant variables denoted by 21 cos and x2 sin( and task- irreevatel noise represented by z; NO00, A1). for i E {3.4.. n. The output layer has to reconstruct the position, which means the target output is y [41 x2|" The predicted output is Wh, where h = Rel.U(Ux)) is the activation at the representations layer (Figure 4h). The network is trained with SGI) and weight decay, and learns represent the position on the ring near perfectly at its output using MSE loss. At this solution, the hidden layer neurons form localized receptive fields (RF) that tile the ring (Figurefpp, similar to (9). However, continued training leads to reorganization of RFs at the hidden layer, while the kernel (representational similarity matrix) stays stable. Interestingly, an increase in A leads to a faster decay of representations at the hidden layer (Figure 4b). This suggests that consistent with the results of linear networks, the perturbation created by task irreleatt stimuli may lead to a lower stability of representations in nonlinear networks.
4.3 MNIST data
In the previous section, used toy Gaussian data which allowed to simplify the drift equations and study the role inselirrelevant stimuli in a controlled manner. Here, we apply the theory to MNIST data [32]. Figure 5h, shows two snapshots of representations the hidden layer of an autoencoder with the bottleneck dimension of p 2 trained on 60000 MNIST images in an online way. We observe approximate rotation of the representations from snapshot another, which is caused by drift of parameters during training. Per the theory in Section 33.2, the drift rate should be a function of the whole spectrum with contributions from the inreevatle stimuli. To test this more systematically, we keep the input data the same but change the output/botlleneck dimension for each network. To make the experiments more computationally manageable, we first project the MNIST data onto the top 20 principal components and use that as the input to the network (n 20). Figure (5p shows drift as a function of output for Oja and SM, and the bottleneck dimension for AE Interestingly, in all cases the drift rate initially increases to maximum and then decreases to zero at m ni (or p = n for AE), showing great agreement with the theoretical predictions. This trend can be explained by a trade-off between two opposing factors. First, the total space available for drift increases with the dimension of the representation layer (this dependency can also be observed the Gaussian data in Section 4.1 and it corresponds to factors in Eq. and in Eq. 12 The
7

---

a)
b)
Receepivve
(0)
sin(f)
(=4
0.9
(
0.
U
W
6u6g
O
E/
E
0.4
3.6
# #ERR"
he R'
0.5
x10
Figure 4: Example of drift in non-linear network. a) Schematic of a two-layer network trained to represent the position on the ring (0). b) Reorganization of receptive fields after continued training, as shown by two snapshots t 1.25x106 and &g 5x10%. The right panel shows the stability of kernel (0 022 = h01) (02)) across time. c) Decay of average cosine similarity and its dependence on To (averages were performed over representations of 5 uniformly chosen angles and 80 runs, shaded area: standard error of the mean). Unless otherwise specified, for all simulations: n = 8, 10,
m 2, 77 0.2, y 0.05 and t 0.25.
second factor can be attributed to the source of noise that exists during learning, which as we saw is driven by the task-irrelevant stimuli. As the output or the bottleneck dimensions increase, this space effectively shrinks in size, reducing the amount of noise and thereby decreasing the drift. This is why, the limit where this dimension equals the input dimension, the drift vanishes altogether.
a)
t= 11
t= l2
3
digit
lr2
hg 0
-1
7
-2
88
-2
0
2
h1
h1
b)
Oja
SM
AE
Tureey Experimcur
(bormolizel)
0.8
D (bormellzoo)
0.8
(porzonlizad)
0.9
0.6
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
15
10
772
372
Figure 5: Drift in MNIST data. a) Two snapshots of hidden layer representations for 10 sample digits in linear autoencoder (n m 784, p = 2. 0.01, t 1.6 10 and 12 10²). To show the fluctuations alongside the drift, a time window tu. = 5000 preceding each snapshot is considered and representations at 100 uniformly chosen times overlaid. b) Normalized rate of drift function of output dimension (m) for Oja and Similarity Matching (SM), and bottleneck dimension (p) for autoencoder (AE). For the results in this panel, projections of MNIST data onto its top n 20 principal components are used as input. The plots show the drift rate for the representation of the top principal vector (shaded area: standard deviation over multiple runs).
8

---

5 Learning noise vs. synaptic noise
So far, we have shown that noise due to online learning sufficient to create drift, even in the absence of explicit synaptic noise. A natural question arises as to how these two types of noise lead to different qualitative and quantitative effects on drift. To explore this question, consider a case where in addition to the intrinsic learning noise, an additive synaptic noise is present during learning [9]. In the case of Oja's network, the noisy update becomes:
AW myy(ar WTy) +
(14)
where eij (O, kyo) is additive Gaussian synaptic noise, and Syyn determines its strength. Unlike what we previously observed in the case of pure learning noise, the projection of synaptic noise the tangent space of the solution manifold is non-zero. This makes the calculation of the diffusion on the manifold simpler than the case of learning noise. Specifically, if T' is an arbitrary unit vector in the tangent space, we have:
(proj(e, Ynoise = noo agm
(15)
By ignoring higher order terms, the perturbations caused by synaptic noise along the tangent space accumulate over time while those in other directions are mean-reverted to zero. It is easy to show that the corresponding pairwise diffusion between two orthogonal representations D3r ny /4, where one factor of 1/2 results from conversion from the parameter the representation space, and another 1/2 from the definition of diffusion. A close comparison of this to the pairwise drift caused by intrinsic learning noise reveals that the synaptic-induced diffusion is isotropic while the learning- induced drift is in general anisotropic (the latter can be observed from Eq. / where for given stimulus, the drift rate of representation along different directions not the same). Hence, the "geometry" of drift shows a major qualitative difference between the two sources of noise. Additionally, we may also compare the overall drift rates between these two sources of noise. For the Gaussian toy data in Section 4.1 total diffusion for representation becomes:
x²
1 (m 1)(n - -m) 7 1 egn (m - 1),
Dy
(16)
where the first term denotes the drift caused by intrinsic learning noise, and the second term stems from additive synaptic noise. The overall drift rate is plotted in Figure as a function of the synaptic noise strength for the Gaussian data. We see that synaptic noise dominates when the strength higher
than Sagn 17λ vn m.. Additionally, as expected, the drift rate does not vanish at small values of a aagni instead it plateaus at a level that depends on the task- isss-irreeevant noise.
Another qualitative difference between the two sources of noise is evident from the dependency of the drift on the output dimension. In the previous section where the only source of noise was learning noise, we showed that the drift rate may have a non- monotonic relationship with the output dimension. Here, we repeat the MNIST experiments with different levels of synaptic noise during learning. As shown in Figure bbp, as the strength of synaptic noise increases. the relationship approaches a monotonically increasing function of the output dimension, yielding a qualitatively different dependency.
6
Discussion
Here, we applied a theoretical framework to study representational drift across a set of canonical net- works and learning rules. Our focus was on understanding drift arising from the inherent stochasticity of online learning. A key finding was that task-irrelevant stimuli can act as a source of fluctuation during learning, potentially leading to long-tern drift in the representations. This phenomenon was consistently observed across all the networks studied, and rendered different predictions than an additive synaptic noise.
One feature of the chosen networks was that they perform similar tasks, but differ in their architectures and learning rules. The exact dependency of the drift on the data distribution was different for each architecture. Nevertheless, in all cases, the extent of stimuli that the network learns to ignore influenced the drift of representations of task-relevant stimuli. At first, this might sound unintuitive; however, the online nature of learning prevents the networks from being fully agnostic part of the data distribution. Such sensitivity allows for adaptation in the case of non-stationary data.

---

al
b)
10
0.8
(mnormalizel
D frad [t]
10-10
vAi
0.8
0
0
00.11
0,002
0.02
0.01
0.2
10
Depertment
0.2
Tivory
-0.5
0
1e-07
19-06
1e-05
0.0001
0.001
0.01
0.1
10 15
Syn
Figure 6: Comparison of drift induced by learning noise from iirklirreeevant data and by Gaussian synaptic noise in Oja's network. a) Drift rate as a function of synaptic noise strength (Tayn). shown for different values of Y in the Gaussian data (n 5, 3, 17 0.01). b) Normalized drift rate for MNIST data as a function of output dimension (m)) shown for three values of sym-- Curves are normalized to their maximum values separately (shaded area: standard deviation over multiple runs).
This may have some resemblance to the problem of ongoing memory storage 14. In the case of gradient- based methods where the loss function is explicit (AE and two-layer networks in our work), this can explained by non-vanishing sample loss which leads to persistent parameter updates even after convergence. Previous work has shown that SGD with weight decay in two-layer autoencoder with an expansive hidden layer can induce drift [25]. Our work extends those findings to different architectures and to supervised learning, and highlights the role of task-irrelevant stimuli distinct source of instability.
Drift has been observed across various cortical areas and under different experimental designs. Nevertheless, we did not find any studies that specifically examined the influence of task-irrelevant stimuli. We can, however, speculate about potential connections to some existing findings. In [33] the amount of drift observed in an association area where different types of stimuli are multiplexed appears to be higher than drift at the origin areas. This might be related to the phenomenon discussed our study as different sets of stimuli are combined with varying levels of relevance depending on the task. Additionally, a potentially related observation comes from the visual cortex, where more complex and naturalistic stimuli show higher drift compared to simpler stimuli [34]. This could also relate to our finding that, in network with limited representational capacity (such the bottleneck in our work), a more complex stimulus activates a higher number input eigenmodes, resulting in a greater amount of "task-irrelevan"" content, and consequently a higher drift. Finally, in our work all stimuli may be simultaneously present and sampled in the same environment. We did not consider cases where the inreevatt stimuli appear in separate context studied in [35].
In this work, the theoretical derivations of drift were limited linear networks. These networks, despite the linearity of the input-output relationship, have been shown to demonstrate nonlinear learning dynamics (36] This indeed underlies many of the findings in our work, namely the dependency of drift on task-irrelevant stimuli, which was observed in both SGD and Hebbian learning. Along this line, we also found rather complicated functional dependency of the drift on the data spectrum, the specifics of which varied for each architecture and only simplified under structured spectra. Nevertheless, we showed numerically that our main finding also holds for nonlinear theoretical framework relies on approximations of small learning rates,
activation. Additionally, and fluctuations around the solution. For large learning rates, effects such as the finite step-size effect or edge of stability may come into play and lead to more complicated dynamics (37 38).
Noise due to sample stochasticity, and specifically task-irrelevant stimuli, might be among many sources of noise simultaneously present in the brain [17]. Here, we showed that in the regime where Gaussian synaptic noise is strong and dominates, the rate of drift increases with the dimension of the representation layer, while more nuanced relationship exists for the sample-induced noise. These different predictions could help guide experiments aimed at uncovering the sources noise and nature of drift (39] Specifically, this could be tested in future experiments where the amount of task- irrelevant stimuli is controllable by the experimenter, such as in olfactory soure--round segregation tasks in which the number and the extent of distracting stimuli can be controlled systematically [40]. Our work lays a thcoretical foundation for those endeavors.
10

---

Acknowledgments
I would like to thank Jacob Zavatone- Vetion for helpful discussion and comments on the manuscript. I am also grateful to Juan Carlos Fernández del Castillo and Venkatesh Murthy for their feedback on an earlier version of this manuscript. This work was supported by the Harvard Center for Brain Science (CBS)-NTT Fellowship Program on the Physics of Intelligence. The computations in this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
Code Availability
The codes and core functions to reproduce the main results are publicly available at: http://ggh...coom/psssaaaaallbld
References
[1] German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
[2] Timo Flesch, Andrew Saxe, and Christopher Summerfield. Continual task learning natural and artificial agents. Trends in neurosciences, 1693)1199-210, 2023.
Carl Schoonover, Sarah N Ohashi, Richard Axel, and Andrew JP Fink. Representational drift primary olfactory cortex. Nature, pages 1-6, 2021.
[4] Yaniv Ziv. Laurie D Burns, Eric D Cocker, Elizabeth O Hamel, Kunal K Ghosh, Lacey Kitch, Ab- bas El Gamal, and Mark Schnitzer. Long-term dynamics hippocampal place codes. Nature neuroscience, 16(3):264-22 266, 2013.
Michael E Rule, Timothy O'Leary, and Christopher Harvey. Causes and consequences of representational drift. Current opinion in newrobiology. 58:141-147. 2019.
[6] Daniel Deitch, Alon Rubin, and Yaniv Ziv. Representational drift in the mouse visual cortex. Current Biology, (194:43737-433 2021.
[7] Laura N Driscoll, Lea Duncker, and Christopher D Harvey. Representational drift: Emerging theories for continual learning and experimental future directions. Current Opinion in Neurobiology, 76:102609, 2022.
[8] Paul Masset, Shanshan Qin, and Jacob A Zavatone-Veth. Drifting neuronal representations: Bug or feature? Biological cybernetics, pages 1-14, 2022.
[9] Shanshan Qin, Shiva Farashahi, David Lipshutz, Anirvan Sengupta, Dmitri B Chklovskii, and Cengiz Pehlevan. Coordinated drift of receptive fields in hehbian/ant--hchhian network models during noisy representation learning. Nature Neuroscience, pages 1-11, 2023.
[10] Kyle Aitken, Marina Garrett, Shawn Olsen, and Stefan Mihalas. The geometry of representational drift in natural and artificial neural networks. PLOS Computational Biology.
2022.
[11] Aviv Ratzon, Dori Derdikman, Omri Barak. Representational drift as result of implicit regularization. Elife. 12:RP90069, 2024.
[12] Maanasa Natrajan and James E. Fitzgerald. Stability through plasticity: Finding robust memories through representational drift. bioRxiv, pages 2024-12, 2024.
[13] Jens Bastian Eppler, Thomas Lai, Dominik Aschauer, Simon Rumpel, and Mathias Kaschube. Representa- tional drift reflects ongoing balancing of stochastic changes by hebbian learning. bioRxiv, pages 2025 01, 2025.
[14] Federico Devalle, Licheng Zou, Gloria Cecchini, and Alex Roxin. Representational drift the consequence of ongoing memory storage. bioRxiv, pages 2024-06, 2024.
[15] David Kappel, Stefan Habenschuss, Robert Legenstein, and Wolfgang Maass. Network plasticity as bayesian inference. PLoS computational biology, 11(1))c1004485, 2015.
[16] Uri Rokni, Andrew G Richardson, Emilio Bizzi, and H Sebastian Seung. Motor learning with unstable neural representations. Neuron, 54(4):653-66 666, 2007.

---

[17] Charles Micou and Timothy O'Leary. Representational drift as a window into neural and behavioural plasticity. Current opinion in neurobiology, 81:102746, 2023.
[18] Gianluigi Mongillo, Simon Rumpel, and Yonatan Loewenstein. Intrinsic volatility of synaptic connec- tions challenge to the synaptic trace theory of memory. Current opinion in neurobiology. 46:7-13, 2017.
[19] David Kappel, Robert Legenstein, Stefan Habenschuss, Michael Hsich, and Wolfgang Maass. A dynamic connectome supports the emergence of stable computational function of neural circuits through reward- based learning. eneuro, 5(2), 2018.
[20] Stanislaw Jastrzkebski. Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer. Yoshua Bengio, and Amos Storkey. Three factorr influencing minima in sgd. arXiv preprint arXiv:1711.04622 2017.
[21] Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects, arXiv preprint 1.XIv:1803.0011 2018.
[22] Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1-10. IEEE, 2018
[23] Sho Yaida. Fluctuation dissipation relations for stochastic gradient descent.
arXiv preprint
2018
[24] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? -a mathemati- cal framework. International Conference on Learning Representations, 2022.
[25] Farhad Pashakhanloo and Alexei Koulakov. Stochastic gradient descent-induced drift of representation in two-layer neural network. In International Conference on Machine Learning, pages 27401-27419. PMLR, 2023.
[26] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of mathematical biology. 15:267-273, 1982.
[27] John Hertz, Anders Krogh, Richard G Palmer, and Heinz Horner. Introduction to the theory of neural computation, 1991.
[28] Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent approximate bayesian inference. arXiv preprini (oXv::1704.04289, 2017.
[29] Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri Chklovskii. Why do similarity matching objectives lead to boobiian/antihhebbia networks? Neural computation, 30(1):84 124, 2017.
[30] Alan Edelman. Tomás Arias, and Steven Smith. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303-355 1998.
[31] Pierre Baldi and Kurt Hornik Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2( 11::53558, 1989.
[32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2322 1998.
[33] Luis M Franco and Michael J Goard, Differential stability of task variable representations in retrosplenial cortex. Nature Communications, 15(1):6872, 2024.
[34] Tyler D Marks and Michael Goard. dimulls-deppndenn representational drift in primary visual cortex. Nature commanications, 12(1):1-16, 2021.
[35] Gal Elyasaf, Alon Ruhin, and Yaniv Ziv. Novel off-context experience constrains hippocampal representa- tional drift. Current Biology, 34(24):5799-5773, 2024.
[36] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
[37] Daniel Kunin, Javier Sagastuy-Bas Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel LK Yamins. The limiting dynamics of sgd: Modified loss, phase-space oscillations. and anomalous diffusion. Neural Computation, 36(1):151-174, 2023.
12

---

[38] Jeremy M Cohen, Simran Kaur. Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs the edge of stability. arXiv preprint arXiv:2133.00655, 2021.
[39] Nitzan Geva, Daniel Deitch, Alon Rubin, and Yaniv Ziv. Time and experience differentially affect distinct aspects of hippocampal representational drift. Newron, 111((5):23572366. 2023.
[40] Dan Rokni, Vivian Hemmelder, Vikrant Kapoor, and Venkatesh N Murthy. An olfactory cocktail party: figure-ground segregation of odorants in rodents. Nature neuroscience, 1729)1225-1222,, 2014.
[41] Crispin W Gardiner et al. Handhook of stochastic methods, volume 3. springer Berlin, 1985.
13

---

Appendix
Summary of Framework
B: Oja Derivations
15
C: Similarity Matching Derivations
D Autoencoder Derivations
20
Two-Layer Network Derivations
A Summary of Framework
Here, we provide summary of the framework, and specifically provide additional details deriving the fluctuations and diffusion dynamics, which are necessary for solving specific networks later sections. The decomposition of dynamics is based the framework in Ref. [2] but extend that work beyond SGD.
A.1 Fluctuations
For small learning rates, the stochastic differential equation (SDE) for the deviations in the normal space (first line in Eq. (5), can be approximated as the following Ornstein Uhlenbeck (OU) process:
n0 HO + Vn CdB..
(17)
Here, replaced C(0) in Eq. 5 with its value the manifold. This means C C(0) )) 1/2 where gx(x) is the sample update on the manifold. We define coordinates p within the normal space below:
N and Hn, Ag d 0.
where N = [nil n2) ... [nx],
(18)
The stationary correlation function in these coordinates can be derived by solving the above OU process:
(px())(((s)) 7/ g. n. g-)= -nu(t-s), (t s, k, [K]).
(19)
A + A
(see
25
41
And finally, the stationary covariance can be obtained from the above by setting =
" (pxpi) = A + (nīg. n g=)z: k, [K].
(20)
To calculate the above terms, one needs to find the normal eigenvectors n& 's, well as the projection of learning update on them. We use the above equation find fluctuation variance for all the learning cases in future sections.
A.2 Diffusion
In this section, we approximate the learning dynamics the manifold to derive effective diffusion coefficients for the representations, The discrete update along the manifold, evaluated at point 0+ Np. can be written as:
A = -- N+Np).
(21)
where 9r = lg) is the projection of the update vector onto the tangent space (note that, H block-diagonalizes in the tangent and normal spaces, this projection is a Euclidean projection. Otherwise, this can be non-orthogonal projection, as is the case for the Similarity Matching shown in Appendix().. The value of gr is calculated a point away from the manifold, and hence depends on p. In the absence of any flow in the tangent space, the displacement along the manifold follows an effective diffusion process, with the diffusion tensor of Dp (g+9*) =,,p in the parameter space.
Since we are interested in rotational drift in the representation space, we can define corresponding angular diffusion coefficients that characterize the rotational diffusion in that space. Specifically,
14

---

define Der to be the angular diffusion rate between the representations h2 and h, (or alternatively. ys and y, if we consider the output layer as the representation layer, which was the case for Oja and SM). To find those coefficients, we need to project Dp along the corresponding axis of the tangent space and make appropriate conversion from parameter to representation space. By expanding Eq. 21 to the leading term in (assuming small fluctuations), the pairwise diffusion coefficients become ):5):
K
11²
DEr == (Apt))a =
(pEPL) "9"","") 1d1=1
Σ
(22)
2
Here, coefficients 9:' (x) characterize the amount of angular perturbation between h, and h, that results from observing sample when deviation is along ny; these can be calculated for each specific network. It should be noted that, in general, if gr does not vanish on the manifold, we should also have zeroth order terms in the above. However, we did not observe that in any of the networks studied.
B
Oja Derivations
Recall from the main text that, if Ee VAVT is the singular value decomposition of the input covariance, the solution satisfies:
W Ql VT.
(23)
Here, Q E R xm is an orthonormal matrix, and I, RR is a rectangular identity matrix, i.e. mm, = o/. We denote the eigenvalues of ,x by 's. This solution, as well as its stability, has been shown previously (see 22
B.1 Average dynamics near the fixed points
The average flow near the solution manifold can obtained by taking the expectation of the update rule AW- 1y( WT, y) with respect to
(AW) w/ (ax" ), W WTW).
(24)
Computing the above at point W 8W near the manifold, and keeping the leading terms yields:
w2) = NoWE E, J,, W W) n, (W wW + ow"w) + O(8WW)) (.)
(25)
The eigenvalue equation for H, the Jacobian of the dynamics (see Eq. 5) satisfies Hvec(N) = dn vec))N, where N E mx and λy are the eigenvectors and the eigenvalues respectively, and the "vec" operator reshapes matrix to a vector. In terms of the N matrix directly, this can be written as:
- NE W) WE, WW N + + W) N
(26)
As mentioned in the main text, an arbitrary deviation from the manifold can be described as (B3)):
oW Ni + N2 +T
(27)
N = K N₂ ZWW,
T=Q
One can show that the H is indeed diagonalized in the above coordinates. Below, we will mention the exact coordinates corresponding to each subspace, as well as associated λи. It is straightforward to show that each of the subspaces satisfies Eq. 26
Subspace N1 :
N: = KH.WW, where K E (n-m( [KH1) = Oipoy E [m], v E [n - m]
X A,. Am+rs
dim m(n m)
In the above, W, E (n-mm)x is full-rank matrix whose row space is orthogonal to that of W
Also, recall that gi 's are the columns of the orthonormal matrix Q in 23 The interpretation of
displacements this normal subspace is that the representations of antr E X1, which are on average zero, move toward representations of a X
15

---

Subspace N2:
z = x; 9: =q.4% i,j E [m], 2
22 = ZW
X,
dim((N2) = m(m + 1)
\ = X +1
2
Subspace T
T T= W, Ω -
(q-9" -4.9");
1,8 E [m]
V2
H
0,
dTm
This is tangential subspace and it corresponds rotations within the m-dimensional output space.
B.2 Fluctuations
Update on the manifold can be derived by replacing W = W the Oja's update equation (Eq.
AW, ny(a VTy)²
(28)
"Wal" - x. WW)
Waww"(I, W W)
In the above, Ti and a are projections of æ onto the m m-principal subspace and the n m dimensional subspace orthogonal to that, respectively. This the same equation as Eq. in the main text.
Next we observe that among the above three subspaces, the projection of AW, is only non-zero along N₁. This can be easily verified by using the equation of inner product between two matrices, namely for A, B Rmx. the inner product is: (A, B) tr(A" B). The projection onto N₁ becomes:
proj K KIOWL) = Tm
(29)
where we defined proj b) = b"a Correspondingly, proj(AW., N₂) = and (A., T) [[bll
Here, Fy = " x is a component of stimulus in the principal subspace, and x, v.T v x component in the task-irrelevant subspace. Replacing the above and the corresponding X in Eq 20 obtain the covariance of fluctuations associated with this subspace:
( 2(xpp \m+vy)
E [m], v E [n],,,
(30)
2(1
and the other cross-terms The above indicates the extent to which the representation vector hm fluctuates toward hn.
B.3 Calculating the Diff'usion
We already saw in the above that the only fluctuations occur in the subspace Ni. We will next approximate the gradient near the manifold point W W + pKW where p indicates the extent of deviation:
AW W W+KKW_ = w/Wxx" (I, W W)
(31)
=n(W + pKW1)xx' (I, (W (WW_) PKWL))
=nwxx (I, WTW)
+ +nolKWixst (In W W) Wax" T W KT W Waw W kW1]+0
16

---

The projection of the above onto the tangent space T = 5 W becomes:
proj [ W
W.KKWWW, S W)
n00m+ (6" 14
(32)
v2
We now Eq. 22 to calculate the pairwise diffusion constants. In that formulation, have the following tensor coefficients:
G 2 J Amm+u(0 8,..), μ E [m], v [n ml
(33)
Recall that this coefficient characterizes the extent of displacement between representations of stimuli v, and upon observing sample æ, when the network is deviated from the solution along n The corresponding pairwise diffusion coefficients become:
D&r T
Σ
Σ
8, E [m)]
(34)
2
ME mm)
WE E m-]
(m ( (PP + (2.2))))))
8
DE ml
As
22 N
16
Amty
Am=uu
DE
ml
In the last line we replaced the fluctuation covariance terms (Phn)) by its value from Eq. 30
C Similarity Matching
As discussed in the main text, the network includes feedforward weight W E Rmxn and recurrent weight M Rmxm The input-output relationship could be represented y Fæ. where F M M is the filter weight matrix. Similar Oja, the solution (fixed points), the VAVT is
network learns to represent the principal subspace of the input at the output 29 the SVD of the data covariance, the solutions satisfy:
If
M = QJ ALT m.n QT,
W = Q
F = P
Qm VT.
(35)
where mm E m rectangular identity matrix with [m,, = 8/, and Q E Rmx is an orthonormal matrix that accounts for the rotational degeneracy of the network. Stability of this network has been previously studied (29
C.1 Average dy namicss near fixed points
The online update rule is:
AW = "yx W)
(36)
AM n(yy" M)
Similar to we find it convenient to change coordinates to = (F, M). The average update at point 0 n+n (F Nr, M + NM) near the manifold of solutions becomes:
A ( M No Ep - FFTF) (AM)a @+n Np E, FT + +F.N Nar
- NF - FNT
(37)
where the averages are performed with respect to samples, and terms of O[[)|² and higher are ignored. The right side of the above equations be written in terms of the Jacobian of the average dynamics, H. The eigenvalue equation for the Jacobian is Hn = Agn, where n = (Np, Nar) and λи are the eigenvectors and the eigenvalues respectively. In terms of Ne and NM. this equivalent to the following matrix equations:
- 'N FTF N ENTF= N N. FT +F,N - NH
(38)
17

---

It is straightforward to show that H is diagonalizable with eigenvectors in four subspaces represented by nix 7m, n2 and
Am+e Ay
Thk (KH. (KKK"FF,,0), λн
dim((rx) - mm( (m)
(39)
nmm (0, A== = dim(nm) =m²
n, -S + 2y)s i2 j, dy 2, dim(n,,) mmmm + 1)
2
t= - F, (A. Sr) r> 8, Aн 0, dim(t)= m(m 1)
In the above, $5.,521 E R are symmetric and skew-symmetric matrices respectively, M e Rmxm, and K K E (=)((=-)) are two arbitrary matrices, and Fi E [R(n=m)nn is a full-rank matrix whose row space is orthogonal to that of F. (,, r, E [m]] and observe that all eigenvalues are real and non-negative. Additionally, it is easy to verify that the above cigenvectors do not form an orthogonal basis for the space This implics that the Jacobian of the average dynamics is not symmetric, and therefore the average dynamics do not follow pure gradient flow (i.c. they are not curl-free).
[n - ml). First, we
C.1.1 Non-orthogonal projections
The non-orthogonality of the eigenbasis for H is main difference between the Similarity Matching network and other networks studied the paper. This complicates the dynamics as the evolution of perturbations along tangent and normal subspaces may not be independent. However, as we will see below, we can use non-orthogonal projections to decouple the dynamics.
Let N E R contain the eigenvectors of H as columns, where K = mn + vn is the dimension of the parameter space. In this space, let E R.K be an arbitrary deviation from the solution manifold. Since N is full-rank, we can have the following decomposition:
n Nx.
(40)
where π E IR contains the coefficients of non- non-orthogonal projections. From the eigenvalue equation for H, the average update (i.e. averaged over all data) for n. becomes:
An - - Hn
HNT AN
(41)
where has X 's's as diagonal entries, and in the rightmost equality we used HN = NA. Addi- tionally, by differentiating Eq. 440, we can also represent the update An NAr. Equating this expression with Eq. 41 and multiplying from the left side by (N" NN--NNN-N we obtain:
AT
(42)
Since A is diagonal, the dynamics for n. decoupled. This motivates using the non-orthogonal pro- jection for studying the dynamics. The above shows, for example, that coordinates ग. corresponding to λи = 0 (tangent space) have purely difffusive dynamics with no flows. Similarly, non-negative λи suggests mean-reverting dynamics for other coordinates.
Since N is known in our problem (see Eq, 139, we can, in principle, find projections coefficients π for arbitrary n by multiplying Eq, 40 from the left side by N)- to get: π NN n. However, since each vector in this equation is a vectorized version of pair of weight matrices (F, M), this could lead to tedious calculations in terms the matrices. Instead, we use heuristics to find for our problem. The right hand side of Eq. 40 can be restructured into:
n NET: Nm + N. + TTę
(43)
where N& gKK (n)) has eigenvectors associated with subspace n. columns, Th (n)) has the corresponding projection coefficients, and similarly for other subspaces. The coefficients vectors nh, nu, n. and ,, could further be indexed according to the defined coordinates within
"This be verified by the definition of the Fuclidean inner product between two vectors, (Ar, As)) and = (Br, Bar in the parameter space, which can written in terms of (F, (. pair as: (a. b) te A Br) tr (5, Bx).
18

---

TH. each subspace, e.g. "k TH etc. One can also write the above in terms of the weight matrices, which leads two matrix equations associated with F and M, respectively. If (Nr, Nas). the equation corresponding to NE becomes:
NF
Σ H.P
k K
F
Σ R J F Σ P
(44)
Interestingly, the above decomposition does not involve the M component of the parameters, and is self-contained terms of the F component. Hence, the associated coefficients can be found from the standard matrix perturbation results. This leads to the following coefficients, which can be easily verified through simple algebra:
tr (K ttr(K (LN))) T
1
to tr(suFFFT)
At -
1 tr tr(12*FN7 V2
/2(1+
(45)
The first set of coefficients ( essentially the Euclidean projections the subspace nR- In retrospect, this makes sense as this subspace is orthogonal to other subspaces. In contrast, the equations for r and n, deviate from orthogonal Euclidean projections.
C.2 Fluctuations
We ready calculate the fluctuation terms caused by online learning noise. The sample updates on the manifold can be derived by substituting M and W from Eq. 35 into Eq. 36 and performing change of variables to get:
A. (2) AF (x)) n Fax (I F) M M FIj
g. (a)
(46)
AM, (x) = (m)la = ny( FFxT FT M)
where the subscript to denote the quantities calculated on the manifold of solutions. As saw above, the projection onto n2 subspace could calculated according to orthogonal Euclidean projection. This leads to:
proj(g- nkk") = ur (AF.())TKPPE) = -i((:: at F M MMKKK
FuaTnt (47)
A
In the above, we used the fact that we can write KHD F1 = q, oT u The dynamics along n& is mean-reverting with eigenvalues Xl."" = 1 fluctuation covariance:
Am++ An
Substituting these into Eq. 20 results in the
n(22= m+t) M 1..
Mdm+y Amln) 1 E [1, v [1, n- m]
((pmin3)) =
(48)
2Aμ
Ap
It is easy to show from Eq. 45 that the projection of g- onto n. and t are zero, and onto N. is non-zero. However, the deviation along Nym does not induce any tangential diffusion. This is because AF(x))
1 Fx | (this follows from calculating the sample update at point M
(F, M M) and performing algebraic simplification). Replacing this term in Eq. 45 obtain the tangential projection leads to ,, +
= 0. Hence, we will skip calculating fluctuations in subspace n.n-
C.3 Diffusion
Next, we proceed to calculate the diffusion into the tangent space. As we saw in the previous section, the relevant fluctuations are along n& subspace. Approximation of the sample update near the manifold shows:
AM l (x) = Fxxx Fr - M + pFax" No. + PNexx FT)
(49)
Al (x) 2o 1MMMi1(Faaa (Fxx" (I, FTF) + NN NNxx (In - F" F FF (FT NF + F))
19

---

where NF
KI.,FI, and terms of Op²2) and higher order are ignored. The non-orthogonal projection (which we denote by proj") onto the tangent space can be derived by calculating the corresponding TT from Eq. 45 to have:
proj" (g(2) p+pp ((r)
1
Lr (Q**)FFT)
p.m+u
Is
6
Pr X
6,
(50)
v2
This form is very similar to Oja's s case (see Eq.
32 Thus, the pairwise diffusion can be derived by summing up the average of the squared term above over all directions of the n2 and weighting each contribution by the corresponding fluctuation covariance in that direction. This yields:
Drr
Σ
Amtul
1 5 + 57) ((oh)))))))
1
(51)
Ag
Ar
ME [m]]
ME EE(2-mm) mm]
11º
1
Am+u
Am+ul
8 +
An
2
Am Ap
ME [m] ME(n-m)
173
1
X m+v
162aAr
Amtr A
Amizz A.
ME(N-m]
D Autoencoder
Here, we study two-layer linear autoencoder with weights U E RR and W E Raxp, The input x E R" and the network prediction at the output is ÿ WUv R We are interested in quantifying the drift of representation in the hidden layer h E R The learning occurs with SGD and batch size of one. The associated sample loss
(xx, y: 0) = - WUx|
(52)
where the reconstruction requires y = x. We are interested in the case where the hidden layer is bottleneck, ie. p In this case, the solutions satisfy:
Ip 0
WÙ
(53)
nX
where I, E PP is an identity matrix. Let the SVD of the input covariance be Ex = VSQ". The balanced solutions, where both weights have the same scale, satisfy:
Ú = Q.Q.IV''' VT, W W
(54)
Here, Ip.n E RPxx is a rectangular identity matrix with [p..li.j = 8/, and Q E Rxx and V Ruxn are two orthonormal matrices. Note that, the solution, the p-principal subspace of data represented in the hidden layer. Hence, the task irrelevant subspace here is (n p)-dimensional.
D.1 Hessian
The updates follow stochastic gradient descent i.e. A0 = (a; 0) (Gu, Gw), where Gy and Gw gradient matrices and can be derived by analytical differentiation of the loss:
{
Gw = (WU - In I=) - GT=
(55)
In I=)
By approximating the average gradient near solution point (W. Ú), we can form a set of Hessian equations Hn = λHn, where = vec(Nw) + vee No) = (Nw. Nu), for weight matrices Nw E =xx and Nu E Rpxn and eigenvalues λи. This leads the following matrix equations:
- 1.),NN + (Nw + (YND),, = Nw {
(56)
NWU + WT + WN = NH
20

---

Similar to the previous networks, we can write arbitrary deviation from the solution manifold as the following.
80 = mh n22 #n3 + t
(57)
ni (aK](KK (KWI)) KK"
n2 = (Z WT n2 = WT t- Ω 'W"
Importantly, the Hessian becomes diagonal in the above orthogonal coordinates. The detailed coordinates for each subspace and the associated Hessian eigenvalues are mentioned below. is straightforward algehra verify that each solution satisfies Eq. 56
Subspace n2
Nw = aND Nu KHO01 = qu where [K** = 91p08, We lp).v E In pl. X Huy = Aptu(1 - app)), dim(n1) = 2p(n - p)
In the above, K (Px(n-P), and Ú1 p(n-p)xn is full-rank matrix whose space is orthogonal to that of U. Also, recall that qi and v; the columns of orthonormal matrices in Eq. [54] Replacing the above in the Hessian equations, yields in the following characteristic quadratic equation for anp.
Aptoage (Ap Apte)a Aptu 0,
(58)
(-(Ap- \ptu)= + V(2, Aptu)² 422
with solutions:
ap,
2xptv
Note that normally there are two solutions to the above, which leads to two different eigenvectors for each pair (H,
Subspace n²:
Nw ZWW,, Nu WT z where Z = vooe i [p]
N 2ds dim(n2) =p²
Subspace n3:
=w = SWdW, Nu -NW where SiJ dioo tu,o,, e
X = 0, dim(na) = p(p+1)
2
Subspace t: (tangent space)
Tu U where n 1 4. -
Tw TT.
q.q.))
r,se [p]
X dim(t) p(p-1)
2
In the above, subspace n2 corresponds deviation from the balanced solution; movements along this subspace scale the weights W and U inversely, such that their product remains fixed. Hence, this is technically also tangent space. As interested in the drift associated with rotational symmetry, we only calculate the drift in the tangential subspace t.
D.2 Fluctuation
By replacing (W, Ú) into Eq. 55 the sample gradient on the solution manifold becomes:
Gw - xxx U
g. (x: θ) =
(59)
GU 0
(I, is the projection operator onto the rask-irrelevant subspace). The inner product of two vectors and b can be written in terms of traces of the corresponding weight matrices. This means (a, b)
21

---

Bw) + tr Bu). Using this equation and some linear algebra, it is straightforward to show that 9. has non-zero projection on n2, while projections on n2 and n2 and t zero. After normalization, the projection onto ny becomes:
proj(n 1. 9--) = Cu
(60)
V
where an functions of Ay and Aptu shown in Eq. 58 the covariance of fluctuations in subspace nz
Replacing this into Eq. 20 get
2a Ap ptt) 2(1 (1 (pp,,))
(pp.))
n. 1 aZ
(61)
2
This is the fluctuation of the task-irrelevant representations toward task-relevant ones.
D.3 Diffusion
To find the diffusion induced by the fluctuations in subspace nı, we have to approximate the gradient in Eq. 55 at a point 0 + pml /u. near the manifold and project that onto the tangent space (f). After some algebra, the projection to the leading order in p becomes:
proj(g [x; + pn? ), 1,, = p ( - -x,5%)
(62)
In reference to Eq. 22 the (1)(()) coefficients become:
1 1N. 4 1 x2
(63)
Subsequently, the the pairwise diffusion rates Drr, for [p), becomes:
2 ((G2))))
Der Σ
(64)
MElμ] pE[n-pl
112
na uus
Σ
16 (2 + 1,0%)1
2(1 + a (1 amws))
iE(1.2 nE(p) wE(n-p)
n1
of
Σ
Σ
Aptudrdal
+
64
(1 2 (1 (g.uz))
+ oz , aras)
(E(1.2) vE(a-p]
(op.e,s, E (1,2) 2} are the two solutions for a - see Eq.
58
The above could be written in a more
compact form:
Dor
Σ
(f Aptv) + f(Ar, )ptu))
64
wE(n p)
where f(A Aptu)= Eec ((a)=0 (I+o")[loj- and Q(a) Apttei + (2y, Aptula Aptu.
2²
22

---

E Two-Layer Network (Supervised)
Here, we study a linear two-layer network trained with supervised learning. The network output prediction is y WUx, x. where W R_xx,, and U IR are two weight matrices (m, n & p). The network is trained with a teacher that dictates the relationship the output (P E Rmxx)). We are interested in the drift of representations in the hidden layer h I&? at the steady state. Specifically, we would like to study how the stimuli that are task-relevan and irrelevant contribute to drift. Note that here, the task-irrelevant stimuli are the ones that lie in the null-space of P. and hence this is more general case than the principal subspace studied for other networks. Additionally, we allow for the possibility that the mapping P is low-rank, and denote its rank with - rank P)) < m. This implies that the task-irrelevan data lie in the (n k))- dimensional null-space of P. For simplicity, we assume that the non-zero singular values of P are x equal one. This allows us express it as P RI VT, where R E R R and V R=**
Px between the input and
are two arbitrary orthonormal matrices, and I m.n - & Jn,n denotes rectangular identity matrix whose first diagonal entries are one, and all others are zero. Finally, note that in [25], drift in a two-layer autoencoder with an expansive layer was studied. Our setup is similar to that work but is more general, as the task is not limited to reconstructing the input.
The sample gradient can be obtained by differentiating the MSE loss for the two-layer network:
{
Gw = (WU P) U + W Gu = WT P)xx" +
(65)
where y is the weight-decay coefficient. Correspondingly, the average gradient is:
{
(Gw) (WU- P)E, + (Gu) = (WU P)E +
(66)
where the averages (.) are taken with respect to the data. We are first interested in finding the manifold of solution, i.. set of (W, Ú) that satisfy (Gw) = 0 and (Gv) = 0 simultaneously. To do so, we first perform the following change of variables "primed" weight matrices:
W
U
(67)
where W' E R and U E RR and can be considered as weights of a reduced two-layer network (see below). Replacing the above in Eq. 66 leads to the corresponding gradient equations for the primed variables:
{ G (W'U -.), +,W" G W' (W'U E, +
(68)
Here, Is E Rlck is identity matrix, and Σx In,n V. VIn, is the effective covariance in the primed coordinates. These equations correspond to a reduced network, which is a two-layer expansive autoencoder with input covariance Σ'= E IR* *x (this sub-network dealing with the task-relevant portion of the data). The manifold of solutions for this two-layer expansive autoencoder has been previously derived in (33,, and satisfies: W'A' = I - -- E/1 and O' - WT, From these, the solutions to the non-primed variables can be derived by the change of variables of Eq. 67
be interpreted as
We will solve drift for Gaussian data a ~ NO Ex). Similar to other networks, we assume the cigenvalues of the input covariance in the task-relevant and task iiseevneelvvntt subspaces are A 1 and rT, respectively. However, note that here the task is determined by P, and the multiplicity of X and A_ are k and subspace task, a2 is not restricted to be smaller than Åy|- This data structure simplifies the effective covariance in Eq. 68 to be Σ' = I This leads to the weight solutions in the primed coordinates to be W = v 7 IR. QT and U = vT - 7Qp where Qp E RPXP is an arbitrary orthonormal matrix. Finally, by the change of variables in Eq. 67 the manifold of solutions for the original network becomes:
k. respectively. Additionally, and unlike in previous cases of the principal
W = VI 111--RAN RII
Ú 11
(69)
23

---

E.1 Hessian
By expanding the average gradient (Eq.
66) near a solution point (W, U). and similar to the autoencoder case, we can form a set of Hessian equations Hn λnn, where = (ec(Nw) - vec(NN) = (Nw. Nu). for weight matrices Nw R and Nu R
PNU + (NwU WNO W + = Nu
(70)
NN P W (NWÜ W WNu yNu NH One can show that the Hessian is diagonalized in the coordinates discussed below.
Subspace ni1= Nw = 0, No. KKIPP P1 = 9 where [K*** 4ip0g u [k], UE E [n - kl), dim(n1)) = k(n k)
H -
In the above K E (n-((-k)) and P1 R(n-k)xn is full-rank matrix whose row space is orthogonal to that of P. Also note that in the above and the rest of the section, and ü; are columns of matrices Q and V respectively.
Subspace 1222
Nw = 0, N K
P1 = qu Ukte: where [K4.) liy = goudz
M + 1.pl, v [n n-h k))
N/ = 7 dim((n2) = (p kn - k)
Subspace n3::
Nw = RIm NW. Nu = -NN k.. VT. where Nw = SW + KW
k(k -
A = 27, dim(nx)) = kp 2
In the above, S E R Rixk is a symmetric matrix, K E R (p-k) an arbitrary matrix, and W E xixp is full-rank matrix whose row-space is orthogonal to that of W
Subspace n.a:
Nw = Rl WWW Nu = W Z I
z E kR
i.j [A] dim(ns) 222
where Z consist of three subgroups:
zio 2 - Z,,T
(1>)) 2/11 (212)] + +2z2,))
d 2(1 -7)
Zij
2/1 - (2/2) 2,2(²) Aн =2
(i>i)
and set of 2; E & form an orthonormal basis for R*
Subspace Tl::
N = - Nu 0, where la E [k), v E [m - k)),
H =, dim(n)s) = k(m - k)
W
(in the above, r; are the columns of the orthonormal matrix R).
Subspace nsg
N Tk-uqT Nu =, where μ E [K: + L,pl, E [m - k],
Xua 7, dim(nes) (p k) k)
Subspace t;: (tangent space)
Tw = W To - - where M 1
2 49 q,
dim(ti) k(k 1(k-1)
x 0
24

---

Subspace tz: (tangent space)
Tw Tutt
To q q where μ E [k], (p k] H =0 dim(tz) k(p-k) k)
The first tangent space (t1) corresponds to rotation of representations within the row-space of W, while the second one (tz) corresponds to rotation toward the (p- dimdiemnsio subspace orthogonal to the row-space of W.
E.2 Fluctuations
Sample gradient on the manifold of solution can be calculated from Eq. 65
Gw r(W -Pxx"
(a:; 0)
(71)
GU 7( "'xx"
The projection on subspace n is:
proj(g.. -
(72)
where the indices here are with respect to the columns of matrix V. i.. In == 8% x, etc. The above
projection leads to the following covariance for the fluctuations (using Eq. 20
(Pw)) 7n 2
7n2
\1 (1(1-7)
(2
(73)
2 (117))+
We can also show that there is non-zero projection on subspace nh::
proj(g., n V2V - x7). proj(g., n - - yx,Ey (i
(74)
Projections other subspaces zero.
E.3 Diffusion
To find the diffusion induced by the fluctuations in subspace n, we have to approximate the gradient Eq. 65 at a point Ø + pm! '.2 near the manifold and project that onto the tangent spaces. Since for subspace 11, we had n = (0, Nv), the approximate gradient becomes:
Gw p+pp!!" (İ + +Nu) - P)xx" +NNU) + +
(75)
7W P.x (TT)OO))a + pW NGwx TÜT -U P
Gu @+p W (W((W( + +NN) P)æx" +7 +NpNo
2 >ū - Pxx""
+ p(W Pwap""" Nuxx +7Nu).
In the above, ignored terms of Opp²) and higher orders. Replacing No." = qu kkt in the above and performing the projection leads to:
proj(g(æ; (x; 0 pmft"), ,tat") " x2
(76)
(Projection onto t2 is zero). Correspondingly, the pairwise diffusion coefficients become:
D
Σ
(19)))))))))
T,8 E [k)), *
(77)
2
VE
k]
1222
n22a4 (n - k)
Σ
16(1
32 (1 + +
vE[n-k]
We also have diffusion term that is caused by the fluctuations in the n. subspace. This corresponds to diffusion of expansive autoencoder with input dimension k and an effective isotropic data, which has been shown previously to be (25
(3 +
D
(78)
16(1 7)
25

---

Hence, total diffusion of the representation for stimulus becomes:
n D.
(n - k)
(1 (111))
(k 1)(k: +
(79)
16(1 7) n (A: (): +2+ (n -k)At),
2
A1 1)
) د
16
2
This summarizes the derivation of drift for two-layer supervised setup with teacher signal y Pa. Note that, unlike the principal subspace tracking networks, here A the input variance in the null-space of P. and is not restricted to be smaller than λlj (Figure(SIp)) Additionally, we see that in this case drift does not depend on the output dimension, but only on the input dimension and k which is the rank of P (Figure S.p.c..
b)
C)
0.8
6.8
0.8
cool Lib h.))
rael (Daa A.))
6.6
0.4
cool
02
0.2
10 0.2
10"
10
Figure S1: Simulations of drift in the supervised two-layer network. In all the plots, the cosine similarity decay of task-relevant representation is shown over time. a) For given P, drift rate increases with the task-irrelevant variance r1. Entries of P are chosen randomly from a Gaussian distribution followed by rescaling set the maximum singular value to be one. The task-irrelevan subspace is the null-space of P, and \li 1. b,c) Drift rate changes with (rank of P, panel b) and (output dimension, panel c). Here, the singular values of P and the input covariance are
one. Simulation parameters for each panel are a: = 50, 70, k m 30, 0.02, = 0.2,
b: 50,y p 70, m 40, and
50, p 70, k 30.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction present the scope and main contributions of the paper. These are further supported in the main paper (Sections 2-5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are stated at the penultimate paragraph of the discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

For the new theoretical results, full derivations and assumptions are stated in the Appendix. Further assumptions and limitations are stated in the last paragraph of Section 6.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the relevant parameters for reproducing the simulations are stated clearly in the main text or the figure caption specific to each result.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the codes and core functions to reproduce the simulations are made available the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We use online learning all of the simulations. This, along with the data or data distributions used for different experiments stated in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: When applicable, we have reported error bars for the measurements on the each plot, and mentioned the corresponding method in the figure caption. Figures 3 and 6a, some of the error bars may not be noticeable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
For estimation of drift rates in simulations for real and toy datasets, multiple parallel instances of the training process. Experiments were executed on local institution's 's internal CPU cluster, with total estimated compute time of approximately 1,000 CPU core-hours The maximum memory usage was 64GB.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research involves theoretical studying of representational stability in neural networks. does not involve human subjects or ethically sensitive applications. Hence, we assume it conforms to the NeurlPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
30

---

Since the paper involves understanding the phenomenon of stability in neural networks and the brain, which is foundational in nature, no immediate positive negative societal impact is foreseen.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not foresee such a risk posed from the paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the publicly available MNIST data in our study. An accompanying citation to the original paper provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
31

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets were produced in this research.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsoureing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects, therefore IRB approval not applicable.
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The research here does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "WnG8lWouMd", "reviewer": "Reviewer_eXNF", "summary": "In this article, the authors analytically study drift of neural representation under continual learning of stimuli with 'non-relevant' directions in linear networks. They provide a general framework to study and quantify the phenomenon, and apply it to a variety of setups. They find good agreement with numerical simulations, and differentiate the resulting dynamics from drift induced by explicit noise added to the weights.", "questions": "See above.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The article addresses a relevant question, and provides explicit and understandable results. The selection of learning setups is sensible and comprehensive.\n## Minor Weaknesses\n2. When introducing the local coordinates for the main SDE (4), the authors could improve the clarity by clearly explaining what the expansion point is, and being more explicit about the assumption underlying the equations.\n3. The results for non-linear models could be extended by investigating how the input spectrum determines the drift rate, and how this compares to the linear case.\n4. Representational drift and its geometry has been studied and characterized in various cortical areas. A brief discussion on which aspects of the experimental observations are consistent with the effects seen in the model and which ones are not would be a valuable addition.", "was_revised": true, "final_justification": "My points were addressed, I believe the results of this paper, while limited in scope, are novel and interesting. I recommend acceptance.", "rebuttal": "We appreciate the reviewer’s positive assessment of our work. Please find specific responses below.\n\n1. **Clarifications on SDE of Eq.4:**\nWe are happy to provide clarifications regarding the assumptions underlying this set of equations. First, these equations are based on approximating the discrete learning update by continuous-time stochastic differential equations, which has been widely used in the context of SGD (e.g. Ref. 1). The main assumption behind this approximation is a small learning rate. We apply this approximation around a stable point on the manifold of solutions, $\\tilde{\\theta}$, where the average loss is zero and the Hessian spectrum is non-negative. When there is an explicit loss associated with the network, the Hessian becomes symmetric and can be block-diagonalized into subspaces with positive and zero eigenvalues. As a result, the dynamics can be decomposed into normal and tangent spaces leading to fast and slow dynamics (e.g. Ref. 2). We found Hessian to be symmetric in all the models except Similarity Matching (for which there’s no explicit loss function). However, we were able to perform a similar decomposition of dynamics using non-orthogonal projections (see Appendix C). Furthermore, our calculation of drift in the tangent space is based on the assumption of small fluctuations around the manifold, which allows us to calculate the diffusion into the tangent space by integrating the first order term of the deviation. When the deviation is large from the manifold, one needs to include contributions from second and higher order terms to the drift.\n\n2. **Dependency on the spectrum in nonlinear case:** \nThank you for this suggestion. We believe that deriving a closed-form equation for the drift as a function of the input spectrum is in general challenging for nonlinear networks. This is mainly because there are often no closed-form solutions in the weight space for nonlinear networks. However, given such solutions, the same methodology could be applied to derive the drift. As was demonstrated in the nonlinear network example, the general relationship – that drift increases as a function of the spectrum’s magnitude in the task-irrelevant subspace– holds. However, the exact dependence on the spectrum may depend on the type of solution amongst other parameters. We would also like to note that even in our linear cases, the theoretical dependencies turned out to be rather complicated in terms of the whole spectrum (see e.g. Eqs 6, 8 and 9), and only simplified under more structured input spectra.\n\n3. **Relation to cortical drift in experiments:**\nThis is an excellent question. Even though drift has been observed across various cortical areas and under different experimental designs, we did not find any studies that specifically examined the influence of task-irrelevant stimuli. However, we can speculate about potential connections to some existing findings. In one related study (Ref 3), the amount of drift observed in an association area – where different types of stimuli need to be multiplexed – appears to be higher than the drift at the origin areas. This might be related to the phenomenon discussed in our study as different sets of stimuli are combined with varying levels of relevance depending on the task.  Additionally, a potentially related observation comes from the visual cortex, where more complex and naturalistic stimuli show higher drift compared to simpler stimuli (Ref 4). This could also relate to our finding that, in a network with a limited representational capacity (such as the bottleneck in our work), a more complex stimulus is effectively equivalent to activating a higher number of input neurons, resulting in a greater amount of “task-irrelevant” content, and consequently a higher drift. As mentioned in the discussion, a more controlled test would be a setup in which the amount of distracting background stimuli can be systematically varied. This could be done, for example, in a vision task of the type used in the first study above, or in an olfactory target detection task with varying levels of background stimuli.\n\nReferences:\n\nRef 1: Stephan Mandt, Matthew D Hoffman, and David M Blei. arXiv preprint arXiv:1704.04289, 2017.\n\nRef 2: Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. ICLR, 2022.\n\nRef 3:  Franco, L.M., Goard, M.J. Nat Commun 15, 6872 (2024).\n\nRef 4: Marks, T.D., Goard, M.J. Nat Commun 12, 5169 (2021)."}, {"review_id": "zcThdjqEgf", "reviewer": "Reviewer_RtGY", "summary": "- The paper aims to understand the mechanisms of representational drift from a dual perspective of Hebbian and SGD-based learning.\n- The authors present theory and experiments in both settings to show that task-irrelevant stimuli, which agents learn to ignore, can be sufficient to create long-term drift of representations of task-relevant stimuli.\n- Through a motivating example of Hebbian-based learning, the authors show that drift rate increases with the variance and dimensionality of the data in the task-irrelevant subspace.\n    - The authors show similar results using a non-linear network and SGD-based learning in the experiments section.\n- With the assumption of continuous-time dynamics, the paper considers that a point on the solution manifold can be decomposed into local normal and tangential spaces. The authors particularly focus on late phase learning where the dynamics in the tangential space are purely diffusive.\n- The authors derive the analytic form of angular diffusion in terms of task-irrelevant subspace properties in four settings: multidimensional Oja's network, similarity matching network, autoencoder, and a two-layer network.\n- Using Gaussian toy data and MNIST, the authors show that drift rates closely match those predicted by the paper's theory.\n- Finally, the authors contrast the dynamics under learning noise and external synaptic noise, showing that they have different characteristics.", "questions": "- Can a more general class of tasks be reduced to the problem of principal subspace tracking? If so, what is this class of tasks and how general is it?\n- Beyond the observation that the two sources of noise have different impacts on the drift rate, are there any other takeaways from Section 5 comparing learning noise and synaptic noise?\n- Typography (no impact on score)\n    - Line 85: missing word: \"we systematically ___ this\"\n    - Line 94: missing space before $\\Delta$.", "limitations": "The paper does not discuss the limitation that it only considers the task of principal subspace tracking.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n- Quality\n    - The paper is motivated by citing relevant work to show that a similar notion of representational drift exists in neuroscience and machine learning.\n    - Both Hebbian and SGD-based learning settings are studied to support the generality of the presented principles.\n- Clarity\n    - The paper is mostly clear and easy to follow.\n- Originality\n    - The analysis performed in the paper is novel.\n\n### Weaknesses\n\n- Quality\n    - The central object of this paper, the notion of representational drift, needs to be more explicitly and rigorously defined, at least in the context of machine learning. Currently, part of this definition has to be inferred from the theoretical and experimental setups.\n    - The notion of \"task\" is discussed very generally throughout the paper. However, a majority of the analysis is specific to the task of principal subspace tracking, so the generality of the notion of task-relevance and task-irrelevance is not supported. In general, depending on the task, task-irrelevant stimuli can have higher variance and dimensionality.\n        - If more general tasks can be reduced to the studied problem of learning principal subspaces, this reduction procedure needs to be described.\n- Significance\n    - The limited theoretical and empirical setup of the paper makes it difficult to evaluate its significance for the community.", "was_revised": true, "final_justification": "The authors engaged in the discussion to clarify some of the points raised in my review. Two notable elements of this discussion are summarized below:\n\n- During the discussion, the authors provided a clearer definition of representational drift. I suggest the authors use a more formal definition to simultaneously introduce some of the notation they use in the paper's analyses.\n- One of my main concerns was that the notion of \"task\" is discussed very generally throughout the paper, but the analysis presented is limited to the specific task of principal subspace tracking. In general, depending on the task, task-irrelevant stimuli can have higher variance and dimensionality. The authors explained how the supervised learning analysis in Appendix E can be presented more generally, where the task-relevant components are not necessarily the principal components. They also plan to include empirical analysis for this more general notion of \"task.\" \n\nAlthough I don't see a strong reason why the general messages regarding late phase learning representation drift would cease to hold in practical extended settings, the overall limited theoretical and empirical setup of the paper is still a concern when determining significance.\n\nBased on these points, I increase my score to 4 (Borderline accept).", "rebuttal": "We greatly appreciate the reviewer’s thorough assessment of our work. Please find responses to specific criticisms below.\n\n1. **Generality of the results and the notion of task-irrelevancy:**\nWe agree with the reviewer that in a great portion of the paper we used principal subspace tracking as the running task – this enabled us to study drift under different architectures and learning rules for the same task, which was a novel contribution of our work. However, as evidenced in the supervised learning setup, our work goes beyond principal subspace tracking task. In this setting, the task is determined by the teacher with $y = Px$, where $P$ is an arbitrary mapping from input to output. Here, task-irrelevant stimuli are those that lie in the null-space of P, for which the associated $y$ is $0$ (notably, this definition is consistent with the principal subspace task). In the two-layer supervised network presented in the main text, we intentionally designed $P$ to represent the principal subspace to provide a fair comparison with other networks. However, in the calculations provided in Appendix E for this network, there is no assumption restricting $P$ to represent the principal subspace. Indeed, the results hold for a more general mapping where the task-relevant subspace can be any arbitrary subspace of the input. In this case, the task-irrelevant induced drift appears as an extra term that is $\\propto (n-k)\\lambda_{\\perp}$, where $k$ is the rank of $P$ (which in general could be less than the output dimension $m$), and $\\lambda_{\\perp}$ is the variance in the task-irrelevant dimension (i.e. the null-space of $P$). Importantly, $\\lambda_{\\perp}$ can have a variance higher than that of the task-relevant subspace. We are happy to clarify this point and add new text and plots to further corroborate this result. We thank the reviewer for this comment, which allows us to better clarify the scope of our work.\n\n\n2. **Comparison of learning noise and synaptic noise:**\nThe key point that we aimed to emphasize in Section 5 is that the learning noise can lead to qualitatively different drift compared to synaptic noise, even when all other parameters are held constant. This is because, for a given network, the subset of solutions that end up being explored depends on the dynamics of learning and the type of noise present during it. Additive Gaussian synaptic noise – which is a common way of inducing drift in the literature –  creates an isotropic noise profile in the parameter space. In contrast, the profile associated with learning noise can be anisotropic and position dependent in the parameter space. In our setup with Oja’s network, this distinction manifests not only in the overall rate of drift for a given stimulus, but also in the geometry of drift within the representation space. By this, we mean the relative rates at which representations of different stimuli drift along each other. We see that in Oja’s case, the synaptic noise leads to isotropic drift in the representation space with a pairwise drift rate of $D_{sr}=\\eta\\sigma_{syn}^2/4$, that is direction-independent (see line 225). By contrast, the corresponding learning noise results in variable drift rates between representations of two stimuli, depending on the input spectrum (see Eq.6). The difference in the drift geometry, and not only the overall drift rate, may be another experimentally measurable aspect of drift that can distinguish these two sources of noise. We thank the reviewer for prompting us to clarify this point.\n\n\n3. **Definition of representational drift:**\nWe share the reviewer’s sentiment regarding the need for a clear and widely accepted definition of representational drift, both in neuroscience and machine learning. In our work, we take drift to be simply the change in the internal representations of the network that occur without any obvious change in behaviour or task performance. In this context, it can refer to a late-stage learning where the performance (loss) is fixed on average. However, one can also imagine late stages of learning where certain types of solutions are gradually favored over others (e.g. slow transition to flatter or more generalizable solutions). In these situations, even though the training loss is not changing, the generalization may be improving."}, {"review_id": "TqEbzicgtN", "reviewer": "Reviewer_6Soi", "summary": "Representational drift occurs in neural networks, yet its mechanisms remain unclear. This study investigates how task-irrelevant stimuli contribute to drift in neural representations of task-relevant inputs, even when these irrelevant stimuli are learned to be ignored. Through theoretical analysis and simulations across various architectures including Hebbian-based learning and stochastic gradient descent models, the research demonstrates that drift rate increases with both the variance and dimensionality of task-irrelevant data. These findings establish important connections between stimulus structure, task context, and learning mechanisms in representational drift, potentially offering new approaches to understand underlying neural information processing in the brain.", "questions": "1）Please further explain the rationale for using Eq. 2 to define the first m dimensions as task-relevant and the remaining n-m dimensions as task-irrelevant. Specifically, what kind of neural representation or feature is this \"task\" intended to learn?\n\n2）Please explain why in Eq. 2 all task-relevant dimension feature values are uniformly set to 1, while all task-irrelevant dimension feature values are set to the same constant less than 1?\n\n3）It is suggested to add the related works in the introduction about the existing contributions of task-irrelevant stimuli in continuous learning tasks, and clearly articulate the innovation points of this research.\n\n4）Regarding section 4.2, please elaborate on the task setting: the input consists of x1, x2, and n-2 dimensions of task-irrelevant Gaussian white noise, while the output is x1 and x2. How does this training method form receptive fields in the hidden layer? What is the mechanism? What role do task-irrelevant signals play, especially in reorganizing receptive fields?", "limitations": "Please see the weaknesses and questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths***: \nThis paper employs theoretical analysis and computational simulation methods to study how task-irrelevant stimuli promote neural representation drift during continuous learning, with solid results. \n\n***Weaknesses***:\n\n1) The theoretical analysis results in this paper are only applicable to linear networks, and the input-output functions can only be linear, having certain limitations.\n\n2) Both sections 4.2 and 4.3 indicate that task-irrelevant signals can cause neural representation drift during the learning process. What impact does this drift have on online learning performance?", "was_revised": true, "final_justification": "I have no major concerns with this work and believe it's a interesting contribution to the field. I support its acceptance.", "rebuttal": "We appreciate reviewer’s insightful feedback on our work. Please find our responses to specific criticism below.\n\n1. **Linear networks:** \nWe agree that our theoretical derivations were limited to linear networks. However, we would like to note that the learning dynamics we analyzed are highly nonlinear functions of the parameters. We believe that the fact that both SGD and Hebbian-based learning lead to drift of parameters due to task-irrelevant stimuli is a consequence of nonlinearity in the learning dynamics that is already captured in linear networks. Although extending the analytics to networks with nonlinear activations would be an interesting avenue for future work, we do not believe it changes the main findings. This is exemplified by the numerical results in the nonlinear network in Section 4.2.\n\n2. **Impact on online learning performance:**\nThat’s a great question. Our main finding is that even after minimum loss is achieved, the presence of task-irrelevant stimuli can perturb the network, potentially leading to overall drift. In general, this effective noise might provide implicit regularization effects, causing  the network to favor specialized solutions. This effect may be observed in generalization performance. In our setup, while the average online performance is fixed, short-term fluctuations in performance increase as a function of the extent of task-irrelevant stimuli.\n\n3. **Questions regarding spectrum in Eq.2:**\nIn this section, we aimed to demonstrate the phenomenon in a simplest setup where the amount of task-irrelevant stimuli can be directly controlled. This design of the input covariance allows Oja’s network with input dimension $n$ and output dimension $m$ to learn to represent the principal subspace at the output. Here, the principal subspace corresponds to the subspace associated with $\\lambda_{||}=1$, and any stimuli associated with $\\lambda_{\\perp}<1$ are task-irrelevant and vanish at the output layer. We show that even in this simple setup, the amount of drift increases with $\\lambda_{\\perp}$. In the later sections, we solve this for an arbitrary input covariance; hence, this motivating example serves as a special case of those results.\n\n4. **Literature on contribution of task-irrelevant stimuli to continuous learning:**\nWe thank the reviewer for this suggestion. We will add additional references related to continual learning, which often involves the flexible learning of multiple tasks in sequence without forgetting. Our finding is novel in that, even for a fixed task, the extent of ignored (task-irrelevant) stimuli affect the internal representations. This is applicable to many lifelong learning scenarios, where the agent is continuously learning from the environment. Understanding the nature of representation stability in these cases allows for better comparison of representations throughout learning and has important implications for areas such as interpretability research.\n\n5. **Explanation of the nonlinear network:**\nWe are happy to provide clarifications on this setup. With this example, we aimed to demonstrate that our main findings translate to a network with nonlinear activations. The input essentially consists of a task-relevant component, which denotes the position of an agent on a ring, as well as additional stimuli that are not relevant to the specific circuit (here represented by Gaussian stimuli). In general, the network can be viewed as performing computation on the task-relevant position variables. To accomplish this, the neurons must faithfully represent position on the ring in the middle layer, from which the downstream output is read out. Since neuron activations are non-negative, this leads to formation of localized receptive fields that tile the entire ring. We show that continuous training leads to repositioning of the receptor fields such that the tiling pattern and overall representation similarity are preserved (Fig 4b). Importantly, the extent of this depends on the extent of task-irrelevant stimuli. This is because the presence of those stimuli induces perturbations in learning, which can displace the system on the manifold of solutions and lead to a rearrangement of receptive fields. We would also like to note that, in the example, we chose reconstruction of position as the task, which causes the network to behave similarly to unsupervised non-negative similarity matching networks, such as ones seen in Ref. However, since our setup is supervised learning, the mapping on the task-relevant position variables can be chosen to be arbitrary, and we still see the same dependency. We are happy to provide additional plots along this line.\n\nRef: Qin, S., Farashahi, S., Lipshutz, D. et al. Nat Neurosci 26, 339–349 (2023)."}, {"review_id": "0N1ZVciNRm", "reviewer": "Reviewer_DyoC", "summary": "In this paper, the authors show how the task-irrelevant portion of input can lead to a drift in internal representations of networks even after the learning in the relevant subspace has stabilized. The authors derive theoretical results for such drift for several network architectures and learning rules, establishing its dependence on factors like the learning rate and the dimension of the task-irrelevant subspace. Numerical results are presented which agree closely with the theory.", "questions": "See strengths and weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written, and the ideas are presented in a way making them easy to follow. The motivating example is quite useful for developing intuition before jumping into the theory. I was unfamiliar with a fair bit of the background work, but the paper does an excellent job at providing enough pointers to not have to jump back and forth between references in order to understand the core ideas. Interestingly, even though the results are developed for linear networks, they seem to generalize to non-linear ones leaving venues for future investigations. The results show close correspondence between theory and simulations. Overall, I recommend the paper to be accepted. Some concerns:\n\n1. While the work provides insight into drift in representational learning, it likely has limited practical applications owing to the limitations authors acknowledge – simple linear networks, assumptions around small perturbations making a differential analysis possible.\n2. Minor: in Fig. 1 consider using another variable name for number of stimuli since m is used for the dimension of relevant subspace, “we systematically this” (line 85); line 128, the dimensions are different on the two sides of the equation $WW^T = I_m$.", "was_revised": true, "final_justification": "I want to thank the authors for addressing my minor concerns, and I maintain my favorable rating.", "rebuttal": "We appreciate the reviewer’s favorable evaluation of our work. Despite the mentioned limitations, we believe our work sheds light on some novel and non-trivial aspects of representational dynamics under continual learning, which emerged even under simple setups. These are due to the nonlinear aspects of the learning signal and the associated dynamics in the parameter space, which is known to exist even in linear networks. To our knowledge, our work is also the first to theoretically study drift under different architectures and learning rules, pointing out the commonalities and differences specific to each setup. This line of research can certainly be expanded upon in future studies. We also thank the reviewer for pointing out the typo in line 128. The correct expression for the weight matrix is $W \\in R^{m \\times n}$."}]

Author rebuttal:

Meta-review: The recommendation is to accept. 

This paper presents a novel and technically sound analysis showing that task-irrelevant stimuli are a significant source of representational drift. The reviewers reached a positive consensus, praising the paper's originality and the strong agreement between theory and simulation across different learning paradigms. The author-reviewer discussion was particularly effective. The authors provided convincing responses to initial concerns, especially regarding the generality of the "task," clarifying that the findings extend beyond principal subspace tracking.

For the camera-ready version, it's essential that the authors incorporate the changes promised in the rebuttal. Specifically, please add the clearer definition of representational drift upfront, expand on the generality of the task framework, and include the discussion connecting your findings to experimental observations of cortical drift.
Decision: Accept (poster)