Sufficient conditions for offline reactivation in recurrent neural networks
Authors: Nanda H Krishna
Colin Bredenberg
Daniel Levenstein
Blake Aaron Richards
Guillaume Lajoie
Venue: ICLR 2024 poster (2024)
Keywords: computational neuroscience
offline reactivation
replay
recurrent neural networks
path integration
noise
Primary area: applications to neuroscience & cognitive science
TL;DR: We present theory and experiments which suggest that task optimization in noisy neural circuits can be sufficient to produce offline reactivation phenomena.

Abstract: During periods of quiescence, such as sleep, neural activity in many brain circuits resembles that observed during periods of task engagement. However, the precise conditions under which task-optimized networks can autonomously reactivate the same network states responsible for online behavior is poorly understood. In this study, we develop a mathematical framework that outlines sufficient conditions for the emergence of neural reactivation in circuits that encode features of smoothly varying stimuli. We demonstrate mathematically that noisy recurrent networks optimized to track environmental state variables using change-based sensory information naturally develop denoising dynamics, which, in the absence of input, cause the network to revisit state configurations observed during periods of online activity. We validate our findings using numerical experiments on two canonical neuroscience tasks: spatial position estimation based on self-motion cues, and head direction estimation based on angular velocity cues. Overall, our work provides theoretical support for modeling offline reactivation as an emergent consequence of task optimization in noisy neural circuits.

Published as a conference paper at ICLR 2024
SUFFICIENT CONDITIONS FOR OFFLINE REACTIVATION IN RECURRENT NEURAL NETWORKS
Nanda H Krishna' 1,2,50
Colin Bredenberg 1,2.0
Daniel Levenstein 1,3
Blake Aaron Richards 1,345
Guillaume Lajoie 1,2,1
Mila Quebec AI Institute
Université de Montréal MMeGill University
Canada CIFAR Al Chair
SCIAAR Learning in Machines & Brains
(nanda, hww.shankar-krisshheee. colin. bredenberg, gui llaume. lajoie)@mill hatpe/eeeaa.uuuueeeeeee.a.eee
ABSTRACT
During periods of quiescence, such sleep, neural activity in many brain circuits resembles that observed during periods task engagement. However, the precise conditions under which task-optimized networks autonomously reactivate the same network states responsible for online behavior is poorly understood. In this study, we develop mathematical framework that outlines sufficient conditions for the emergence of neural reactivation in circuits that encode features of smoothly varying stimuli. We demonstrate mathematically that noisy recurrent networks optimized to track environmental state variables using change-based sensory in- formation naturally develop denoising dynamics, which, in the absence of input, cause the network to revisit state configurations observed during periods of online activity. We validate our findings using numerical experiments on two canonical neuroscience tasks: spatial position estimation based on self-motion cues, and head direction estimation based on angular velocity cues. Overall, our work provides theoretical support for modeling offline reactivation an emergent consequence task optimization in noisy neural circuits.
INTRODUCTION
It has been widely observed that neural circuits in the brain recapitulate task-like activity during periods of quiescence, such as sleep (Tingley & Peyrache, 2020). For example, the hippocampus "replays" sequences of represented spatial locations akin to behavioral trajectories during wakefulness (Nádasdy et al., 1999; Lee & Wilson, 2002; Foster, 2017). Furthermore, prefrontal (Euston et al., 2007; Peyrache et al., 2009), sensory (Kenet al., 2003; Xu et al., 2012), and motor (Hoffman McNaughton, 2002) cortices reactivate representations associated with recent experiences; and sleep activity in the anterior thalamus (Peyrache et al., 2015) and entorhinal cortex (Gardner al., 2022) is constrained to the same neural manifolds that represent head direction and spatial position in those circuits during wakefulness.
This neural reactivation phenomenon is thought to have number of functional benefits, including the formation of long term memories (Buzsáki, 1989; McClelland al., 1995), abstraction of general rules or "schema" (Lewis & Durrant, 201 1), and offline planning of future actions (Ólafsdóttir et al., 2015; Igata et al., 2021). Similarly, replay in artificial systems has been shown to be valuable in reinforcement learning, when training sparse or expensive (Mnih al., 2015), and in supervised learning, to prevent catastrophic forgetting in continual learning tasks (Hayes et al., 2019). However, where machine learning approaches tend to save sensory inputs from individual experiences in an external memory buffer, or external networks that are explicitly trained to generate artificial training data (Hayes & Kanan, 2022), reactivation the brain autonomously generated in the same circuits that operate during active perception and action. Currently, it is unknown how reactivation can emerge in the same networks that encode information during active behavior, or why is so widespread in neural circuits.
Previous approaches to modeling reactivation in neural circuits fall into two broad categories: genera- tive models that have been explicitly trained to reproduce realistic sensory inputs (Deperrois et al.,

---

Published as a conference paper at ICLR 2024
2022), and models in which replay is emergent consequence of the architecture of network models with a particular connectivity structure (Shen & McNaughton. 1996; Azizi et al., 2013; Milstein Litwin-Kuman & Doiron, 2014; Theodoni et al., 2018; 8; Haga & Fukai, 2018; Asabuki & Fukai, 2023). Other approaches have modeled replay and theta sequences or input modulation (Kang & De WeWess, 2019) in continuous attractor network models. Generative modeling approaches have strong theoretical guarantees that reactivation will occur, because networks are explicitly optimized to provide this functionality. However, modeling approaches that argue for emergent reactivation typically rely on empirical results, and lack rigorous mathematical justification.
et al., 2023) or local synaptic plasticity mechanism (Hopfield, 2010;
emergent consequences of firing rate adaptation (Chu et al., 2024)
In this study, we demonstrate that a certain type of reactivation diffusive reactivation can emerge from a system attempting to optimally encode features of its environment in the presence of internal noise. We trained continuous-time recurrent neural networks (RNNs) to optimally integrate and track perceptual variables based on sensations of change (motion through space, angular velocity, etc.), in the context of two ethologically relevant tasks: spatial navigation and head direction integration. Critically, training in this manner has been shown to produce grid cell (Cueva & Wei, 2018; Sorscher et al., 2019) and head direction cell representations (Cueva al., 2020; Uria et al., 2022), which correspond to neural systems in which reactivation phenomena have been observed (Gardner et al., 2022: Peyrache et al., 2015). We see that these networks exhibit reactivation during quiescent states (when subject to noise but in the absence of perceptual inputs), and we explain these phenomena by demonstrating that noise compensation dynamics naturally induce diffusion on task-relevant neural manifolds in optimally trained networks.
2
MATHEMATICAL RESULTS
2.1 SETUP AND VARIABLES
In this study, we will consider noisy discrete-time approximation of a continuous-time RNN, ds(t) receiving change- based information dt about an Ng-dimensional environmental state vector st)). The network's objective will be to reconstruct some function of these environmental state variables, [s((t): : R.N. + TR No, where N₂ is the number of stimulus dimensions and N₂ is the number of output dimensions (for a schematic, see Fig. la). An underlying demand for this family of tasks is that path integration needs to be performed, possibly followed by some computations based on that integration. These requirements are often met in natural settings, as is widely believed that animals are able to estimate their location in space st) through path integration based exclusively on local ds(t), motion cues dt and neural circuits in the brain that perform this computation have been identified (specifically the entorhinal cortex (Sorscher et al., 2019)). For our analysis, we will assume that the stimuli the network receives are drawn from a stationary distribution, such that the probability distribution p(s(t)) does not depend on time-for navigation, this amounts to ignoring the effects of initial conditions on an animal's ' state occupancy statistics, and assumes that the animal's 's navigation policy remains constant throughout time. The RNN's dynamics are given by:
r(t + At) = r(t) + Ar(()
Ar(t) = @ ((()),s(t), ds(t) dt ) ) on(t). on(t).
(2)
where Ar(t) is a function that describes the network's update dynamics as a function of the stimulus, <(.) is sufficiently expressive nonlinearity, nn(t) [N00, At) is Brownian noise, and At is taken to be small as to approximate corresponding continuous-tin time dynamics. We work with a discrete-time approximation here for the sake of simplicity, and also to illustrate how the equations are implemented in practice during simulations. Suppose that the network's output is given by o Dr((), where D is an N. x N, matrix that maps neural activity outputs, and N, is the number of neurons in the RNN.
We formalize our loss function for each time point as follows:
((t) F., (((()) Dr(1))l>
so that as the loss is minimized over timesteps, the system is optimized to match its target at every timestep while compensating for its own intrinsic noise. Our analysis proceeds as follows. First, will derive an upper bound for this loss that partitions the optimal update Ar into two terms:
2

---

Published as a conference paper at ICLR 2024
b
1e-5
position Recoding error
c
on
(()))s(((())
0.6
explained variance
active
(s(()
p(o)
test loss
4
0.4
0.8
0.7
3
p(o)
2
0.2
0.5
quiescent
(tt)
*MM
0.0
0.4
V2on
1000
2000
1 4
910
batches
# of PCs
g
PC-2
1.0-
-055-
-1
0
-0.0
start
end
x
PC-1
distance from center in y)-space
PC-1
PC-2
PC-2
0
-1
-1
-1
PC-1
PC-1
0.00 0.15 0.30 0.45 0.60 0.75 density
0.90 1.05
1.20
0.0 0.2
0.4
0.6
0.8 density
1.0
1.2
1.4
1.6
Reactivation in a spatial position estimation task. a) Schematic of task. b) Test metrics as function of training batches for the place cell tuning mean squared error loss used during training (gray) and for the position decoding spatial distance eтror (black). e) Explained variance of the number of principal components (PCs) in population activity space during task engagement (blue) and during quiescent noise-driven activity (red). d) Sample decoded outputs during active behavior. Circles indicate the initial location, triangles indicate the final location. Same as (d), but for decoded outputs during quiescence. f) Neural activity projected onto the first two PCs during the active phase. Color intensity measures the decoded output's distance from the center in space. g) Neural activity during the quiescent phase projected onto the same active PC axes as in (f). h) Two-dimensional kernel density estimate (KDE) plot measuring the probability of occupaccy over 200 decoded output trajectories during active behavior. i-k) Same as (h), but for decoded outputs during quiescence (i), neural activity projected
Figure 1:
function
the first two PCs during the active phase (i), and during the quiescent phase (k). Error bars (b-c) indicate 1 standard deviation over five networks.
one which requires the RNN estimate the predicted change in the target function, and one which requires the RNN to compensate for the presence of noise. Second, we derive closed-form optimal dynamics for an upper bound this loss, which reveals decomposition of neural dynamics into state estimation and denoising terms. Lastly, show that under certain conditions, these optimal dynamics can produce offline sampling of states visited during training in the absence stimuli but in the presence of noise
2.2 UPPER BOUND OF THE LOSS
To derive upper bound, first assume that oi (F((),s(t), de(() can be decomposed into two
different functions so that Eq. becomes:
Ar(() = (r(t)) Arz ((t)),((). s(t). ds(t)
dt + +o(())
(4)

---

Published as a conference paper at ICLR 2024
where will ultimately show that both functions scale with At. We assuming that these terms have different functional dependencies; however, for notational conciseness, we will subsequently refer to both updates as Ar, (t) and Ary(t). The first, Ar, (t), is function of r(t) only, and will be used to denoise rtt) such that the approximate equality r(t) - Ar (t) + on(() R ((((t) still holds (Ari() (1) cancels out the noise corruption (n)).. where D is the right pseudoinverse of D. This maintains optimality in the presence of noise. The second, Ar2()). is also a function of the input, and will build upon the first update to derive simple state update such that D (r At)) ~ f(s(t+At)) first consider an upper bound on original loss, which will label Cupper- Exploiting the fact that At is infinitesimally small relative to other terms, we will Taylor expand Eq. first order about small timestep increment At:
To construct this two-step solution,
C(t + At) E,, f At) - D (r(()) Ar(
(5)
En [(((t) + df(s(t) As(() - D(r(t) + Ar(t)) ds(()
(6)
df(s(tt) As(() DArz(() + ((s()) D((() + Ari( on()) an(()))n((()
-Bn ds(()
(7)
where As(t)= da(t) At. Next, using the triangle inequality, note that the loss is upper bounded by a new loss L2, given by:
df(s(t)) As(t) DAr + +1)((((t) D(r(t) + (t) + (n ds(t)
S L2 E,y
(8)
df(((t)) As(() DAr220) + E,, (((((()(())) - + Ar) + ds(() =s(((
(o)
(9)
which separates the loss into two independent terms: one which is function of (ry and the signal, while the other is a function of Ar, (t) and the noise. The latter, Ar, (f)-dependent term, allows Ar to correct for noise-driven deviations between (s(()) and Dr(t). Here, we will assume that this optimization has been successful for previous timesteps, such that r(() Dt (s(()). where Dt is the right pseudoinverse of D. By this assumption, we have the following approximation:
df(()) As(t) (t) ds(()
C2
(E,))((s(()) - D(D
f(s(t) D(D) (r) + +on(t))ll
(10)
df(tt) As(() - (t) ds(t)
+ E,, | [(Ar +n)
(11)
Thus, the second term in Ly trains Ar, to greedily cancel out noise in the system on()). To show that this is similar to correcting for deviations between D' D (s(()) and r in neural space (as opposed output space),
the Cauchy- Schwwar inequality develop the following upper bound:
df(s(t)) As(()= - Dtr2(t) + 111,,,, (() + (n(())l
L2 S C3
(12)
ds(())
This allows the system to optimize for denoising without having access to its outputs, allowing for computations that are more localized to the circuit. As our final step, we use Jensen's 's inequality for expectations derive the final form of our loss upper bound:
La S Lupper = ds(s(() As(() D1rr(1) + +IDDII E.I + (n(1)
(13)
ds(t)
= Liiggnal (Ar2) + Lnoise (rr))
(14)
where Lsignal =
d((a(U))
As(t)= DDr2(t)) is dedicated to tracking the state variable s(t). and ds())
Lnoise | 2 F., (r) + (1) is dedicated to denoising the network state. the next section, will describe how this objective function be analytically optimized in terms of Ar (t) and (ry(t) in a way that decomposes the trajectory tracking problem into a combination of state estimation and denoising.
4

---

Published as a conference paper at ICLR 2024
2.3 OPTIMIZING THE UPPER BOUND
Our optimization will be greedy, so that for each loss ((++ we will optimize only Ar((), ignoring dependencies on updates from previous time steps. Lnoise is the only term in our loss that depends on (1) Ignoring proportionality constants and the square root (which do not affect the location of minima), we have the following equivalence:
(15) Essentially, the objective of Ar1(t) (t) is to cancel out the noise on(t) as efficiently as possible, given access to information about r()). This is a standard denoising objective function, where an input signal is corrupted by additive Gaussian noise, with the following well-known solution (Miyasawa, 1961; Raphan & Simoncelli, 2011):
argiminari (t) Lnoss argmin E,llArr(t)) (t) on(())l
(t) = 0² d logp(r(t))2r At, dr(t)
(16)
where p(r) = f p((()) (())) is the probability distribution over noisy network states given imput stimulus s(t), prior the application of state updates (r, and (r2 By assumption. ((r())((()) (DD) fs(()), of At). We note that परं.y (og((r(t) is the same function for all
time points because for the stimulus sets we consider, p(s(t)) does not depend time (it is a stationary distribution); this demonstrates that the optimal greedy denoising update is not time- dependent. These dynamics move the network state towards states with higher probability, and do not require explicit access to noise information (t)).
Next, optimize for Ar Loignat is the only term in Lupper that depends on Arz(t), so minimize:
df(s(t)
Esignal
As(t) DAr
(17)
ds(t)
By inspection, the optimum is given by: Ar²(t) D dynamics, the presence of noise, are given by:
d((a(e) ds(()
As(t).
Thus the full greedily optimal
Ar (t) 0² d logp(r(t)) + D df(t))) ds(t) dr(t)
At + on()).
(18)
ds(t) dt
This heuristic solution provides interpretability to any system attempting to maintain a relationship to stimulus in the presence of noise. First, denoise the system (Arî). Second, use instantancous
changes in the state variable (ds(t) to update state information. In theory, this solution should hold for any trained network capable of arbitrarily precise function approximation (Funahashi & Nakamura, 1993). In the next section, we will demonstrate that this approximately optimal solution will have interesting emergent consequences for neural dynamics in the continuous-ting limit (At - 0).
2.4 EMERGENT OFFLINE REACTIVATION
Having derived our greedily optimal dynamics, we are in a position to ask: what happens in the absence of any input to the system, as would be observed in a quiescent state? We will make two assumptions for our model of the quiescent state: 1) ds(t) 0, so that no time-varying input being provided to the system, and 2) the variance of the noise is increased by factor of two (deviating from this factor is not catastrophic as discussed below). This gives the following quiescent dynamies it:
de
Ai(t) = 0² logg(r(t)) At V2on(t). dr(t)
(19)
Interestingly, this corresponds to Langevin sampling of p(r) (Besag, 1994). Therefore, we can predict an equivalence between the steady-state quiescent sampling distribution p(r) and the active probability distribution over neural states p(r) (so that p(r) p(r), and consequently p(o) = ((o)). There are two key components that made this occur: first, the system needed to be performing near-optimal noisy state estimation; second, the system state needed to be determined purely by integrating changes in sensory variables of interest. The final assumption- that noise is doubled during quiescent states-B is necessary only to produce sampling from the exact same distribution p(r). Different noise variances will result in sampling from similar steady-state distributions with different temperature parameters. When these conditions are present, we can expect to see statistically faithful reactivation phenomena during quiescence in optimally trained networks.
5

---

Spatial Position Estimation. In this task, the network must learn path integrate motion cues in order to estimate an animal's spatial location in 2D environment. We first generate an animal's motion trajectories Stp(t) using the model described in Erdem & Hasselmo (2012). Next, we simulate the activities of nsp place cells for all positions visited. The simulated place cells' receptive Published as a conference paper at ICLR 2024
3 NUMERICAL SIMULATION TASK SETUP
To validate our mathematical results, we consider numerical experiments on two canonical neuro- science tasks, both of which conform to the structure of the general estimation task considered in our mathematical analysis (Fig. la). For each task, we minimize the mean squared error between the network output o and the task-specific target given by f(s(t)) summed across timesteps.
field centers c(t) (where i = ,,...,,, nsp) are randomly and uniformly scattered across the 21) environment, and the activity of each for a position is given by the following Gaussian tuning curve:
(S)(s) = exp ( ||s [ls-c(013 10
(20)
20'S
where osp is the scale. We then train our network to output these simulated place cell activities based on velocity inputs (t)) from the simulated trajectories. To estimate the actual position in the environment from the network's 's outputs, we average the centers associated with the top k most active place cells. Our implementation consistent with prior work (Banino et al., 2018; Sorscher et al., 2019) and all task hyperparameters are listed in Suppl. Table A.1.
Head Direction Estimation. The network's goal in this task is to estimate an animal's bearing (t)(t) in space based on angular velocity cues (t) where s(t) is a 1-dimensional circular variable with domain [-T, T). As in the previous task, we first generate random head rotation trajectories. The initial bearing is sampled from uniform distribution U(-T, x). and random turns are sampled from normal distribution NO,, 11.52)-this is consistent with the trajectories used in the previous task, but we do not simulate any spatial information. We then simulate the activities of NHD head direction cells whose preferred angles 6 (where i I...., NHD) are uniformly spaced between - and æ,, using an implementation similar to the RatInABox package (George et al., 2024). The activity of the 2th cell for bearing s is given by the following von Mises tuning curve:
(H) = exp(o HD cos(s O()))
(21)
2T ( HD)
where HDD is the spread parameter for the von Mises distribution. With these simulated trajectories, we train the network to estimate the simulated head direction cell activities using angular velocity as input. We estimate the actual bearing from the network's 's outputs by taking the circular mean of the top k most active cells' preferred angles. Hyperparameters for this task are listed in Suppl. Table A.2.
Continuous- time RNNs. For our numerical experiments, we use noisy "vanillla" continuous- time RNNs with linear readouts (further details provided in Suppl. Section A.3). The network's activity is transformed by a linear mapping to predicted place cell or head direction cell activities. During the quiescent phase, we simulated network activity in the absence of stimuli, and doubled the noise variance, as prescribed by our mathematical analysis (Section 2.4).
4 NUMERICAL RESULTS
Spatial Position Estimation. We used several different measures to compare the distributions of neural activity in our trained networks during the spatial position estimation task order to validate our mathematical analysis. First, we found that the explained variance curves as function of ordered principal components (PCs) for both the active and quiescent phases were highly overlapping and indicated that the activity manifold in both phases was fow-dimensional (Fig. Ic). It is also clear that decoded output activity during the quiescent phase is smooth, and tiles output space similarly to trajectories sampled during the waking phase (Fig, Id-e). This trend recapitulated by quiescent neural activity projected onto the first two PCs calculated during the active phase (Fig. If-g). To quantify in more detail the similarity in the distributions of activity during the active and quiescent

---

Published as a conference paper at ICLR 2024
quiescent
B
Ra
active
8
2.5
5.0
7.5
10.0
start
end
KL divergence
d
avg. variance
0
00-
0.02
-1
0.01
X
0.00
noisy unbiased
noiseless unblased
noisy biased
noiseless
0.00 0.15 0.30 0.45 0.60 density
0.75
0.90
1.05
1.20
blased
Figure 2: Biased behavioral sampling and distribution comparisons for spatial position estima- tion. a-b) Decoded positions for networks trained under biased behavioral trajectories for the active (a) and quiescent (b) phases. c-d) KDE plots for 200 decoded active (c) and quiescent (d) output trajectories. e) KL divergence (nats) between KDE estimates for active and quiescent phases. = unbiased uniform networks, B biased networks, 4 the uniform distribution, R random networks, and the a superscript denotes noisy networks. Values are averaged over five networks. f) Box and whisker plots the total variance (variance summed over output dimensions) of quiescent trajectories, averaged over 500 trajectories. Each plot (c-f) is for five trained networks.
phases, we computed two-dimensional kernel density estimates (KDEs) on the output trajectories (Fig. Ih-i) and on neural activity projected onto the first two active phase PCs (Fig. 1j-k). As our analysis predicts, we indeed found that the distribution of activity was similar across active and quiescent phases, though notably in the quiescent phase output trajectories and neural activities do not tile space as uniformly as was observed during the active phase.
Our theory additionally predicts that if the distribution of network states during the active phase is biased some way during training, the distribution during the quiescent phase should also be biased accordingly. To test this, modified the behavioral policy of agent during training, introducing a drift term that caused to occupy a ring of spatial locations in the center the field rather than uniformly tiling space (Fig. 2a). We found again close correspondence between active and quiescent decoded output trajectories (Fig. 2b), which was also reflected in the KDEs (Fig. 2c-d). These results hold whether or not we increase the variance of noise during quiescence (Suppl. Fig. C. We further found that these results hold for continuous-timg gated recurrent units (GRUs) (Jordan et al., 2021) (Suppl. Section A.3 and Suppl. Fig. C.2), showing that these reactivation phenomena are not unique to particular network architecture or activation function. In practice, GRUs were more sensitive to increases in quiescent noise, and other architectures would require more hyperparameter tuning.
To compare activity distributions more quantitatively, we estimated the KL divergence of the distribu- tion of active phase output positions to the distribution of quiescent phase decoded output positions using Monte Carlo approximation (Fig. 2e). We compared outputs from both biased and unbiased distributions, and as baselines, we compared to true uniform distribution, as well as decoded output trajectories generated by random networks. By our metric, we found that unbiased quiescent outputs were almost as close to unbiased active outputs as a true uniform distribution. Similarly, biased quies- cent outputs closely resembled biased active outputs, while biased- to to-unbiased, biased-to soneed-to-rrnne random, and unbiased-to-ren to-random comparisons all diverged. These results verify that during quiescence, our trained networks do indeed approximately sample from the waking trajectory distribution.
7

---

Published as a conference paper at ICLR 2024
n
10
2
77
# n
-10
-10
-10
10
-10
0
10
PC-1
bearing
PC-1
Reactivation in head direction estimation task. a-b) Distribution of decoded head direction bearing angles during the active (a) and quiescent (b) phases. c-d) Neural network activity projected onto the first two active phase PCs for active (c) and quiescent (d) phase trajectories. Color bars indicate the decoded output head direction.
Figure 3:
We decided further the necessity of training and generating quiescent network activity in the presence of noise. By the same KL divergence metric, we found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions. This likely due to the fact that even networks trained in the absence noise still learned attractive task manifolds that reflected the agent's trajectory sampling statistics. However, we found that networks without noise in the quiescent state exhibited less variable trajectories, as measured by their steady-state total variance (Fig. 2f). This demonstrates that individual quiescent noiseless trajectories explored a smaller portion of the task manifold than did noisy trajectories (see Suppl. Fig. C.3a-d for comparison of example noisy and noiseless quiescent trajectories). This failure of exploration could not be resolved by adding additional noise to networks during the quiescent phase: we found that without training the presence of noise, quiescent phase activity with an equivalent noise level generated erratic, non-smooth decoded output trajectories (Suppl. Fig. C.3e-f), with much higher average distance between consecutive points in trajectory than quiescent trajectories associated with noisy training (Suppl. Fig. C.4a-b). Thus. noisy training stabilizes noisy quiescent activity, which in turn explores more of the task manifold than noiseless quiescent activity.
Head Direction Estimation. To demonstrate the generality of our results across tasks, we also examined the reactivation phenomenon in the context of head direction estimation. Here, as in the previous task, we found that the distribution of decoded head direction bearings closely corresponded across the active and quiescent phases (Fig. 3a-b). Furthermore, we found that the distributions of neural trajectories, projected onto the first two active phase PCs, closely corresponded across both phases (Fig. 3c-d), showing apparent sampling along ring attractor manifold. To explore whether reactivation dynamics also recapitulate the moment-to-moneet transition structure of waking activity, we biased motion in the head direction system to be counter-clockwise (Suppl. Fig. C.5). We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were reflected during quiescence, but less clearly than during waking. However, reversals in the trajectories still occurred. These results demonstrate that the type of "diffusive" rehearsal dynamics explored by our theory are still able to produce the temporally correlated, sequential reactivation dynamies observed in the head direction system (Peyrache al., 2015).
DISCUSSION
In this study, we have provided mathematical conditions under which reactivation is expected to emerge in task-optimized recurrent neural circuits. Our results come with several key conditions and caveats. Our conditions are as follows: first, the network must implement a noisy, continuous -itme dynamical system; second, the network must be solving state variable estimation task near-optimally, by integrating exclusively change-based inputs ( (aa()) to reconstruct some function of the state variables (f (s(t))) (for a full list of assumptions see Suppl. Section B). Under these conditions, we
demonstrated that greedily optimal solution to the task involves combination of integrating the
8

---

Published as a conference paper at ICLR 2024
state variables and denoising. In absence of inputs (quiescent phase), we assumed that the system would receive no stimuli (dete) 0) so that the system is dominated by its denoising dynamics, and that noise variance would increase slightly (by factor of 2). Under these conditions, we showed that the steady-state probability distribution of network states during quiescence (f(r)) should be equivalent to the distribution of network states during active task performance (p(r)). Thus, these conditions constitute criteria for form of reactivation to emerge in trained neural systems. Note that though we empirically observe reactivation that mimics the moment-to-me mommoent transition structure of waking networks (Suppl. Fig. C.5), our theory does not yet explain how this phenomenon emerges.
di
We have validated our mathematical results empirically in two tasks with neuroscientific relevance. The first, a path integration task, required the network to identify its location in space based on motion cues. This form of path integration has been used to model the entorhinal cortex (Sorscher et al., 2019), a key brain area in which reactivation dynamics have been observed (Gardner et al., 2019; 2022). The second task required the network to estimate a head direction orientation based on angular velocity cues. This function in the mammalian brain has been attributed to the anterodorsal thalamic nucleus (ADn) and post-subiculum (PoS) (Taube et al., 1990: Taube, 1995; 2007), another critical locus for reactivation dynamics (Peyrache et al., 2015). Previous models have been able to reproduce these reactivation dynamics, by embedding a smooth attractor in the network's 's recurrent connectivity along which activity may diffuse during quiescence (Burak & Fiete, 2009; Khona & Fiete, 2022). Similarly, we identified attractors in our trained networks' latent activation space- we found smooth map of space in the spatial navigation task (Fig, 1f-g) and ring attractor in the head direction task (Fig. 3c-d). In our case, these attractors emerged from task training, and consequently did not require hand crafting cenftinn-smmilarr -imilar to previous work positing that predictive learning could give rise to hippocampal representations (Recanatesi et al., 2021). Furthermore, beyond previous studies, were able to show that the statistics of reactivation in our trained networks mimicked the statistics of activity during waking behavior, and that manipulation of waking hehavioral statistics was directly reflected in offline reactivation dynamics (Fig. 2). Thus, our work complements these previous studies by providing a mathematical justification for the emergence of reactivation dynamics in terms of optimal task performance.
Our results suggest that reactivation in the brain could be a natural consequence of learning in the presence of noise, rather than the product of an explicit generative demand (Hinton et al., 1995; Deperrois et al., 2022). Thus, quiescent reactivation in a brain area should not be taken as evidence for only generative modeling: the alternative, as identified by our work, is that reactivation could be an emergent consequence of task optimization (though it could be used for other computations). Our hypothesis and generative modeling hypotheses may be experimentally dissociable: while generative models necessarily recapitulate the moment-to-moment transition statistics of sensory data, our approach only predicts that the stationary distribution will be identical (Section 2.4). This, for instance, opens the possibility for changes in the timescale of reactivation (Nádasdy et al., 1999).
While the experiments explored in this study focus on loffllocalizatin and head direction estimation, there are many more systems in which our results may be applicable. In particular, while the early visual system does not require sensory estimation from exclusively change-based information, denois- ing is a critical aspect of visual computation, having been used for dehlurring, occlusion inpainting, and diffusion-basse image generation (Kadkhodaie & Simoncelli, 2021)-the mathematical princi- ples used for these applications are deeply related to those used to derive our denoising dynamics. As a consequence, it is possible that with further development our results could also be used to explain similar reactivation dynamics observed in the visual cortex (Kenet et al., 2003; Xu al., 2012). Furthermore, the task computations involved in head direction estimation are nearly identical to those used in canonical visual working memory tasks in neuroscience (both develop ring attractor structures) (Renart et al., 2003). In addition, evidence integration in decision making involves similar state-variable integration dynamics as used in spatial navigation, where under many conditions the evidence in favor of two opposing decisions is integrated along a line attractor rather than two-dimensional spatial map (Cain et al., 2013; Mante et al., 2013). Thus our results could potentially be used to model reactivation dynamics observed in areas of the brain dedicated to higher-order cognition and decision making, such as the prefrontal cortex (Peyrache et al., 2009).
In conclusion, our work could function as a justification for variety of reactivation phenomena observed in the brain. It may further provide a mechanism for inducing reactivation in neural circuits in order to support critical maintenance functions, such memory consolidation or learning.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Our code is available on GitHub at httpss//gl thub. hhtp://annnkkkiiss////RRRtattttt All hyperparameter values and other details on our numerical experiments have been provided Suppl. Section A.
ACKNOWLEDGMENTS
NHK acknowledges the support scholarships from UNIQUE (https: //ww: http:/wwww.ique..ee.eeee www.ue.uuu... .nique.quebecc and IVADO (ht ca). DL acknowledges support from the FRQNT Strategic Clusters Program (2020-R34-265502 - Centre UNIQUE Unifying Neuroscience and Artificial Intelligence Québec) and the Richard and Edith Strauss Postdoctoral Fellowship in Medicine. BAR acknowledges support from NSERC (Discovery Grant: RGPIN-2020-05100 Discovery Accelerator Supplement: KGPAS-2020-00033 Arthur B McDonald Fellowship: 566355 20222) and CIFAR (Canada AI Chair; Learning in Machine and Brains Fellowship). GL. acknowledges support from NSERC (Discovery Grant: RGPIN-2018 2018-04821) CIFAR (Canada Chair), and the Canada Research Chair Neural Computations and Interfacing. The authors also acknowledge the support of computational resources provided by Mila (https: //mila..m.aa.///m htt//m..aauuueeemmmmmma.mmeaaaep and NVIDIA that enabled this research.
REFERENCES
Toshitake Asabuki and Tomoki Fukai. Learning rules for cortical-like spontaneous replay of an internal model. bioRxiv, 2023. DOI: 10. 1011100023..2.11..22 1w..0....71...... 17.529958. URL: 17.528958.
biorxiv.
org//contenn/eerr///22//////////
Amir Hossein Azizi, Laurenz Wiskott, and Sen Cheng. A computational model for preplay in the hippocampus. Frontiers in Computational Neuroscience, 7, 2013. ISSN: 1662-5188. DOI: 10, 3389//ncom.2013.0016 URL: https://www. wwwntiersinnnnnooomoe wrtparticlee///ttt. 3389/fncom. 2013.00161.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick. Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector based navigation using grid-like rep- resentations in artificial agents. Nature, 557(7705):422 433, 2018. 10.1038/541886-0188
ISSN: 1476-4687. DOI: URL: https: //doi org/10. 1038/ s41 586 018 118-888 0102-6.
Julian E Besag. Discussion of "Representations of knowledge in complex systems" by Ulf Grenander and Michael Miller. Journal of the Royal Statistics Society B, 56(4):591-59) 1994.
Yoram Burak and Ila R Fiete. Accurate path integration in continuous attractor network models of grid cells. PLOS Computational Biology, 5(2):1-16, 2009. DOI: 10. 10.1371/joummal.pp 100.00009.. 1000291. URL: httpps://dol org/o/..//....... 1371/journal. pcbi. 1000291.
György Buzsáki. Two-stage model of memory trace formation: A role for "noisy" brain states.
Neuroscience, 31)))5511-570, 1989. ISSN: 0306-4522. DOI: 10110160366-5522289994455
URL: htt.:tttps///oooorrttee org/ii..rr/e/.. 101 6/0306 (502
Nicholas Cain, Andrea K Barreiro, Michael Shadlen, and Eric Shea-Brown. Neural integra- tors for decision making: A favorable tradeoff between robustness and sensitivity. Journal of Neurophysiology, 109(10):2542-25 25599, 2013. DOI: 10.1152/j..0097.2011 URL: https:
aeddii.org/10 1152/ 00976. 20976.2012.
Tianhao Chu, Zilong Ji. Junfeng Zuo, Yuanyuan Mi, Wen-hao Zhang, Tiejun Huang, Daniel Bush, Neil Burgess, and Si Wu. Firing rate adaptation affords place cell theta sweeps, phase precession and procession. bioRxiv, 2024. DOL: 10. 10.1101/2022.111 1..1...2..1.14..1600600 14.50 URL: https: www.bbirxxvv. 11. 11.45554444111
wtgp/comtent//ealy////////////
10

---

Published as a conference paper at ICLR 2024
Christopher Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recur- rent neural networks to perform spatial localization. International Conference on Learning Representations, 2018. URL: http:///pppeeeeeeeee.eeeerrr/rad
Christopher J Cueva, Peter Y Wang. Matthew Chin, and Xue-Xin Wei. Emergence of functional and structural properties the head direction system by optimization of recurrent neural networks, In International Conference Learning Representations, 2020. URL: Forum?id=HKISeRE
net/
Nicolas Deperrois, Mihai A Petrovici, Walter Senn, and Jakob Jordan. Learning cortical represen- tations through perturbed and adversarial dreaming. eLife, 1::76384, 2022. ISSN: 2050-084X. 7554/eLife. 76384.
DOI: 10. 10.7554/eL.ffe.766 76384. URL: https:://bii.oorgggeee
Uğur M Erdem and Michael Hasselmo. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. European Journal of Neuroscience, 35(6):916-931, 2012. DOI: 10.1111/j.1400-111 1460-9568. 000.1191....220066600 2012.08011...... URL: https://onliieelbbaayy wwwphbrrrrrwwwlly............eee com/ doi/abs/10. /10. 10..111111 1111/j.1j.... 1000-14685590 2012. 2212.088 08015.x. x.
David R Euston, Masami Tatsuno, and Bruce L McNaughton. Fortarorward playback re- cent memory sequences in prefrontal cortex during sleep. Science, 318(5853):114-111 2007. DOI: 10.1 10.111 science. 2122//iieecee11499990 URL: http:://www.scienc..oom http:/sceence..rg/dd//bbbb////oe 1126/
David J Foster. Replay comes of age. Annual Review of Neuroscience, 40:581-602, 2017. ISSN:
1545-4126. DOI: 10. 11t.:///aareeeeeee---01111111010 URL: https: /ttp::://wwww./ http://wwwwnnaaaeeeeeeeeee
wrg/connent/journalss/
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time
recurrent neural networks. Neural Networks, 666))801-806, 1993. ISSN: 0893-6080. DOI: 10.
1016/500993-6080005)881333 pi1/S0893308000888122
URL: http:://wwww.scceeddrrree..... cwm/ http://tt.ccmmmcccce//rrrtioii/t
Richard J Gardner, Li Lu, Tanja Wernle, May-Britt Moser, and Edvard I Moser. Correlation
structure of grid cells is preserved during sleep. Nature Neuroscience, 22(():598-600 2019.
ISSN: 1546-1728 DOI: 10.1038//415933019943660000 URL: https://doi.org//1 541593-019 00100360-000---111
Richard Gardner, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A Baas, Benjamin A Dunn, May-Britt Moser, and Edvard I Moser. Toroidal topology of population activity in grid cells. DOI: 10. 0011008744118---1144466--- URL: https://doi, 00. 1038/s4 541566 021 021-04268-7
Nature, 602(7895); 2022. ISSN: 1476-4687.
Tom M George, Mehul Rastogi, William de Cothi, Claudia Clopath, Kimberly Stachenfeld, and Caswell Barry. RatInABox, a toolkit for modelling locomotion and neuronal activity continuous environments. eLife, 13:e85274, 2024. ISSN: 2050-084X. DOI: 10.7554/eLiffe.52744 URL:
https://doi.org: 7554/eLi5 wiw.....feeeeeeeeeeeee 85274.
Tatsuya Haga and Tomoki Fukai. Recurrent network model for learning goal goal-directed sequences through reverse replay. eLife, 7:e34171, 2018. ISSN: 2050-084X. DOI: 10.7554eeLife.34171 URL: httt::::/doo.rrgg/ orgo 000.///1.1100 7ww4/55555e 34171.
Tyler L Hayes and Christopher Kanan. Online continual learning for embedded devices. In Sarath Chandar, Razvan Pascanu, and Doina Precup (eds.), Proceedings of The 1st Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pp. 744 766. PMLR, mlr wwwppp/esssvveee http///vv99haaaee2aa.a html.
2022. URL: https://proceete wtt.aedii........./.eet....m
Tyler I. Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9769-9776. IEEE Press, 2019. DOI: 10.1 00.1000ICAA22019887993833 URL: https://doi.org/// 1189/ICRA. 2019. 8793982.
Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 158 1995. DOI: 10. 126//ciccee 7761831. URL: https: htws:///ww.c. http:/cieeccc..g/////ooaooeeee 1et.//11/isisse sciencc..ceee..e.eeeeeeeeee 7761831.

---

Published as a conference paper at ICLR 2024
Kari L Hoffman and Bruce L McNaughton, Coordinated reactivation of distributed memory traces in primate neocortex. Science, 297(5589) 2002. DOI: 212//cciecc..107335555. URL:
https://www. science /ttg//o///s/////// /10/ 1126/ 1873538.
John J Hopfield. Neurodynamiies of mental exploration. Proceedings of the National Academy of Sciences, 107(4)) 147(4))1 2010. DOI: 10. 1991077/paas.01199991 1007 URL: https://wwwt pna...
org/doi/abs/10 1073/pnas. 091399110)
Hideyoshi Igata, Yuji Ikegaya, and Takuya Sasaki. Prioritized experience replays on hippocam- pal predictive map for learning. Proceedings of the National Academy of Sciences, 118(1): e201 02011266118, 18, 2021. DOI: 10.10733pnas.200 1266118 18. URL: https: /wwp: www...ppaa..oom org/doi/abs/ 10. 10.1007/pnas. 2011 2011266118.
lan D Jordan, Piotr Aleksander Sokół, and Il Memming Park. Gated recurrent units viewed through the lens of continuous time dynamical systems. Frontiers in Computational Neuroscience, 15, DOI: 10. 0tt99//fcoo.2221..77777 org/articles/1e. 3389/ fncom. 2021. 678158.
2021. ISSN: 1662-5188.
URL: https://wwwtt frontiersin.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse prob- lems using the prior implicit in denoiser. Mare' Aurelio Ranzato, Alina Beygelz- imer, Yann N Dauphin. Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13242-13254, Curran Asso- ciates, Inc., 2021. URL: https: https: 14le//6224999/daaeefffff66662222
Louis Kang and Michael DeWeese. Replay as wavefronts and theta sequences as bump oscillations in grid cell attractor network. eLife, 8:e46351, 2019, ISSN: 2050-084X. DOI: 10.7554/eLife. 46351. URL: https:://ool org/ /ttgg///.1.0 /555/555/fff 46351.
Tal Kenet, Dmitri Bibitchkov, Misha Tsodyks, Amiram Grinvald, and Amos Arieli. Spontaneously
emerging cortical representations of visual attributes. Nature, 425(6961)::54 956, 2003. ISSN: 1476-4687. DOI: 10. 1038/mature02078. URL: https: http:://ddoiooooo orw.:io..rg/1//0 10t.r///....11//111 033/nature02078.
Mikail Khona and lla R Fiete. Attractor and integrator networks the brain. Nature Reviews
Neuroscience, 23(12):444-766, 2022. ISSN: 1471-0048. DOI: 10. 10.10388441133-022-006642
URL: //tttp::/doi.orrr www.oi...rg/1...
Albert K Lee and Matthew A Wilson. Memory of sequential experience in the hippocampus
during slow wave sleep. Neuron, 3666): 2002. ISSN: 0896-6273. DOI: 10.1016/
https:/t/////o..o http::///...ooorroo..ooo//////20 0966-6.
Penelope A Lewis and Simon J Durrant. Overlapping memory replay during sleep builds cognitive schemata. Trends in Cognitive Sciences, 15(8):333-351, 2011. ISSN: 1364-6613. DOI: 10.10
1.s.20011.06.044 URL: httpss://oii. org/10. 1016/j. ticc.ii.... 2011 .0..00000000 004.
Ashok Litwin-Kumar and Brent Doiron. Formation and maintenance of neuronal assemblies through synaptic plasticity. Nature Communicatioons, 5(1):5319, 2014. ISSN: 2041-1723, DOI: 10.1038/
ncomms6319. URL:: https:://oi...oo org/10 0038/ncomms619.
Valerio Mante, David Sussillo, Krishna Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78-84 2013. ISSN: 1476-4687. DOI: 10. 1038/maatrr URL: https:://oi.onnn org/10. n3t/n
James L McClelland, Bruce L McNaughton, and Randall C O'Reelly, Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychologicul Review, 102(3):419-457, 1995. ISSN: 0033-295X. DOI: 10. 013770033-225Xx10223 0033-295X. 3.2.3.419.
URL: httpp::/doo..org/11 1037/
Aaron D Milstein, Sarah Tran, Grace Ng, and Ivan Soltesz. Offline memory replay in recurrent neuronal networks emerges from constraints online dynamics. The Journal of Physiology, 601(15):3241-3264 2023. DOI: 10. 10.1113//P282.6. URL: https: //hp:///h/ccccc//cc onlinelibrary.. wiley. cttoo///o///b//////// wwtp.///ddd/bbbb////////////211 1113////2.8...1.
12

---

Published as a conference paper at ICLR 2024
Koichi Miyasawa. An empirical Bayes estimator of the mean of normal population. Bulletin of the International Statistical Institute, 38(4): 1961.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53. 2015. ISSN: 1476-4687. DOI: 10.1038/nature 4236. URL: https://doi. org/10. 1038/ 14236.
Zoltán Nádasdy, Hajime Hirase, András Czurkó, Jozsef Csicsvari, and György Buzsáki. Replay and time compression of recurring spike sequences in the hippocampus. Journal of Neuroscience, 19 (21):9497-9507, 1999. ISSN: 0270-6474, DOI: L552/INEUROSCII.-222000000.
URL:
https://ww. http::///ww..jeuosssccc jneurosci httpg//coonettt1///// /999///////////977 /9497.
Adrien Peyrache, Mehdi Khamassi, Karim Benchenane, Sidney I Wiener, and Francesco P Battaglia. Replay of rule-learnigg related neural patterns the prefrontal cortex during sleep. Nature Neuroscience, 12(7):919 (77)):99-9999 926, 2009. ISSN: 1546-177 1726. DOI: 10. 1038/mn.2337. URL: https:
//doi.org/1e 1038/nn. 2337.
Adrien Peyrache, Marie Lacroix, Peter Petersen, and György Buzsáki. Internally organized mechanisms of the head direction sense. Nature Neuroscience, 18(4):569- 575, 2015. ISSN: 1546-1726. DOI: 10. 1038/mn.3968.
URL: https://doi. hrtp////.or/////// 1038/nn.396.
Martin Raphan and Eero P Simoncelli. Least squares estimation without priors or supervision. Neural Computation, 23(2):374-420, 2011. ISSN: 0899-7667. DOI: 10. 10.1162/NECO_L a__06_66___0_000000 URL:
https://doi. wrg/oogg///// /00.g//..110 10. 1162/NECO_a_@000
Stefano Recanatesi, Matthew Farrell, Guillaume Lajoie, Sophie Deneve, Mattia Rigotti, and Eric Shea-Brown. Predictive learning network mechanism for extracting cow-dimensional latent space representations. Nature Communications, 12(1):1417, 2021. ISSN: 2041-1723. DOI:
10.10388441447-021-216666
URL: https://doi http:///oooor////.../// 1038/s4 067-021- -2 -21696-1.
Alfonso Renart, Pengcheng Song, and Xiao-Jing Wang. Robust spatial working memory through homeostatic synaptic scaling in heterogeneous cortical networks. Neuron, 38(3):473-488, 2003. ISSN: 0896-6273. DOI: 10. 1101000000896-667733000000203300 S0896- (8886-6273(83)80255--5
URL:: http::ttt//:/oo.ooo http:////ooooggg....// 1016/
Bin Shen and Bruce I. McNaughton. Modeling the spontaneous reactivation of experience-specific
hippocampal cell assembles during sleep. Hippocampus, 6(6):685-692) 1996. DOI: 10.1002/(SICI)
1098- (UU0://0SS//.000000.....DDDDDDD.
URL: https::/
wiley.
(1996) 6x33685 :AD-:668::::DD-HHPPAAA -AD-HHPP1133333.... 200
Ben Sorscher, Gabriel Mel, Surya Ganguli, and Samuel Ocko. A unified theory for the origin of grid cells through the lens of pattern formation. In Hanna M Wallach. Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- neurips neurips.cc/paper wttp://peeelale//p/eeeeeeeeeeeee Paper444PPPPPpppp pdff.
ciates, Inc., 2019. URL: https::/ Flle//e/ddd555bee7ffffff66622622
Jeffrey S Tauube. Head direction cells recorded in the anterior thalamic nuclei of freely moving rats.
Journal of Neuroscience, 15(1):70-86, 1995. ISSN: 0270-6474. DOI: 1./5///EEUOOS..
15-01-00070.1995 URL:
Jeffrey S Taube. The head direction signal: Origins and sensory-motor integration. Annual Re-
view of Neuroscience, 30: 18--07, 2007. ISSN: 1545-4126. DOI: 10.1 1.l.annnnnuu...eeurr.
29.051605.1 11005.112880 112854. URL: https: /ttp:://www... mmnurev.neurgn 29. 051605. 112854.
Jeffrey Taube, Robert Muller, and James B Ranck, Jr. Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations. Journal of Neuroscience, (0(2):4364447 1990. ISSN: 0270-6474. DOL: 101122//EEROOSCI.000000000
1990. URL: http://pp:/www..ierrrrrror/////o
13

---

Published as a conference paper at ICLR 2024
Panagiota Theodoni, Bernat Rovira, Yingxue Wang, and Alex Roxin. Theta-modulation drives the emergence of connectivity patterns underlying replay in a network model of place cells. eLife, 7:c37388, 2018. ISSN: 2050-084X. DOI: 10.7554/cLifc..73388. URL: https://doi.org// 7554/eLife. 37388.
David Tingley and Adrien Peyrache. On the methods for reactivation and replay analysis. Philosoph- ical Transactions of the Royal Society B: Biological Sciences, 375((799):20190231,,0 2020. DOI: URL:
10. wtt1099//tt..19990 rstb. 2019. 0231.
http://www..ccttoooooooo/si///// /098
Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zambaldi, Dharshan Kumaran, Demis Hassabis, Caswell Barry, and Charles Blundell. A model of egocentric to allocentric understanding in mammalian brains. bioRxiv, 2022. DOL: 10. 11101/220.11.11..3988........... biorxiv. org connteen/errll//222///////////// 11.11.11111111111111111111111111
URL: nttps:///ww.
11. 17.3781
Pauli Virtanen, Raif Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J van der Walt, Matthew Brett, Joshua Wilson, K Jarrod Millman, Nikolay Mayorov, Andrew RJ Nelson, Eric Jones, Robert Ker,, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, EA Quintero, Charles R Harris, Anne M Archibald, Antônio H Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3):261-272, 2020. ISSN: 1548-7105. DOI: 10. 10.10088/44592-019--000 nttps://dol. orgoorg////// 10. 100000009011122119---00
URL:
Shengjin Xu, Wanchen Jiang, Mu-ming Poo, and Yang Dan. Activity recall in a visual cortical ensemble. Nature Neuroscience, 15(3):449-455, 2012. ISSN: 1546-1726. DOI: 10.1038/mn.3036
URL: htw.httts:///i.rrrg 10380/.000001111..mmmmmmm 3036.
H Freyja Ölafsdóttir, Caswell Barry, Aman B Saleem, Demis Hassabis, and Hugo J Spiers. Hippocam- pal place cells construct reward related sequences through unexplored space, eLife, 4:e06063, 2015. ISSN: 2050-084X. DOI: 10.7554/eLife.06000 06063. URL: https: http::///ooo.o//////// www...ggg///// 7554//5L5feee 06063.
14

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A NUMERICAL SIMULATION DETAILS
A.1 TASK HYPERPARAMETERS
Table A.I: Hyperparameters for the spatial position estimation task.
Hyperparameter
Value
TASK
Environment size
2.2 m 2.2 m
Border region
0.03 m
Border slowdown factor
0.25
Position initialization
u(-2.2,2.2)
Rotation velocity bias
Orad/s
Rotation velocity std. dev.
11.52rad/s
Rayleigh forward velocity
0.2
Biasing anchor point
(0,0)
Biasing drift constant
0.05
# place cells
512
OSP Sequence length
0.2
active 100, quiescent 200
0.01 = 0.0707
v0.02
NETWORK
recurrent units
512
0.1
TRAINING
Batch size
200
# batches
2500
Optimizer
Adam
Learning rate
0.001
Table A.2: Hyperparameters for the head direction estimation task.
Hyperparameter
Value
TASK
Bearing initialization
U(-m, n)
Rotation velocity bias
0 Drad/s
Rotation velocity std. dev.
11.52 rad/s
head direction cells
512
THD
Sequence length
active
100, quiescent 200
0.1 R 0.7071
v0.02
NETWORK
recurrent units
128 0.04
TRAINING
Batch size
200
batches
20000
Optimizer
Adam
Learning rate
0.001
15

---

Published as a conference paper at ICLR 2024
A.2 ADDITIONAL DETAILS
Kernel Density Estimation (KDE). To compute KDEs, we used the stats. gaussian_kdes method from scipy (Virtanen et al., 2020), with all hyperparameters set their default values. For all KDE plots, we computed the density estimates using 200 trajectories of 100 timesteps each for the active phase and 200 timesteps each for the quiescent phase. For Fig. 2e, we estimated densities using 200 trajectories of 100 timesteps each for the active phase and 1000 timesteps each for the quiescent phase (to get a better estimate of the steady-state distribution). We used 2500 samples from each distribution to compute the KI. divergence by Monte Carlo approximation.
Quantifying exploration during quiescence. To analyze the importance noise for exploration during quiescence, we computed the steady-state total variance (variance summed over output dimensions) for 200 quiescent trajectories from each trained network. For Fig. 2f, we generated trajectories of 2000 timesteps each and truncated the first 1000 timesteps before computing the variance, to demonstrate that noise facilitates greater and continued exploration over long timescales.
A.3 NOISY CONTINUOUS-TIME RNNS
"Vanilla" RNNs. For most of our numerical experiments, we use noisy "vanilla" continuous- RNNs with linear readouts. The equations for network updates and output estimates are follows:
1 [ + ReL.U(""" + wi As()))][+++
Ar(1) =
(A.1)
o= Dr((,,
(A.2)
where r(() represents the network activity time I, As(() is change-based input the network, Wrec and W are the recurrent and input weight matrices respectively, 7 is the RNN time constant, and (t) NO0 At) is Brownian noise. The continuous-tima dynamics are approximated using the Marer-Maruyama method with integration timestep At 0.02s. The network's 's activity is transformed by linear mapping D to predicted place cell or head direction cell activities o. During the quiescent phase, we simulated network activity in the absence of stimuli (As(t) 0), and doubled the noise variance, as prescribed by our mathematical analysis (Section 2.4).
We set the value of for each task to ensure that the RNN is able to respond quick enough to inputs and ensure optimal performance. Further, for each task we choose different training values for o to scale the Brownian noise to establish effective signal-to-nowl -o-ise ratio that is high enough to accurately solve the task. From this baseline noise level, quiescent trajectories were calculated with doubled variance.
Gated recurrent units (GRUs). We also used continuous time GRU formulation similar to the one described by Jordan al. (2021). The equations for network updates and output estimates are:
z (t) sigmoid (Wrecr(t) + W As()))
(A.3)
g(t) = sigmoid (Wy + W As()))
(A.4)
Ar(t) 1 [(1 z(t) @ (tanh Wrec (g(() ② r(t)) + W As()) -r(t))] At +o((((t) 7
on(t) (A.5)
o= Dr(tt,
(A.6)
where r(t) represents the network activity at time t, z(t) and g(t) are gates, As(t) is change-based input to the network, Wree, Win Wree Win constant, and (t) ~ (00 At) is Brownian noise. The continuous-time dynamics are approximated using the Muler-Maruyama method with integration timestep At = 0.02 Just as with the vanilla RNN, the network's s activity is transformed by a linear mapping to predicted place cell or head direction cell activities During the quiescent phase, we simulated network activity in the absence of stimuli (ts(t) = 0), but we do not double the noise variance due to noise- sensitivity. We set appropriate values for and a, just as with the vanilla RNNs.
g Wree and Win are weight matrices, T is the time
16

---

Published as a conference paper at ICLR 2024
B KEY ASSUMPTIONS FOR MATHEMATICAL RESULTS
1. We consider discrete-time approximations of noisy continuous-time RNNs. Using continuous-time RNNs is common practice in the literature, and we their discrete-time approximations to be in line with our numerical simulations.
2. The network must be performing some variant of path integration, by integrating change- based information about environmental state variables to some function of these variables. This condition often met in natural settings, such as spatial navigation.
3. We assume that the inputs to the network are drawn from a stationary distribution, which amounts to ignoring the effects of initial conditions state occupancy statistics, and also assuming that the behavioral policy remains constant throughout time.
4. We consider greedy optimization of the loss at every timestep. Greedy optimization is sensible way of partitioning effort across time in this task: the network does the best that it can each timestep, assuming that at each previous timestep the best possible job has been done. In the absence of noise, the greedily optimal solution is equivalent to path integration, which is also a globally optimal solution.
5. We assume that the network performing optimally the presence of noise. practice, we train our networks until their loss reaches very low, near-zero values.
6. We assume that our network dynamics can be decomposed into two terms with different functional dependencies. The first term depends only on the activity, while the second term depends on all original dependencies the activity, the state, and the change-based inputs.
7. For the quiescent phase, we assume that the change-based sensory inputs are zero. This is reasonable because for tasks like those we have considered, which involve integrating self-motion cues, de must be zero during periods of quiescence like sleep, where the animal is not moving and hence does not receive sensory inputs associated with self-motion
da(t)
8. While we assume that the noise variance is doubled during quiescence to show exact equivalence with Langevin sampling, this is by no means necessary to witness reactivation (Suppl. Fig. C.2). This noise variance is equivalent to a temperature parameter for the sampling.
17

---

Published as a conference paper at ICLR 2024
C
ADDITIONAL NUMERICAL SIMULATIONS
a
b
d
0
start
end
g
0
-1
-1
0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 1.20
density
Figure C.I: 1: Spatial position estimation results without increased noise variance during quies- cence. a-h) Decoded output trajectories during active (a) and quiescent (b) phases for network trained on the unbiased task. c-d) Same as (a-b) but for a network trained on the biased task. e-f) KDE plots for 200 decoded active (e) and quiescent (f) output trajectories for the unbiased task. g-h) Same as (e-f) but for the biased task.
b
d
X
start
end
X
00-
>0-
0
-1
-1
X
X
0.00 0.15 0.30 0.45 0.60 0.75 0.90 density
1.05 1.20
Figure C.2: Spatial position estimation results for GRU networks. a-b) Decoded output trajectories during active (a) and quiescent (b) phases for network trained on the unbiased task. c-d) Same as (a-b) but for network trained on the biased task. e-f) KDE plots for 200 decoded active (c) and quiescent (f) output trajectories for the unbiased task. g-h) Same as (e-f) but for the biased task.
18

---

Published as a conference paper at ICLR 2024
C
0
-1
0
1
start
end
b
d
0
-1
X
X
Figure C.3: Example decoded quiescent trajectories under different noise conditions for spatial position estimation. a-b) Example noisy quiescent trajectories for a network trained the presence of noise, for the unbiased (a) and biased (b) tasks. c-d) Same (a-b), but for noiseless quiescent trajectories for a network trained without noise. e-f) Same as (a-b), but for noisy quiescent trajectories for network trained without noise.
a
0.06
avg. distance
avg. distance
0.4
0.04
0.2
0.02
0.00
DT active quiescent active quiescent noisy quiescent
NT
NT
DT
DT
DT active quiescent active quiescent noisy quiescent
NT
NT
DT
DT
Figure C.4: Distributions of average distances between consecutive points in output trajectories for unbiased and biased spatial position estimation. a) Distributions for the unbiased task. NT denotes noisy training while DT denotes deterministic (noiseless) training. b) Same as (a) but for the biased task. In each case, the box and whisker plots show the distribution of within-trajectory average point-to-poin distances, computed for 200 output trajectories and random seeds.
19

---

Published as a conference paper at ICLR 2024
a
b
t nt
t π
t π
3m
33
d
0.0151
f
0.0005
0.2
0.3
Proportion
Proportion
0.15
Proportion
0.2
0.10
0.1
0.05
0.1
0.0
0.00
0.0
-0.5
0.0
0.5
-0.2
0.0
0.2
-0.2
0.0
0.2
Angular velocity
Angular velocity
Angular velocity
Figure C.5: Biased noment-to-momenn transition structure for the head direction estimation task. a-b) Decoded trajectories from the active (a) and quiescent (b) phases, for a biased network trained on counter-clockwise trajectories. c) Decoded trajectory from the quiescent phase for an unbiased network. Sequence length is 50 timesteps for active phase trajectories (a) and 500 timesteps for quiescent trajectories (b-c). d-f) Distributions of angular velocities during the active (d) and quiescent (e) phases for a biased network, and the quiescent phase for an unbiased network (f). Dashed line denotes the mean.
20


Reviews:
[{"review_id": "xLb3ISgGKn", "reviewer": "Reviewer_xvHM", "summary": "This submission seeks to model the reactivation of brain activity during periods of quiescence using task-optimized network. The paper develops some mathematical formalisms to show that under certain assumptions, recurrent neural networks trained to perform a task develop denoising dynamics. \n\nWhile there are some interesting ideas presented in this submission, overall it feels that the results presented in the current version are preliminary and not surprising.", "questions": "Would it be possible to discuss more explicitly how their results are connected or supported by the empirical data? In particular, how is the “diffusive reactivation” related to neural data?\n\n The mathematical analysis relies on a number of assumptions, to the extent that it is difficult to judge whether the conclusion would actually be applicable to the numerical experiments. Can these assumptions be justified or better motivated (beyond the sake of mathematical convenience)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "— Understanding the neural mechanisms underlying reactions/replays is an important question.\n\n-- The paper seeks to analyze RNNs trained to perform a class of tasks theoretically, which is a somewhat rare excise in this literature. \n\n— The mathematical analysis of the loss function used in the training and the connection to the Langevin dynamics, while under somewhat strong assumptions, remain interesting.", "weaknesses": "1. A number of relevant studied were not cited and discussed. This include work using RNNs and attractor dynamics to model the replay and theta sequences in the hippocampus, e.g.,\nHopfield, John J. \"Neurodynamics of mental exploration.\" Proceedings of the National Academy of Sciences 107.4 (2010): 1648-1653.\nKang, Louis, and Michael R. DeWeese. \"Replay as wavefronts and theta sequences as bump oscillations in a grid cell attractor network.\" Elife 8 (2019): e46351.\nChu, Tianhao, et al. \"Firing rate adaptation affords place cell theta sweeps, phase precession and procession.\" bioRxiv (2022): 2022-11.\n\nThe paper also misses several pieces work in training RNNs to study the grid cells systems and HD systems.\nCueva, C. J., & Wei, X. X. (2018). Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. ICLR.\nUria, B., Ibarz, B., Banino, A., Zambaldi, V., Kumaran, D., Hassabis, D., Barry, C. and Blundell, C., (2020). The spatial memory pipeline: a model of egocentric to allocentric understanding in mammalian brains. BioRxiv, pp.2020-11.\nCueva, C.J., et al (2020). Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks.ICLR.\n\n2. The main results are not surprising after considering what we know so far on this topic. Prior work has shown that training the RNNs to perform path integration or angular integration task leads to networks that exhibit attractors dynamics that is similar to continuous attractor models. Furthermore, it is well known that attractor dynamics can perform denoising. Thus it is not surprising that RNNs trained on these previous studied tasks can perform denoising. \n\nIt should also be noted that prior work has further characterize the rate of diffusion along the low-d manifold, e.g., see Fig 6 of the following paper: \nBurak, Y. and Fiete, I.R., 2009. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 5(2), p.e1000291.\n\n\n3. As described in the paper, the trained network follows diffusion dynamics when the stimulus turned off. This is naturally expected because the diffusion of network state on a low-d manifold. This would result in brownian-motion like trajectory. Furthemore, the replay trajectories observed in the hippocampus typically described by a systematic drift towards one direction (e.g., in linear or circular tracks), not diffusive dynamics. Can the authors be more explicit about the experimental data they were modeling?\n\n4. One more point that confuses me—the authors stated “we found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions. ” This seems to argue against the denoising dynamics, and make the usefulness of the mathematical analysis questionable.\n\n5. The mathematical analysis relied on a set of approximations and assumptions, which were not well justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTZ2umIavx", "reviewer": "Reviewer_pnLC", "summary": "The authors develop a theoretical framework for understand why offline reactivation occurs in recurrent neural networks. In particular, the authors argue that neural noise during awake states is essential for the emergence of faithful and varied reactivation of neural trajectories during quiscience (i.e. in absence of external stimuli).", "questions": "- In section 2.1 it may be relatively obvious that p denotes probability, but I would still clarify this\n- In equation 15 it's a bit odd to provide a new definition for L_{noise} having just defined it previousply with the D term involved\n- the delta t is given as 0.02. What's the unit, seconds?\n- I would recommend a limitations section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- the paper is generally well written\n- the mathematical theory is well presented and interesting (as far as I'm aware it is novel, but I am not certain of this)\n- offline reactivation should be of interest both to machine learning and neuroscience researchers; the paper does make some theoretical and experimental headway into why/how it occurs", "weaknesses": "- perhaps this is not in the scope of the paper, but the authors do not provide any theoretical/numerical support for the functional benefit of offline reactivation. The authors mention other works which demonstrate that it may aid the formation of long-term memories, schema, planning. Do the theoretical/numerical results in this paper support these possibilities, or any one in particular?\n- Relatedly, a key feature of this study is that the RNNs considered are noisy during awake states. It's unclear to me exactly why this is necessary. The authors \"found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions\"; is the point that there's more space exploration with noise, giving rise to a functional benefit? Or that somehow the introduction of noise better captures observed neural data?\n- I found the link between the theoretically optimal solution for update dynamics and the actual application of RNNs somewhat unclear. Are we to take that (presumably backprop-trained) RNNs employ update dynamics similarly to the theoretical solution; is it biologically reasonable for e.g. equation 18 to be implement by an RNN? If so, is it possible to show that they do; e.g. by comparing these recurrent inputs to the optimal solution? If not, are the experimental results relevant to the theory? \n- Relatedly, perhaps this is overly harsh, but I also do not find the result that RNNs without stimuli visit similar states as with stimuli very surprising.\n- The authors suggest an interesting discrepancy in experimental prediction for their noise-based theory of offline reactivation and generative modeling; specifically, that \"while generative models necessarily recapitulate the moment-to-moment transition statistics of sensory data, our approach only predicts that the stationary distribution will be identical\". I am confused by this statement and not sure if it is true. Is that to say that the proposed model would not encode sensory transitions but would simply replicate the overall probability distribution of sensory states? I would find this surprising for an RNN whose recurrent weights are trained on sensory data; that is, I would suppose that the RNN weights would themselves somehow capture transition statistics of the task variables (e.g. maybe this is valuable as a kind of denoising effect on noisy observations). Perhaps I am wrong though", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1V4ocSchE", "reviewer": "Reviewer_aZpo", "summary": "The authors demonstrate that training network dynamics on an upper bound of the cost function for learning a function of an input stimulus in the presence of network noise results in network dynamics that separate into a denoising component and a signal integration component.  Furthermore, they provide conditions such that, in the absence\tof input, the network will reproduce typical states of the (learned function of the) input stimulus.  Notably, this strictly requires the input noise to be different in the quiescent vs input-driven cases.  The authors demonstrate numerical simulations in which networks reflect these results.", "questions": "Last sentence of the introduction:  I think \"ecologically\" should be \"ethnologically\"\n\nQuestions:\n\n- Does this work for more general RNNs (more general nonlinearities, interactions between recurrence and signal integration, etc.)?\n\n- Is there reason to believe the noise distribution changes in the quiescent state?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper does a nice job of outlining how the cost function can be deconstructed into two components, the denoising component and the signal integration component.  The idea of replay being a result of signal integration in the presence of noise is an interesting one, and the authors present a simple mechanism whereby this can arise. This is a contribution to an important question in neuroscience.  The theoretical approach, results, and the numerical experiments are well described and supported.", "weaknesses": "The paper is potentially confusing to some readers in that the presentation (and the numerical experiments if I have followed) regard training a network with a fixed nonlinearity and trainable weight parameters (i.e. a typical RNN) but the theoretical analysis considers not a fixed nonlinearity, but rather the space of all possible dynamics that could govern the system.  I suspect this may misdirect some readers regarding the basic argument (possibly me as well).  \n\nTraining a regular RNN on the cost function, L, of course, does not guarantee anything about the cost function L_upper, so one must make some additional assumptions about the structure of the network dynamics, namely something akin to demanding that the space of permissible dynamics admits the derived minimum of L_upper.  In the case of the numerical experiments, the authors use ReLU and decompose the input to the nonlinearity into a recurrent part and a signal integration part, which should approximately allow for the decomposition that leads to L_upper.  One needs to add an argument that this decomposition is reasonable for certain classes of RNNs. E.g. If the network is in the active regime, the ReLU doesn't contribute much and then the composition amounts to considering the activity updates from the recurrent part and the input part separately.  One could make a similar argument about sigmoids.  In short, the authors should, I think, be direct about the assumptions on the RNN necessary for their theory. \n\nRegarding the quiescent state, I think the authors are perhaps a little too cavalier with the fact that, strictly speaking, the noise term must change in order to guarantee replay.  As presented at the moment, it feels to this reader like a weakness that is not adequately addressed.  The authors should either supply 1) a reason to believe this sort of change happens in real networks or 2) a demonstration that the deviations are not sufficient to qualitatively change the main result.  There is also the possibility that this is an opportunity for the authors.  The deviations from perfect replay if the noise does not change amount to a prediction, and one which may have computational advantages (e.g. facilitating exploration in terms of a planning algorithm).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFpDyO3Yej", "reviewer": "Reviewer_NzXs", "summary": "The paper proposes a training algorithm for continuous-time RNN integration in the presence of noise that relies on an upper bound for the loss that is split into a denoising and a tracking part. It is then shown for a angular and a motion velocity integration task that the noise compensation dynamics induce diffusive reactivations in a quiescent state.", "questions": "How is the bias exactly defined for figure 2?\nIn Figure 2, what distribution are random networks sampled from? Further, what does it mean that a random network s trained and tested in the presence of noise (as would be implied in the column $\\text{R}^\\sigma$).\n\n\nWhy is the kernel density estimate the best (or a good) way to compare trained networks?\n\nHow does the final trained network in terms of its parameters compare to the full greedily optimal dynamics?\nHow do different trained networks compare to each other in terms of the reactivation statistics?\n\nIs it truly a ring attractor that solves the angular integration task? On the given time scale it could be the case that the solution is provided by a line attractor that gets mapped to the ring by $D$.\n\nIn Equation (17) is the optimum unique?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel method of task-optimization for RNNs that rely on an upper bound for the loss that is split into two parts: the first term optimizes for denoising and the second term is optimized for tracking the signal. This is the first paper that discusses the contribution of learning in the presence of noise to have implication to reactivations.\nThrough this framework a new perspective is proposed on reactivations in the brain being the consequence of the presence of noise.\nFinally, this framework provides interpretability to the system that solves an integration task in terms of what part of the dynamics corresponds to denoising the systems and which to integrating instantaneous changes of the state variable.\nThe effectiveness of the approach is demonstrated against a couple of integration tasks highlighting its applicability. The theorems and proofs are sound.\nFurthermore, the paper is well written and the contributions are clearly explained with possible implications of the work for neuroscience.\nThese implications for neuroscience might provide a new perspective on how neural dynamics that implements neural integration might be decomposed into a denoising and an integrating component and how networks that learn in the presence of noise then exhibit reactivations.", "weaknesses": "It is unclear what is meant with tuning of the value of $\\tau$. Is this for a fixed noise level? Or is it chosen as the optimal for all used noise levels for training?\n\n\\paragraph{Figures}\nUse some transparency in Figure 1 f and e so that overlapping points are visible. At the moment it is unclear what proportion of the points is hidden behind the first layer and what their distribution is.\n\n\n\\paragraph{Contribution}\nThe the demonstration contribution of this framework could benefit from some additional experiments.\nComparison to training without noise and added noise in quiescent state is missing.\nComparison to other ring attractor/head direction models is missing.\n\nFinally, some of the implications of the proposed framework require more justification.\nThe conclusion that reverse replay could be explained by diffusive reactivations seems like a stretch and should be substantiated better. \nAlso, reactivations such as replay are not the type of reactivations that are found in the networks in the paper (this is admitted in the paper) even though the introduction is discussing those for a large part.\n\n\\paragraph{Noise}\nThe fact that training without noise resulted in erratic output trajectories might be explained by the statistics of the input. \nDoes the used input static reflect best what happens in animal behavior?\nIt would be good to compare the statistics of state sequences and reactivation sequence on the level of the sequences themselves.\nBecause in terms of optimality of exploration erratic trajectories might be more optimal to fully explore, see for example McNamee (2021). But see also Supplementary Figure A.1 (a) vs (e) that seems to show that a network that  has been trained without noise (e) explores a bigger part of the state space than a network that has been trained in the presence of noise (a).\n\nFinally, the claim that even with the addition of noise the failure of exploration could not be corrected (page 8, middle) should be better substantiated with a comparison based on distributions rather than just example trajectories (currently Suppl.Fi. A.1e-f is used as justification).\n\n\nMcNamee, D.C., Stachenfeld, K.L., Botvinick, M.M. and Gershman, S.J., 2021. Flexible modulation of sequence generation in the entorhinal–hippocampal system. Nature neuroscience, 24(6), pp.851-862.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)