Memory by accident: a theory of learning as a byproduct of network stabilization
Authors: Basile Confavreux
Will Dorrell
Nishil Patel
Andrew M Saxe
Venue: NeurIPS 2025 poster ()
Keywords: Synaptic plasticity
Spiking networks
Toy models
Learning and Memory
Familiarity detection
Primary area: neuroscience_and_cognitive_science
TL;DR: We show that learning rules that are optimized to stabilize networks also create memories as a byproduct, in linear rate networks and large recurrent spiking networks.

Abstract: Synaptic plasticity is widely considered to be crucial to the brain’s ability to learn throughout life. Decades of theoretical work have therefore been invested in deriving and designing biologically plausible learning rules capable of granting various memory abilities to neural networks. Most of these theoretical approaches optimize directly for a desired memory function; but this procedure can lead to complex, finely-tuned rules, rendering them brittle to perturbations and difficult to implement in practice. Instead, we build on recent work that automatically discovers large numbers of candidate plasticity rules operating in recurrent spiking neural networks. Surprisingly, despite the fact that these rules are selected solely to achieve network stabilization, we observe across a range of network models -feedforward, recurrent; rate and spiking- that almost all these rules endow the network with simple forms of memory such as familiarity detection - seemingly by accident. To understand this phenomenon, we study an analytic toy model. We observe that memory arises from the degeneracy of weight matrices that stabilize a network: where the network lands in this space of stable weights depends on its past inputs---that is, memory. Even simple Hebbian plasticity rules can utilize this degeneracy, creating a zoo of memory abilities with various lifetimes. In practice, the larger the network and the more co-active plasticity rules in the system, the stronger the memory-by-accident phenomenon becomes. Overall, our findings suggest that activity-silent memory is a near-unavoidable consequence of stabilization. Simple forms of memory, such as familiarity or novelty detection, appear to be widely available resources for plastic brain networks, suggesting that they could form the raw materials that were later sculpted into higher-order cognitive abilities.

Memory by accident: a theory of learning as a byproduct of network stabilization
Basile Confavreux
William Dorrell
Gatsby Computational Neuroscience Unit University College London
Gatsby Computational Neuroscience Unit University College London
basile. conf confavreuxx@gailll........e... com
Nishil Patel
Andrew Saxe
Gatsby Computational Neuroscience Unit University College London
Gatsby Computational Neuroscience Unit University College London
Abstract
Synaptic plasticity is widely considered to be crucial to the brain's ability learn throughout life. Decades of theoretical work have therefore been invested deriving and designing biologically plausible learning rules capable of granting various memory abilities to neural networks. Most of these theoretical approaches optimize directly for a desired memory function; but this procedure complex, finely-tuned rules, rendering them brittle to perturbations and difficult implement in practice. Instead, we build on recent work that automatically discovers large numbers of candidate plasticity rules operating in recurrent spiking neural networks. Surprisingly, despite the fact that these rules are selected solely to achieve network stabilization, we observe across range of network models- feedforward, recurrent; rate and spiking- -tht almost these rules endow the network with simple forms of memory such as familiarity detection seemingly by accident. To understand this phenomenon, we study an analytic toy model. We observe that memory arises from the degeneracy of weight matrices that stabilize a network: where the network lands in this space of stable weights depends on its past inputs-thal is, memory. Even simple Hebbian plasticity rules can utilize this degeneracy, creating a zoo of memory abilities with various lifetimes. practice, the larger the network and the more co-active plasticity rules in the phenomenon becomes. Overall, our findings suggest that activity-silent memory is near-unavoidable consequence of stabilization. Simple forms of memory, such as familiarity or novelty detection, appear to be widely available resources for plastic brain networks, suggesting that they could form the raw materials that were later sculpted into higher-order cognitive abilities.
lead
system, the stronger the memory-by-aceedde
Introduction
Understanding how the brain orchestrates its plastic synapses is holy grail of neuroscience, potential stepping stone on the way to generalist models of brain function. Yet, despite concerted effort, empirical plasticity data remains scarce, largely due to the difficulty of recording synapses in vivo during learning. Consequently, theory and computation have played an out-sized role in developing and testing plasticity hypotheses [T] Most theories derive their proposed learning rules by optimizing for some reasonable neural goals; for a memory system these might include the capacity, reliability decodability of memories, while enforcing biological plausibility constraints such as
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

locality or slowly changing weights (2.]]. Rules so derived are promising candidates, but their fine-tuning to specific function can make them brittle to perturbations and hard to implement in practice [6-8. Besides, though frameworks analyzing the learning dynamics of spike-timing dependent plasticity rules at steady state in spiking networks exist-mainly mean-field methods [9] it remains challenging to capture analytically the interactions between co-active plasticity rules operating at different synapse types in parallel, restricting most theories to study rules in isolation.
11
Instead, recent work has used numerical optimization to propose candidate learning rules, via meta- learning techniques [2 19]. This has led to the discovery of entire families of novel co-active learning rules able to robustly stabilize large recurrent spiking networks [19] 20] Interestingly. despite being selected only ensure that the spiking network stays stable, the majority of rules display range of interesting memory behaviors, such as novelty detection, contextual novelty and replay 20). seems that basic memory abilities are a natural byproduct of network stabilization.
Here we seek to understand this link between stability and memory. By stability, we mean networks with "biologically plausible" activity and weight dynamics over wide range of inputs i.c. plausible asynchronous neural activities and slowly changing weights bounded within reasonable ranges as in previous work (20]. We employ a broader definition of memory in this paper than, for example, associative memories in Hopfield networks [21.. Instead, we focus on familiarity/novelty detection: "the ability to discriminate between the relative familiarity or novelty of stimuli" [22] which has a
long history in psychology, experimental and computational neuroscience [23] [2 24]. This form of
memory is simpler, and can exist without, for instance, pattern completion or an attractor state for each memory.
We study range of network models from large recurrent spiking to shallow linear feedforward rate networks-and show that the link is robust: across models, learning rules built to stabilize neural activity consistently encode memories with various lifetimes and properties. By reverse engineering spiking networks and studying analytic toy models, we show that these memories have a simple origin: for any input there is a degeneracy of stable weight matrices. Which matrix the learning rules select often depends on the network's history, providing the basis for memory storage and recall. Large-scale simulations suggest that the bigger the system and the more co-active learning rules, the longer-lasting and more robust the memories. The remarkable case with which such memory abilities arise in stable spiking networks may form the basis of higher-order cognitive abilities as compositions of simpler, ubiquitous, memorization skills. As such, we present these ideas as fresh take on the emergence of memory in the brain: memory by accident.
Memory is a common byproduct of stabilization by co-active plasticity rules
Our starting point was a manifold of co-active synaptic plasticity rules that enforce stable dynamics in large recurrent spiking networks. There are four types of synapses in these networks Excitatory (E)-to-E, E-to--nbbbbttory (I), I-to-E and I-to-I- each governed by its own plasticity rule. The rules are parameterized by their dependence on the pre-synaptic spike train, post-synpatic spike train and a Hebbian term dependent on both (fig. M (0)):
dw(() dt (1 + KXY:Fposi (1)) + Speos (Bxr + pe (e)))) X, E (E,1)
(I)
Pre-symaptic
Hobbian
Post-synaptic
Hebbian
where Spre(t) and (t) are spike trains, ppre(t) and Zppaa (t) are their low-pass filtered versions. In sum, for each plasticity rule, there are 6 parameters, four as in eq. ① (a, 3, k), and two timescales 7 ), one for each low-pass filtering. Recent work meta-learned thousands of choices of these parameters that led to networks with stable dynamics- meaning the activities and weights in the
networks remained in plausible ranges across many inputs for at least 4 hours [20). This led to a "stability manifold". a stable subset of the 24 dimensional plasticity parameter space.
Despite selecting these rule quadruplets for stability, previous work observed that simple forms of memory were a near ubiquitous byproduct of network stabilization (20). For example, the networks were tested on familiarity detection task (fig. [A]: during training period the networks were presented with a subset of the stimuli, then, after variable time period, the network weights were frozen and the network was presented with both novel and familiar stimuli. The network was said to remember the familiar stimuli if the average firing rate was significantly different between the familiar and novel stimuli presentations. In this task almost all the rules showed some form of memory, with lifetimes ranging from seconds to hours (fig.) IB). In other words, most rules created memory traces
2

---

A Familiarity detection task
Memory lifetimes of stable
AWEY
100
meta-learned rules
C
Exc
Inh
do of rules
AWES (0)
AW (0)
(w)
Novel stimuli
Familiar stimuli
05
10s
1min
Smin 20min Umemn
4n
C
Emem (O),
0.20
0.5
0.0
reg. coeffs
Fexc (4h) (4h) R(0,, ,² 0.72
21
(nn(44) RO). ,² 0.67
0
1
T
Tpost
3 K
Tpost
B
Tpost
Epost B K
Figure 1: A: Recurrent spiking network (Ne = 4096, Ni 1024) with four co-active plasticity rules undergoing familiarity detection task. B: Distribution of elicited memory lifetime in the familiarity task among 2500 stable meta-learned rule quadruplets. Memory lifetime was defined as the last time point at which the network population firing rate was significantly different during novel vs familiar stimulus presentation (Student t-test, p < 0.05 for Nunals = 5). Original data from [20]. C: Linear regressions, predicting from the values of the 24 plasticity parameters: the memory lifetime of each quadruplet on the familiarity task (top): the mean firing rate of the excitatory population after 4h (middle); the mean inhibitory firing rate after 4h (bottom). Dataset of 2500 rule quadruplets taken from 20 While mean rates can be readily predicted from plasticity, memory lifetimes cannot.
with rehavioraall--elleann lifetimes from single stimulus presentation, despite having Hebbian pairing windows of tens hundreds milliseconds.
We wondered which features of the plasticity rules determined their memory abilities. Simply linearly predicting the memory lifetime from the plasticity parameters impossible (fig. (C) while, control, it was possible to predict other, more basic, network properties such as the excitatory and inhibitory mean firing rates using this strategy (fig. [C). To understand this memory-by by-cideen phenomenon, we therefore take different approach. Tn the following sections, we reverse engineer these memories across range of models, including a simple analytic toy model, before returning to test the intuitions we develop on the full co-active spiking networks in the final section.
3 Reverse engineering memory by accident in spiking networks
3.1 Recurrent spiking network with four co-active rules
First, we investigated how the simple co-active rules from the stability manifold were both creating long-lived memories and enforcing stability in recurrent spiking networks. We focused
rule
quadruplet which responded significantly to the familiarity task for over 4h (fig. A This quadruplet stabilized the network by driving
an activity setpoint at around 2Hz during pre-training phase of random background inputs (fig. [A). The network was then perturbed by the input of the (not yet) familiar patterns, though the activity recovered its pre-training setpoint less than minute later (fig. N However, probing the network with either familiar or novel stimuli elicited responses of different magnitudes (fig. E2 and fig. [22), and, as advertised, these differences persisted over the remaining hour of simulation. The meta-learned quadruplet thus elicited form of long- lasting. activity-silens memory (25). Further, unlike the network activity, neither the mean or variance the weights recovered to their pre-training values (fig. A). This indicated that the weight matrices had experienced long-lasting changes that enabled the different network responses observed for familiar novel stimuli.
3

---

To analyze the weights in this network we defined the "engram" for each stimulus as the neurons with the highest 10% of activities in response to the stimuli. Since, by construction, even the neurons in naive networks were tuned to particular inputs (see fig. S2) this engram was meaningful even during pre-training phases. We observed that training mainly affected the weights to and from the familiar stimuli's engrams (fig.BB. time point 2, strengthening of WIE engram weights, and weakening of WEE engram weights), and these speccfc changes persisted long after the end of training (fig. 2B, time point 3). We hypothesize that it is these familiar-specific weight changes that enabled long-term memory recall. other words, among the many (WEE, WEI, WIE, Wn)) matrices able to stabilize the network in background state (the degeneracy of solutions to the stabilization problem), the specific weight matrices reached by plasticity reflected the system's history-heer past input stimuli-givee rise to memory.
A
B Post-synaptic neurons
0.2
At
200 ms
5
fbl
Pre-synaptic neurons
0.055
Tnov
Faam
0.3
10.05
(Wee)
AUJ.
olWee)
(Wri)
0
o(Wai)
(wie)
O(Wie)
(We)
8 0.72
o(Wi)
5
60
Train
Break
Feedforward spiking model
15
- Tnov
Hz
Tham
LIF
Poisson inputs
(Wic) o(WiE)
10
-ral -Tnov -Iaa
60
H7
5
MMm un
Time since training (min)
Excitatory neurons
0
Inhibitory
3.5
WI
Train
5min
1h
Time since training (min)
Figure 2: A: Example meta-learned rule quadruplet undergoing the familiarity task. Top: visualization of the rule quadruplet- normalised weight changes elicited by pair of pre-synaptic and post-synaptic spikes with different time-lag between spikes. Second plot: excitatory population firing rate without any active stimuli (red, baseline) during the task, dashed lines denote the start and end of training. After training, average excitatory population firing rate in response to four novel (green) and three familiar stimuli (purple). Third & Fourth plot: Evolution of the mean and standard deviation the weights during the task. B: Weight matrices of the network for each synapse type, at three time points: (1) immediately before the start of training, (2) immediately after training, (3) one hour after the end of training. We sort the weights according to neuron response class: green neurons responding to novel patterns; purple neurons responding to familiar patterns (shown during training); grey all other neurons. C: Same as A&B, but for recurrent spiking network with iSTDP (I-to-E E only, (II). D: Feedforward spiking network model. Excitatory and inhibitory Poisson inputs project on a single output neuron. The familiarity task followed similar structure to the recurrent case. Middle: output neuron firing rate. Bottom: evolution of the 200 inhibitory (plastic) weights. Weights from input neurons belonging the familiar stimulus are shown in purple. Means of the groups ("familiar" weights vs rest) are in bold.

---

3.2 Recurrent spiking network with only I-to-E plasticity
We next sought to test our intuitions in a more tractable setting recurrent spiking networks with a single operational plasticity rule. Within the stability manifold we recognized iSTDP, widely-used rule for stabilising spiking networks [T] 26 (7]] This is symmetric Hebbian rule operating only on the I-to-E connectivity that has been studied analytically (fig. (C). Networks operating under this rule are known to settle to a stable mean firing rate whose value can be calculated using a mean-field approach
We simulated networks evolving under iSTDP and observed very similar patterns to that of the long-term memory quadruplet: the activity settled both before and after training to a setpoint (3Hz). Further, despite showing no activity-related memory traces, an imprint of the familiar stimuli could be seen in the weights (fig. (C). though in this case the memories lasted only a few minutes. It seemed that the network exhibited a separation of timescales: fast dynamics that restored the activity to the set point, and slower dynamics that erased the imprint of the memory from the connectivity weights (fig. (C). This separation led to a period of time in which activity was flat yet the network was able to produce reliable cemmoryby--aciideen matching our findings in the quadruplet rule case.
3.3 Feedforward spiking network with I-to-E plasticity
Finally, we study one further simplified spiking network: a feedforward network undergoing the familiarity detection task (fig. 2D). This model comprised a single eakky-nneeratte-ndd- neuron receiving inputs from 800 excitatory and 200 inhibitory Poisson neurons. Only the inhibitory weights were plastic, following the iSTDP rule [Τ]. Initially, all input neurons fired at the same rate, then we performed stimulus presentation by elevating the activity of 100 excitatory and 25 inhibitory input neurons- the "engram neurons" in this setting (see Supplementary for more details).
output
Once again, transient memorization of the familiar stimulus could be seen (fig. P)). Matching our observations in recurrent networks, we observed that weights from inhibitory engram neurons relaxed towards the background weight distribution at two different timescales after stimulus presentation. Fast dynamics with time constant of few seconds returned the output neuron's 's firing rate to the firing rate set point; while a slower relaxation, lasting from minutes to hours, erased the imprint of the memory from the weights. Since the only force that drove weight change in this network was deviation of the output neuron's ' firing rate from a target, these slower changes seemed to be driven by random fluctuations that slowly overshadowed the memory. leading to observable memory- byy- accident phenomena persisting for behavioral timescales. We also verified that this insight was not specific to I-to-E plasticity by running a similar network and task with an E-to-E rule instead (fig. (S5).
Overall, our analysis of each plastic spiking network confirmed the idea that the degeneracy of weight configurations capable of stabilizing the network provided the substrate for the formation of "accidental" memories. To understand this precisely we now turn to some analytic toy models.
4 Building toy model for memory by accident
4.1 Explicit feedforward model
So far we have analyzed few networks, each following a single set of co-active plasticity rules. However, these simulations do not help us to understand the ubiquity of memory-by-accided -cciieet across the set of all stabilizing plasticity rules. To that end, we build an analytic toy model from which we can derive the memory properties networks following various stabilizing plasticity rules. We thus turn to minimal firing rate model capable of self-stabilization.
We considered linear feedforward network with two inputs x (zo, #1) and single output y: (t) Wo(t (o(t)xro(t) + wi (1(t)x1(t) (fig. 3). All activities were nonnegative (though weights were unconstrained), and for simplicity we made the inputs unit norm. We considered a four- porrmmarreeter set of Hebbian/non -ebbiian plasticity rules inspired by the full spiking model:
(w,(t) Gu (t), + (2v(t) + (t)(())((t) Øt
Not all these plasticity nules are meaningful, we therefore restricted to rules that produced stable output activity for all possible inputs, redefinition of stability for this simple feedforward model. In
5

---

A-B-A task
(t)
Xng
Xstim
Xag
1
Figure A: Feedforward linear network with two input neurons and one output neu- ron. B: Simplified version of the familiarity task. Starting from steady state solution for Tbg. the network was shown Dstim un- til convergence then 2bg until convergence. C: Dynamics of four learning rules in the A-B-A task, shown in weight space. All networks were initialized at the same state Wo W The denoted as "start". D: Phase portrait of the relative improvement R. as function of the values of 6g and 0,. This was computed for 2 = and Oz = 6 + 4' Grey denotes unstable rules. E: Same pa- rameter sweep as C, but plotting the time to convergence т.
x
(+)
wy (t)
0
0
t
W. Xstmm
Vig.
x -y")
(-x: y
W2
dm (y-)")
start
W. Xbg Veu
da
(0. 60. 1xy y")
W1
E 2
z 103
T scaie
10a
10a
-2
-2
2
-2
0o
0
particular, we chose a target output firing rate, y", and derived constraints the plasticity parameters (0, Josiss such that the output activity eventually converged to y no matter which unit norm input was presented (see Supplementary for derivation). These assumptions reduced the set of feasible learning rules to the following two-parameter model:
(w,(t) (v(() (0, + (1x;(()), s.l. O, +0 < and 200 + +01 <0 Øt
(3) In this setting we devised a simplified task assess memory, the A-B-A task BB). The network received two stimuli, Fbg and Tsim, both unit norm and aligned at random, smaller than 90", angle from the x axis. For each input there is stable subset weight configurations, and in this case, due to the linearity of the problem, they form line. We define W [w, w.F y²} as the line of weight vectors that produce a stable output firing rate (y = y²) for a given input æ. We initialized the network at wo W 2nn" i.e. at a stable point for the background input Thg. We then presented the stimulus, ran the network to convergence, before returning again to the background input and running convergence (fig. (C). In this simple setting, we defined the "memory" of the system in two ways. First via the relative improvement RI:
RI(O,, 01,y")= (Why' Whg) (Whgmsttm Why
(4)
Whggastm Wigg 1
Whg denotes the steady state weights for input The the start of the task while Wogr are those for Tng at the end the task. Whgnssim is the intersection between W ang and W main" The magnitude of RI encodes the distance between the initial and final state of the network, while its sign indicates whether the final state was closer to the intersection between the lines of fixed points not. Zero indicates that the final weights are identical to the initial weights, while positive (negative) values indicate net movement towards (away from) the weight matrix that stabilizes both Tbg and Daim- Second, we evaluated memory with the time to convergence to the fixed point 7(0, 0,, yy) min (t, (t) - y"1 S p). with p = 0.01 a threshold.
Almost all rules exhibited some form of memory in this toy model, i.e. the final and initial network states differed indicating dependency on the past input Fsum (fig. (D&E). Both the ubiquitous exis- tence of memory, and the wide diversity of memory timescales were consistent with the observations made the spiking models (figs. aand 2)), suggesting that the degeneracy of weights was indeed the key factor in the memory by accident phenomenon. In fact, the only rules that did not yield memories were the purely non- non-Hebbian ones (0, 0). We verified that these results held qualitatively when extending this toy model to higher dimensions (see Appendix and fig. S8)

---

Besides qualitatively reproducing the observation that most stabilizing plasticity rules exhibit memo- ries, this toy model led us formulate several predictions:
1. Trade-off between memory and stability: rules that elicit the strongest memories--logeer-lavte and more robust- should be closest unstable regions (fig. 3DFE..
2. Learning rates and memory: a priori, the role of the learning rate was unclear to us, as influenced both the learning and forgetting phases. In the toy model, the learning rate of the rules did not affect RI-the fixed points reached-only the speed convergence т. Therefore lower (non-zero) learning rates should lead to longer-lasting memories.
Hebbian vs non-Hebbian terms and memory: the model predicted that the only rules that do not dwl
elicit memory are exclusively Hen-Hebbian: "t x y y*. Further, rules combining the plasticity parameters could be more efficient, i.c. with longer-lasting memories, than exclusively Hebbian or non-Hebbian rules.
4.2 Feedforward implicit model
A
B
a=1
c=1.0
>1.
3
b=0
start
C=1
22
W. Xbg
Ytg
a=1
b=-1
0 ž
1
C=1
0
W. Xstim
Ytg
a=1
b=0.5
2 a
< 1.1
-1 0
C=1
W1
Figure 4: A: Fixed points for three rules of the implicit toy model in the A-B-A task, shown in weight space. All networks are initialized at the same state wo W B: Phase portrait of the relative improvement RI function of the values of and b, for fixed This computed for O m.., and θ am
6 + 4- Grey denotes unstable rules.
One potential concern with our previous toy model (and indeed, all of previous approaches) is the dependence on particular parameterization of the learning rules. In this section we therefore defined more abstract class of plasticity rules, operating in the same linear feedforward network as in the previous section. In the vein of recent work (28), we forego an explicit parameterization and instead considered rules that minimize different distance metrics in weight space. More specifically, we again considered learning rules that lead to stable output firing of y² for any input x:
A (æx yo: Wo) E 2x122 lim y(t, ar yo. wo) 1²
(5)
"++xx
Then we assumed that the stabilizing weight configuration that network would choose W would be the closest according to some distance metric:
(x, yo: wo) E 2xxxx lim w(t, x. you wp) argmin lw - woll weW:
(6)
"1++00
with |1-l/x the norm induced by the Mahalanobis distance D:
v(W1, W2) E 2x222 Dz(W1, w2) V - w2) T ((w W2)
a b) [ b c leaving with three plasticity parameters: a, b, and, To be well-
We defined Σ
defined distance, Σー needs to be positive semidefinite leading the further constraint that > 0 and ac - 2 From these assumptions, we could calculate the system's 's fixed point as a function of the initial state (see Supplementary for the full derivation). Despite only describing the fixed points, not the dynamics used to reach them, we found some matches between explicitly and implicitly parameterized rules (fig. S For instance, the rule minimizing the L2 norm (a 1, = 0, 1) corresponded to 6n = 0, d 1.2 mean squared error between the current activity and the desired activity y" (see Supplementary).
in the explicit model, which could also be seen as minimizing the
Again, it appeared that with these stabilizing rules, memory by accident was the rule and not the exception (fig. 43). This arose for almost all choices of metric minimization, independent of the exact
7

---

dynamics chosen to implement these weight updates, suggesting our ideas generalized beyond the particular plasticity rule parameterization we might choose.
Testing insights from toy models in spiking networks
Finally, we returned to the full recurrent spiking network with four co-active plasticity rules to test the predictions from our toy models.
A
-noov
Tiam
B
Tnov
- faam
C
noo
- famm
D
15
10
-1
n
0.005
B 0; Y, ==
dww Ut
+
5
0
-1.2
30
60
15
10
0.01
B 0.2; V, K
0.5
initial quadruplet stable pert.
1
AEE
1.2
5
unstable pert.
50
15
20
rules with T.em S 10s
0.02
B 0.4: K
0
rules with Tmem 4h
30 60
30
60
1 2 3 4 5
Distance to instability (1) Figure 5: A: Recurrent spiking network with variants of iSTDP (I-to-E plasticity only) in the familiarity task, with different learning rates. Simulations were repeated times, averages across seeds are shown in bold, dashed lines denote the last timestep at which the population firing rate in response to familiar stimuli was significantly different to novel responses (Student t-test, p < 0.05). B: Feedforward spiking network with I-to-E plasticity in the familiarity detection task. Three rules were tested from top bottom: - /poo 20ms,
Time since training (min)
-0.12, 3 = 0, = 7 = (original iSTDP) pre = post 20ms. = 3 0.2, = = 0.5 (half post-only, half Hebbian), prr post 20ms, a = 0.12, = 0.4, k = = 0 (see SS for full simulations). C: Same as B. but for different I-to-E plasticity rule with spike-independent term, du 3S (t) C, de
with 3 and C' plasticity parameters, and Spee the pre-synaptic spike train. D: Top: example base rule quadruplet in black, and stability of perturbations along 10 random directions in plasticity parameter space. Bottom: distribution of indices of the first unstable perturbation along each direction. Results for 10 quadruplets with long memory lifetimes Tmem 2 4h and 10 others with Tmem S 10s.
Learning rate and memory: We varied the learning rate in the recurrent network with only iSTDP (section 3.2); within the range tested, the slower the rule, the longer the memory lifetime (fig-5A) predicted.
5A), as
Trade-off between memory and stability: we sought to verify that rule quadruplets with the longest memory lifetimes were those closest to the edge of the stability manifold. Calculating such distances involves simulating networks perturbed in all directions in the 24 dimensional parameter space, so is very compute-intensive. Here, therefore, we only report a trend when perturbing 10 stable rule quadruplets with memory lifetimes of 10s or less vs perturbing 10 stable rule quadruplets with memory lifetimes of 4h or more (D)). Overall, the distance to the instability border depended both on the rule quadruplet and on the direction chosen, and marginally more directions were immediately unstable for the long-memory quadruplets. Whether this is reflection of a limited-compute budget or a real phenomenon remains to be seen.
Hebbian vs non-Hebbian terms and memory: we focused on I-to-E plasticity, to avoid potential confounds induced by co-active rules. Using mean-field analysis as in previous work (IT), we derived the activity setpoint for spiking rules from eq.
-OpTTin inh
Texe
SIE + rinh (RIET post
pre + METE
(8)
The iSTDP rule 11 chooses 3T. O, thus simplifying rinh out of the equation, which effectively ensures network stabilization for all inhibitory activity levels. In recurrent networks, rules with Pm # 0 display unstable network dynamics (see fig. (SD)). However, many unstable I-to-E rules were in fact stable as part of quadruplet (20). To be able to test these rules in controlled setting,
8

---

we reverted to the feedforward spiking network. We tested Lto-E rules that had the same learning rates and target firing rates, but which traded off Hebbian terms i. and y for the post-only term B. Surprisingly, the more post-only potentiation, the longer-lasting the memory (fig. 3)
These experiments verified two predictions once, though in unexpected way. First, adding terms besides the Hebbian term could indeed be useful for eeemorybb--aciderr Second, we found an unexpected verification of the stability-memory trade-off: the higher the which destabilizes the rule, the longer the memory. Finally, we verified that the only rule predicted not to elicit memory by accident, aww du B.Spp pre (t) + C. Note that such rule was not part of the search space defined in eq. ( rule, dr
8t X y" behaved as predicted in the spiking case. To do so, we defined an equivalent
We verified that did not elicit memory by accident the feedforward spiking model (fig. 5CC)
6 Discussion
This study sought to understand puzzling observation: almost all plasticity rules that produce stable network activity also lead to memory abilities [20]. By studying three different spiking network models we linked this phenomena to the degeneracy of weight matrices capable of stabilizing network for a given input. In essence, from amongst the degenerate space of stable weights, the configuration that the plasticity rules chose depends on past inputs, creating förm of memory. We demonstrated the existence of this memory-by neemory-b--cccide by-accident phenomena all the way down to a 3-neuron linear rate model. We then used these tractable toy models to understand the phenomenon's 's ubiquity, finding it in nearly all rules, whether parameterized explicitly or implicitly. Instead of seemingly fortunate accident, our analysis leads us to pose basic memory abilities as a unavoiaaeee consequence of network stabilization.
In our toy models the only non-memorizing rules were purely non-Hebbian; do our findings boil down to "Hebbian learning creates memories"? We think not. While the link between Hebbian rules and memory is natural and longstanding, we find a wide diversity of combinations of Hebbian terms that produce long-lasting memories. Further, we found in our feedforward spiking models that the rules with the longest-lived "memories-frooe fre---aci-aacett" were exclusively non non-Hebbian (fig. 5B). Rather, we argue this is a generic property of stabilizing plasticity rules.
and
A promising aspect of memory by accident is that it seems to benefit from the system's 's complexity: the more neurons and weights, the more degenerate solutions; the more co-active learning rules, the more opportunities to exploit this degeneracy. This near-trivial phenomenon in toy models transferred to large recurrent network models and elicited complex and long-lasting memory abilities. Our recurrent spiking models, though relatively complex by today's s standards, are but a simplistic reflection of brain regions, which are composed of orders of magnitude more neurons and synapses, and use hundreds of different synapse types (29), each of which could have their own plasticity rule. For now we can only speculate on the potential computations that this phenomenon could unlock in systems of comparable scale to the brain.
The memories we have been discussing exhibit a diversity of timescales, from seconds to hours (and potentially beyond). While our longer-lasting memories look like classic episodic memories that have often been modeled with spike-timing dependent plasticity rules like ours, the shorter of these timescales instead match those discussed by the activity-silent working memory literature (25). Interestingly, classic models in this area rely on qualitatively different plasticity mechanisms such as short-term plasticity (3) 31 It is exciting that our theory proposes a unifying explanation for both phenomena operating on different timescales.
The systematic emergence of basic forms of memory from relatively simple unsupervised and local plasticity rules could also provide hints on the evolution of the complex learning abilitics observed. For plastic neural networks to be useful they must be stable. We find that as soon they are stable they permit memory, potentially presenting an easy stepping stone to higher-order cognitive abilities from compositions of readily available memory motifs.
Given the compute-load of simulating large plastic recurrent spiking networks with a high dimensional plasticity space, we could only partially verify the predictions made by the toy models. Moreover, many other phenomena influence the memory of the system besides those captured in the toy model. For instance, noticed that some rule quadruplets don't respond to any
Limitations:

---

memory task, but have very high firing rates (> 30Hz), effectively making them recurrent driven and oblivious their inputs.
We have also not modeled the effects of co-active rules, which in practice appeared to be a key element to make memories generated by the pairwise rules robust and noticeable behavioural timescales [20). Indeed, most of the original set of co-active plasticity rules used as a starting point for this study [20] were unstable when considered in isolation, in mean-field model with co-activity (fig. ST. This effectively prevented us from drawing conclusions on their memory capacity in this work. However, understanding how rules unstable isolation can be stable together and produce longer-lasting memories than their individual counterparts is an important avenue for future work, perhaps using more refined versions of mean-fiell analysis than was performed here 32
10 33
We might also wonder how these mechanisms could be extended to embed lifelong memories. In our networks it appears that memories are ultimately erased by random fluctuations (fig. (2C), something not captured by our toy models. However, as a first approximation, the larger the RI (the further away the final state is from the initial), the longer the memory will last before being erased by noise. Further, some learning rules seem to be able to dramatically postpone this eventual forgetting, especially through co-activity (fig.24) AA). Together these mechanisms seem sufficient encode memory long enough for it to be consolidated by other systems, such as the wake-sleep cycle. On the contrary, a memory system with graceful forgetting and robust stability enforced as default may be desirable.
To sum up, our distillation of automatically discovered plasticity rules in large recurrent networks resulted in remarkably general and simple phenomenon, memory by accident, that highlights the unreasonable effectiveness of simple unsupervised rules at making memories.
Acknowledgments and Disclosure of F Funding
We thank Tim Vogels for his help and support, as well as Lukas Braun, Everton Agnes, Peter Latham and Antonio Sclocchi for useful discussions. This work was supported by Schmidt Science Polymath Award, the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/7/1//Z) and the Gatsby Charitable Foundation (GAT3850).
References
[1] Larry F Abbott and Sacha B Nelson. Synaptic plasticity: taming the beast. Nature neuroscience, 3(81)111781118, 2000.
[2] Erkki Oja. Simplified neuron model a principal component analyzer. Journal of mathematical biology, 15(3):267-273, 1982.
[3] Elie L Bienenstock, Leon N Cooper, and Paul W Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction visual cortex. Journal of Neuroscience, 2(1):32-48, 1982.
[4] Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A hebbian/anti-hebbian neural network for linear subspace learning: A derivation from multidimenssonal scaling of streaming data. Neural computation, 27(7): (177))4461119999 2015.
[5] Manu Srinath Halvagal and Friedemann Zenke. The combination of hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Nature Neuroscience, 26(11):1906-1915 2023.
[6] Abigail Morrison, Markus Diesmann, and Wulfram Gerstner. Phenomenological models of synaptic plasticity based on spike timing. Biological cybernetics, 98:459-478, 2008
[7] Friedemann Zenke, Guillaume Hennequin, and Wulfram Gerstner. Synaptic plasticity in neural networks needs homeostasis with a fast rate detector. PLoS computational biology, 9(11): e1 e1003330, 2013.
[8] Friedemann Zenke, Everton J Agnes, and Wulfram Gerstner. Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories spiking neural networks. Nature communicationss, 6(1):1-13, 2015.
10

---

[9] Richard Kempter, Wulfram Gerstner, and J Leo Van Hemmen. Hebbian learning and spiking neurons. Physical Review E. 59(4),4498 1999.
[10] Richard Kempter, Wulfram Gerstner, and Leo Van Hemmen. Intrinsic stabilization of output rates by spike-based hebbian learning. Neural computation, 13(12):2709-2744 2001.
[11] Tim Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerst- ner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science, 334:1569-1573, 2011.
[12] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. JJCN-- 91-Seattle International Joint Conference on Neural Networks, 2:969, 1991. doi: 10.1109/ 19CN..191.11558
[13] Luke Metz, Niru Maheswaranathan, Brian Cheung. and Jascha Sohl-Dickstein. Learning unsupervised learning rules. arXiv preprint, 1804.00222, 2018.
[14] Jack Lindsey and Ashok Kutmin.Kumar. Learning to learn with feedback and local plasticity. Advances in Neural Information Processing Systems 34 (NeurIPS), 2020.
[15] Basile Confavreux, Friedemann Zenke, Everton J Agnes, Timothy Lillicrap, and Tim P Vogels. meta-learning approach to (re) discover plasticity rules that carve a desired function into a neural network. Advances in Neural Information Processing Systems 34 (NeurIPS), 2020.
[16] Jakob Jordan, Maximilian Schmidt, Walter Senn, and Mihai A Petrovici. Evolving interpretable plasticity for spiking networks. eLife, 10:e66273, 2021.
[17] Danil Tyulmankov, Guangyu Robert Yang, and LF Abbott. Meta-learning synaptic plasticity and memory addressing for continual familiarity detection. Neuron, 110(3):544-537, 2022.
[18] Thomas Miconi. Learning to acquire novel cognitive tasks with evolution, plasticity and meta-meta-leamiig. International Conference on Machine Learning, pages 24756-24774 2023.
[19] Basile Confavreux*, Poornima Ramesh Pedro J. Goncalves, Jakob H. Macke, and Tim P. Vogels. Meta-learning families of plasticity rules in recurrent spiking networks using simulation- based inference. Thirtt-seventh Conference on Neural Information Processing Systems, 2023.
[20] Basile Confavreux, Zoc Harrington, Maciej Kania, Poornima Ramesh, Anastasia N. Krouglova, Panos Bozelos, Jakob H. Macke, Andrew Saxe, Pedro J. Goncalves, and Tim Vogels. Mem- ory by a thousand rules: Automated discovery of multi-type plasticity rules reveals variety degeneracy at the heart of learning. bioArXiv, 2025.
[21] John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 2955):2554-2558, 1982.
[22] Rafal Bogacz and Malcolm W Brown. Comparison of computational models of familiarity discrimination in the perirhinal cortex. Hippocampus, 13(4):494522, 2003.
[23] Lionel Standing. Learning 10000 pictures. The Quarterly journal of experimental psychology, 25(2):207-222, 1973.
[24] Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. Visual long-term memory has massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105(38):14325-1433) 2008.
[25] Mark G Stokes. 'activity-siled silent' working memory in prefrontal cortex: dynamic coding framework. Trends in cognitive sciences, 19(7):394 405, 2015.
[26] Ashok Litwin-Kuman and Brent Doiron. Formation and maintenance of neuronal assemblies through synaptic plasticity. Nature Communications, 2014.
[27] Auguste Schulz, Christoph Miehl, Michael J Berry II, and Julijana Gjorgjieva. The generation of cortical novelty responses through inhibitory plasticity. Elife, 10:e65309, 2021.
11

---

[28] Roman Pogodin, Jonathan Corford, Arna Ghosh, Gauthier Gidel, Guillaume Lajoie, and Blake Aaron Richards. Synaptic weight distributions depend on the geometry of plasticity, The Twelfth International Conference on Learning Representations, 2024.
[29] Zizhen Yao, Cindy TJ van Velthoven, Michael Kunst, Meng Zhang, Delissa McMillen, Changkyu Lee, Won Jung, Jeff Goldy, Aliya Abdelhak, Matthew Aitken, et al. A high-resolution transcriptomic and spatial atlas of cell types in the whole mouse brain. Nature, 624(7991): 317-332, 2023.
[30] Gianluigi Mongillo, Omri Barak, and Misha Tsodyks. Synaptic theory of working memory. Science, 319(5869) 2008.
[31] Matthijs Pals, Terrence C Stewart, Elkan G Akyürek, and Jelmer P Borst. A functional spiking-neure neurnn model of activity-silen working memory in humans based on calcium- caleium-mediated mediated short-term synaptic plasticity. PLoS computational biology, 16(6)=e107736, 2020.
[32] Ofer Hendin, David Horn, and Misha V Tsodyks. The role of inhibition in an associative memory model of the olfactory bulb. Journal of computational neuroscience, 4:173-182, 1997.
[33] Mark CW Van Rossum, Guo Qiang Bi, and Gina G Turrigiano. Stable hebbian learning from spike timing-dependee plasticity. Journal of neuroscience, 20(23):8812-882) 2000.
[34] Friedemann Zenke and Wulfram Gerstner. Limits to high-speed simulations of spiking neural networks using general-purpose computers. Frontiers neuroinformatiis, 8:76, 2014.
[35] Charles R. Harris, Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(78255:357-362, September 2020.
[36] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Banderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. 2018. URL htttp://iitub. com/ jaa-ja/xxaaaaa ml/jax
[37] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9
(3):90-95, 2007.
[38] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-274 2020.
[39] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Seikit-learn: Machine learning in Python, Journal of Machine Learning Research, 2-2885-2830, 2011.
12

---

A Technical Appendices and Supplementary Material
The code and data to reproduce the results in this paper can be found on Github Spiking network simulations were performed on single CPU using the Auryn simulator [4 The perturbations of rule quadruplets in fig. 5C (=:000 simulations) were performed on 500 CPUs the ISTA HPC cluster. Rate models and analysis of all simulations was performed using numpy [35]. JAX [6), matplotlib 33,, SciPy 38 and scikit-learn (39
A.1 Recurrent spiking network model
A.1.1 Network model
We considered recurrent spiking network with NE = 4096 excitatory neurons and Ni 1024 inhibitory neurons (leaky-integrate and fire point neurons with variable threshold, AMPA and NMDA currents, and conductance--asee synapses), This network was based on previous work 20]. The membrane potential dynamics of neuron j (excitatory inhibitory) followed:
d Tm dt
(1( = (V)(1) - Viea ) - (1) - FEE) (V)() Fn),
(9)
with Tm = 20 ms, Vres = 70 mV, ES. = mV and Ej = -80 m
A postsynaptic spike occurred whenever the membrane potential (() crossed threshold VTh with instantaneous reset Vreset 70 mV. This threshold (t) was incremented by Vith
spike
100 mV every time neuron j spiked and otherwise decayed following:
Tth dt
(t)(t)
V/uue (t),
(10)
with 1/th
-50 mV. The excitatory and inhibitory conductances, g and g evolved such that
of(() agj
AMPA
(t) + (1-a) NMDA (t) and
di 0)(1)=
d
o (t)
Σ (t)(t)
TGARA
Elln
96 JAMPAA (t)
(11)
with
dt g AMPA
Σ ws (t), and
TAMPA
d NMDA(t) 9j dt
9 AMPA (t)
9
NMDA (t)
TNMDA
with Way(t (t) the connection strength between neurons and j (unitless), a = 0.23 (unitless). TGABA 10 ms, TAMPA ms, TNMDA 100 ms, (t) Σ 6(t tt) the spike train of presynaptic neuron i, where I denotes the spike times of neuron k, and ā the Dirac delta.
The network was initialized with random sparse connectivity (10%), with WHE w 0.1 and wint wintt 1.
The excitatory neurons in the network received Nmp = 11025 inputs from Poisson neurons firing at "bg 2 When stimulus was active, subset of the input neurons increased their firing rate "'ctive 100 2. The connectivity from input neurons excitatory and inhibitory neurons for each recurrent neuron, we selected a random input neuron as the center
inp
was receptive-field-iie of the circular receptive field of radius 8. The connections from neurons of this circular patch of input neurons to the considered recurrent neuron was Winp = 0.075, and 0 to all other input neurons. The inhibitory neurons received inputs from Nap->> 4096 Poisson neurons with Winp and similar receptive field connectivity than for the excitatory population. However, inhibitory neurons only input, and no specific stimulus patterns. received background inputs (rog
13

---

A.1.2 Plasticity parameterization
This parameterization of plasticity rules included variations of teiiiiindeeeendeeee plasticity, and was taken from previous work [19] 20). The weight from neuron to neuron j of type X and Y (excitatory inhibitory) evolved such that:
dw,, (t) n/(s(t) (axx KXY +t) S,( (8xx + (xx=:(t))
(12)
dt
12) the spike train of neuron i, the Dirac delta function denote the presence of pre (post)-synaptic spike time t. The synaptic traces I; and zy are low-pass filters of the activity of presynaptic neuron i and postsynaptic neuron j, with time constants Tpre and Tpost, such that:
with 7/ = 0.01 a fixed learning rate, Si(1) = Σ. (t)
d x((1)) 2t(2) di
2,(t) XY post
S((1) and di ,,
+ S,(1,
(13)
XY
pre
Overall, this search space comprised 6 tunable plasticity parameters per synapse type XY (X, Y E (E, I)): xxx = |axx, 5xy, Txy, Kxy, 'pre synapse types.
-XY 7XX) for total of 24 plasticity parameters across all four
Note that all weights in the network were capped at all times, in the [0, Wmax]] range, with Wmax 20, though rule quadruplets in (20] were considered unstable if more than 10% of the weights at any synapse type reached these extreme values.
A.1.3 Familiarity detection task
We considered a familiarity detection task that was similar to previous work 20 The network first re- ceived nonspecific background inputs for 1h th--rrnt phase-, followed by a 40s training phase during which four non-overlapping input patterns-the the familiar stimuli-wer active in alternation. When given stimulus was active, ss x10% of the input neurons to the excitatory population had ele- vated firing rates, while the others remained at background. After training, we reverted background inputs and regularly probed the network with familiar and novel stimuli for nour-post-trrinilee phase.
Given the input structure and connectivity described above, each stimulus, novel or familiar excited a different subset of recurrent neurons. We defined "engrams" for each stimulus pattern (novel or familiar) by probing the network before training started, and labeling the top 10% of excitatory and inhibitory neurons as part of the engram for the presented stimulus. In practice, since the stimuli were non overlapping, the engrams defined this way also had little overlap.
Note that each stimulus elicited different network response, the sense that each stimulus preferen- tially excited a different subset of recurrent neurons (this can be seen in fig. S2) and thus the stimulus identity could be decoded from the population vector of neuron activities. However, whether this stimulus has been encountered by the network in the past -its novelty or familiarity was unknown.
We chose simple decoding strategy for stimulus familiarity: the mean firing rate of the excitatory population (each excitatory neuron contributes equally to the decoding). In the paper that inspired this work [20). Student t-test was performed over several seeds/simulations/siinna over mean firing rates response to familiar or novel stimuli to determine whether novel stimuli elicited statistically different mean firing rates than familiar stimuli. Note that by design, all stimuli elicited statistically indistinguishable mean firing rates in static or naive plastic networks. Thus this task flagged a stimulus-specific change in network activity due synaptic plasticity.
A.2
Feedforward spiking network model
A.2.1 Network model
1000 Poisson neurons (800 excitatory and 200 inhibitory) projected onto a single output neuron, with the same neuron model and parameters in the previous section. The excitatory weights were fixed at Wee = 0.1, the inhibitory weights were plastic with the parameterization defined for the recurrent spiking case, and initialized at wie = 1.
14

---

Note that in fig. EF, we used different learning rule, not part of the plasticity parameterization du described above. For this rule, at = (1) + C with 3 0.4 and C = 0.00012 to obtain a target firing rate of 3Hz given the integration time-step of the simulation (0.lms).
A.2.2 Familiarity detection task
The task closely resembled the task in the recurrent case. During a pre-training phase of 1h, all input neurons fired at 10Hz. During the training phase that lasted 60s, 100 excitatory and 25 inhibitory increased their firing rates to 100Hz and 50Hz respectively (the "familiar" stimulus). After the training phase, the network was regularly probed its network response to the familiar stimulus and another, novel stimulus of the same structure than the familiar stimulus but with different neurons (no overlap).
A.3
Toy model I: Explicit parameterization
A.3.1
2D version
The model is a linear feedforward network with two inputs x: (10, projecting single output neuron y:
v() mo(t))zo()) wi (t)
(14)
Besides, Vt 2 O, y(t 2 0, (t) 2 0, x (t) 2 0: x = with ||-I] the L2 norm. Weights were plastic and unconstrained.
Initially, we considered a four-parameter set of Hebbian/non- plasticity rules inspired by the full spiking model (see mean-field section below for the relationship between spike-based and rate-based plasticity):
(w, v + (()) + 02)(1) + 032r(1)v(()) Öt
(15)
with Go 01. 0g and 0g four plasticity parameters, and 1) 0.01 fixed learning rate (omitted helow). From this full search space, we only considered rules that admitted a target output firing rate y" E R stable fixed point for all inputs considered here. The existence of y fixed point implied that for all inputs xi
0o 012, + B2y" Ogray" = 6, -02y" and 01 -0sy"
(16)
Thus the search space became two-dimensional:
awo
(y 8" (0u + 0120)
Øt
w Aw + B
(17)
Own
(v y² (00 +1xz)
Øl
(00 20)) (0 + 0
x (0. 0,20) x (0n 0,
and B = yy" 000 0120
with A
0o 0,
This system has only one zero eigenvalue: \ (x + 21) ) + 01 (x + vf). The system thus has neutral mode (Au 0), for the system to converge
point on the line attractor, we need
A1 < O. Because |x||| and rr. `1 2 0, xo + 21 E [1, /2] and z + x 1. As a result, we need Oo and 6, be below the lines On + 0, = and 0 /2 + 0 =
For the numerical results reported the paper, we simulated the system the A-B-A task with: y" 1, we (1, 0.27) W m Big L F. We the system until convergence each phase of the A-B-A task, in practice found that T 20000 epochs was sufficient.
We verified that the results of the parameter sweeps on 0g and 0, had similar trends for different
stimuli angles and initializations.
15

---

A.3.2 Extension to higher input dimensions
In the main text, we chose the smallest model possible (2D) for ease of visualization. However, the findings readily extend to higher dimensions (N > input neurons).
We consider Nin input neurons with activity æ projecting on single output neuron with weights w: W x. We choose æ to be unit norm with nonnegative entries. The plasticity rules are the same as in the 2DD case:
Ow:s
(t) (y(() - (6) + (1), i Ot
_Nin
(18)
This is an affine system of ODEs, which can be written in vector form Aw + b, with A = x a000 + 0,æ), and a Nn-ddimensional vector of ones.
For constant input x, this system is rank one, and the non-zero eigenvalue is À 0, E Zi 0, -i= x = fo Σm xy + 01 for unitary norm inputs. Note that this generalization of the derivation presented above.
Nin
Edge of stability: For the system to be stable, we need À O, which translate for unit-norm, nonnegative inputs to O + 01 < and 0g + 01 < O.
Defining metric to evaluate memory: As in the 2D case, we define Wbg. the steady state weights for input 2og at the start of the A-B-A task, and Wbgr are those for Dng at the end of the task. However, the 2D definition of RI (eq. 4 does not generalize readily to N-dimensions,
Wbgrstim- the intersection between the hyperplanes W "² Tbg y² and W main w" auim y is not unique (assuming these hyperplanes are not parallel). As proxy, we define RI l|wber Wog ||. which only evaluates how far the final state is from the initial one, and not whether the change is pushing towards the intersection not.
Overall, as can be seen in fig. 58 increasing the dimensionality of the toy models did not change qualitatively the findings reported in the main paper.
A.4 Toy model 2: Implicit parameterization
A.4.1
2D version
This toy model only described the fixed point reached by an implicitly-define class of learning rules operating in the same linear feedforward network as in previous section.
Specifically, we made two assumptions the learning rules:
Stabilization: V (æ, yo. wo) E 2xx1xx lim y(t, æ, Mo, wo) y²
(19)
lim w(t, x. yo: wo) = argmin|| l/w Distance minimization: V (xx yo, wo) E 2x1xxx too+o
wa
1|:
wEW
(20)
with ||.||x the norm induced by the Mahalanobis distance D:
v(W1, W2) E 22 Ds(WI.w2) = (w11w,) W2)
(21)
aa 6) 0). where a, b, and.c are the three plasticity parameters in this search space. To be well-defined distance, --1 needs to be positive semi-definite leading to the further constraint that a 2 0 and ac
with Σ-1
b22 2 O.
From these assumptions, derived the final network state w/ (w w/)) as a function of the initialization w' (wg, wf), æ² (input for which the network is initialized), and (fixed) input s!
Since the final state belongs to W t have w T x! = = w/ y" - We minimize the distance D:
16

---

11- D(w,
= + +bo y² + wgró - waaff
+ cxó
wffof - y²
xf² wo +2
x1 12
wo
(22)
2bnugoy (f-20faf 5*)
+ ||w1| +
(23)
x
g-1 ||x11|
Since
> O, the expression above has a single minimum:
21
w awjer bay (wfaf wfzz 1
v²) + -
(24)
We applied the result above twice, for each phase of the A-B-A task.
Relationship between explicit and implicit toy models: although don't tt't have general mapping from the implicit to the explicit rules in both toy models, some rules have the same steady states in both cases.
y)" (0, 0, 1) had identical fixed point to the rule minimizing the Fuclidean distance in the implicit model (a = 1, b = 0, = 1), see fig. SS6). The explicit form of this rule could also be seen as the gradient ceeccett update wrt the loss function
Notably, the Hebbian rule Sw -y की
C
(y-y")² L.. The;
ac
( y²)
dhg
A.4.2 Extension to higher input dimensions
This toy model has the same network architecture as above, but the number of input neurons does change the number of plasticity parameters, as the rules are this time parameterized by a distance metric of the Mahalanobis family (with covariance E E Na,.N, leaving us with Nn,(N. plasticity parameters.
2
Edge of stability: This corresponds to Σ losing its positive semi-definite property (i.e. at least one eigenvalue becomes 0).
A.5 Toy-model 3: Mean-field-inspine model
common method study spike-tining-dependenl plasticity is perform mean-field analysis 33 6 M.. which assumes a large network of uncorrelated neurons. Under these assumptions, the weight updates in the spiking network eq. become:
dw dw)(1) = m/|preexxx + Fppsttxxy + Trre ToreTpost (EXY TXY + +XXYXY dt
post
pre
(25)
with pre and post the firing rates of the pre- and post-synaptic neurons. This method allowed us to get a rate "equivalent" of each spike-timing dependent rule defined ineq. (T). We embedded the rate-equivalent rule quadruplets in 2-neuron linear recurrent network (2RNN) undergoing the
familiarity detection task (fig. ST)). The activities TE, Fı of the 2RNN, representing the excitatory and inhibitory spiking populations, followed:
(r + = Wee (1)r(()) -ne (1)r + xg(() ry(t +1) (.rrgg(()) wa (1),(()) xy(1)
(26)
The activities r and weights w were constrained to be positive at all times. The familiarity task was ported to this setting by providing background input to the network Tbg = (1, 1), followed by training period with stimulus 2atim (1.5. 1) before reverting to Bbg- Over 95% of the rule quadruplets that were stable in the recurrent spiking model elicited diverging weight or activity dynamics in the rate model, such as the rule quadruplet shown in fig. (fig. SJ Nevertheless, this 2RNN satisfyingly approximated some rules, particularly those evolving isolation, such as the rate-equivalent of iSTDP (fig. SIC..
17

---

A
00
W
1
x
W
01
10
X
ht
1
W
B
C
5
10
hg
0
0
2
50
0
0
0
4500
0
4500
t
t
Supplementary Figure S1: Mean-field-inspireed model. A: Linear 2-neuron recurrent network
(2RNN) and notations. B: 2RNN evolving with the rate-equivalent of the iSTDP rule during the familiarity task (see fig. 2A for spiking equivalent). Dashed lines denote the onset and offset of training. Top: network activities during the task. Bottom: evolution of the four recurrent weights. Colors match the cartoon in A. C: Same as B, but for the meta-learned rule quadruplet shown in fig.
Overall this suggested that the assumptions made to obtain the rate equivalent rules were not valid in the case of co-active rules. Indeed, the one rule for which the 2RNN model performed qualitatively similarly like the spiking model is iSTDP, which thus ensuring that the assumption of uncorrelated neuron activities holds. We thus moved to a more abstract setting to understand the memory by accident phenomenon.
shown decorrelate neuronal activities [1]],
A.6 Supplementary figures
18

---

A
Ebreak
05
1s
Cbreak 0s
1s
100 neurons
100 neurons
Ubreak 1h
1s
100 neurons
100 neurons
Supplementary Figure S2: Visualization of network activities for fig. 0 A,B: Raster plots during two test sessions of the familiarity detection task. Neurons are colored by which engram they belong to (see methods for definition of engrams, gray shows neurons not part of any engram).
A
C
D Non-Hebbian rule
E
Hebbian rule
0.5
X
W
01
W
10
hag
hig
0.0
X
h
W!
nHebb
Hebb
GD
B
A-B-A task
t
0
1.1
0.5
0.5
hegl
lh:-
-055
0.0
0
0.0
θ
(9) 0
90
5
Supplementary Figure S3: A linear RNN reproduces aspects of memory by accident A: Network and notations, activities are restricted to be positive. B: Example input activities in the A-B-A task. Each stimulus presentation chosen be long enough for any potential fixed point to be reached. C: Hebbian rule in the A-B-A task: Aur post
(hig he yh?re with heg fixed target (1). Top: Recurrent neurons activities, Bottom: 4 network weights. D: Same as C for non-Hebbian
rule: Am post x (h2g - h post E: Distance between the network weights at the end of the first or the second presentation of stimulus A: averaged over many simulations for the three learning rules tested. "GD" is online gradient descent on the mean squared error loss of the network activity compared the target activity heg- F: Network response profile for stimuli of various angles at different timepoints of the A-B A-B-A task.
19

---

Vanilla ISTDP
B
+
4 +
2x learning rate
+
+
Wider Hebblan window
15
15
15
Fexc
10
10
10
#5
-Tan
-Wer) -llwe)
Time since training (min)
Time since training (min)
Time since training (min)
Half-Hebbian
pre- pre-post only
post-pre only
15
10
15
10
10
-fam
10
8
(Weel
o(wi)
60
60
60
Time since training (min)
Time since training (min)
Time since training (min)
Supplementary Figure S4: Variations of iSTDP and memory by accident: Recurrent spiking network undergoing the familiarity detection task, for 6 variants of the iSTDP rule. All variants have
the same target excitatory rate 3Hz. A: TIF post 20ms, aIE = -0.12, BTg = 0, KIE = pre
TIE = 1. B: TIE
pre
TIE
post
20ms, aSE = -0.24, BIF = O, KIF = 71E = 2. C: TIE = TIE
post
40ms, 01E = 0.12 31g = DIE = "E = 0.5. D: TIE
pre = TIE
post 20mss OIE = -0.12 BIE
0.06, RIE 7IE 0.5. E: E post 20ms, aIE 00.12 31E О,K 0, = 1. F:
post 20mss aTE -0.12,3 0,KTE 1,7E O. TIE TIE
Texc -Tnov -frmm
0.5
WHE
60
Time since training (min)
Supplementary Figure S5: Feedforward spiking mode with E-to-E plasticity. Top: output neuron firing rate. Bottom: evolution of the 800 excitatory (plastic) weights. Weights from all excitatory input neurons are in red, the subset of weights belonging to the familiar stimulus are overlayed in purple. The means of the groups ("familiar" weights rest) in bold. The plasticity rule used
pre post 100ms, Ogg 1,3EE -0.8, KEE YEE -1
TEE
TIE
20

---

A
B
b=0.5
a=1.5
c=1.0
2
2
>1.
2
>1.
o
0 &
0
2
-2
-2 b
<-1.1
-2
<-1.1
C
D
3.5
start
c=1.0
>1.1
WXbg Vtg
W2
O
0
WX = Yig
2
1.0
<-1.1
1.0
0.0
3.5
Supplementary Figure S6: Additional analysis of the implicit feedforward toy model: A: Similar parameter sweeps as fig. 4B, but varying other parameter combinations. B: Same parameter sweep as in fig. 4B, but for an angle of 3 between the two inputs. C: Grey: dynamics of the 6 = 0, 01 -1 rule from the explicit parameterization in the A-B-A task. Black dots represent the fixed points the rule associated to = 1, b = O, c 0 in the implicit parameterization. D: Parameter sweep on the plasticity rules (varying a and b, fixed. But relaxing the assumption that Σ needs be positive semi semi-definite.
B
D 10
8 = 0.4; K
B 0.00 v. 0.5
W +
t ppsi
Touc Frov -Fam
Texc -Toww "Fam
-Fexc "Toov -Pam
10
(Hz)
pPoat 40rms
0
60
60
60
30 60
since training (min)
Time since training [min)
Time training [min)
Time since training (min)
Supplementary Figure S7: Additional analysis on testing predictions from toy models. A, B, C: Top: firing rate of the post-synaptic neuron during simulation associated to fig. BB& (red), well as the firing rate response to the novel and familiar stimuli. Bottom: inhibitory weights, weights from inhibitory input neurons aprt of the familiar stimulus are in purple, the rest is in teal. Averages of the two groups are in bold. Dashed lines indicate training onset and offset. D: Recurrent spiking network with variants of iSTDP (I-to-E plasticity only) in the familiarity task, with different Hebbian windows. Simulations were repeated 5 times, averages across seeds are shown in bold, dashed lines denote the last timestep which the population firing rate in response familiar stimuli was significantly different to novel responses (Student (-test, < 0.05).
21

---

Nín =
Nin 3
Nn 10
2
0.5
0.5
2
0.5
650
0
650
0
<00
0 2
-2
2
-2
-2
0
-2
00.5
0c
0g
8
10
10
10
2
T (llg scale)
(log scale)
(log scale)
10a
102
10c
65
<50
104
10a
10
-2
-2
-2
0
0g
8
00
Supplementary Figure S8: Extending the explicit toy model to higher dimensions. A: Top: Phase portrait of the relative improvement RI as a function of the values of 0g and 0,. Note that here, Rt is defined in appendix A.3.2] to extend to Nm Bottom: Same parameter sweep as C. but plotting the time to convergence т. These two plots are generated for Nu 2 (same main). B,C: Same as A, but for Nu 3 and Ni 10.
input dimensions. Grey denotes unstable rules.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show simulations and derivations in four models (recurrent and feedforward spiking networks, two toy feedforward linear models).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We included a dedicated paragraph in the discussion. We include and discuss negative results from some large scale numerical experiments in the last figure.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings,
22

---

model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For the three parts of this paper that rely on analytical results (mean-field analysis for I-to-E plasticity compute target firing rates), and the two toy models, included dedicated section in appendix for the full derivation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included a more detailed method section in Supplementary with the exact parameter values to run our models. The code to reproduce our simulations and analysis is also provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
23

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., .. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The dataset for meta-learnes rule quadruplets is publicly accessible from previous public work. The code to reproduce all simulations and analysis available on github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc/pbbli/ggi.dee
wwwpbbmisssooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
24

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Supplementary methods go into more details the exact experiments pre- sented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All statistical tests performed are described, and the code provided includes the analysis.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Paragraph in Supplementary material about the compute-requirements of this work.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: this paper is very fundamental analysis of the emergence of memory in the brain, and has no immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the models used in this study have no foreseeable potential for misuse.
Guidelines:
26

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All libraries upon which the simulations and analysis built on are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
a link to the code provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
27

---

Justification:
No human subjects are used in this study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were used in this study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy http:e/r/eeerppppecc///oferereep for what should or should not be described.
28


Reviews:
[{"review_id": "U2WHbX34IW", "reviewer": "Reviewer_wTCy", "summary": "The authors address a recent result whereby plasticity rules that ensure stability of firing rates also endow networks with memory of past stimuli. The main insight is that weights are degenerate, and hence for a given input, there is a set of weights that can cause firing rate to have a desired “output”. Because plasticity was designed for stability, it will lead the weights to an arbitrary point in this set. Switching an input (writing a memory) will change the set of allowed weights, and cause the weights to converge towards a point on this new set. In general, when switching back the weights are not expected to return to the original configuration, hence providing memory.\nThe authors demonstrate this concept in various simplified settings, and also show links between the timescale of emergent memories and the stability of the network.", "questions": "Results on gradient descent after zero loss (Blanc et al, COLT 2020; Li, Wang, Arora ICLR 2022) suggest that weights might drift on the allowed manifold over time, towards flat areas. Do you see evidence of this in your case?\nFigure 1C caption typo “after a 4h”\nWhat is the justification for freezing plasticity after the first stimuli in the familiarity task?\nIn Figure 2A, the weights are still evolving after training ends. Are weights frozen?\nFigure 2B. Only W_IE seems to strengthen the familiar engram in point 2. And for W_EE it’s the opposite. The description in the main text (line 102-104) seems inconsistent with the figure.\nRules with the best RI are close to instability (Fig 3,4). The other direction does not hold – there are “bad” rules close to instability. Do you know how this generalizes in higher dimensions?\nRelated to the previous question – did you check the edge of stability for a simple linear model in high dimensions? There it should be easier to compute.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThis is a simple and elegant explanation of a seemingly strange observation.\nThe toy models help gain insight into the phenomenon.\nThe writing is mostly easy to follow, but see some questions below.\n\nWeaknesses:\nThe toy models give some insights, but are limited to two dimensions. It would be useful to understand whether dimensionality changes anything qualitatively.", "was_revised": true, "final_justification": "I am keeping my strong support for acceptance - 5. \nI am open to discussions with other reviewers/AC on whether it should be a 6.", "rebuttal": "We thank the reviewer for their very encouraging appraisal of our work and for their suggestions. Please find below a detailed answer to all points raised.\n\n**Extending the toy model to higher dimensions**:  \nWe chose the smallest toy model possible for ease of analysis and visualization, but as pointed out by the reviewer, the findings readily extend to higher dimensions ($N_{input}>2$ input neurons), even though we did not emphasize it in the submission. Note that we could also increase the number of postsynaptic neurons $N_{output}$. However, because of the shallow feedforward architecture, this would effectively correspond to $N_{output}$ uncoupled networks receiving the same inputs. For the explicit parameterization (Fig.3), we could increase the dimensionality of the plasticity parameter space, for instance by adding higher order terms in the Taylor expansion ($pre^2$, $post^2$). We have not done so in the current paper as the rules from the spiking simulations (eq.1) naturally translated into the search space defined in eq.2 (see eqs.8, 24 and [1]), but this is definitely an interesting avenue for future work.\n\n*Description of the $N_{input}>2$ toy model, explicit parameterization*:   \nWe consider $N_{input}$ input neurons $\\mathbf{x}$ projecting on a single output neuron $y$ with weights $\\mathbf{w}$: $y = \\mathbf{w}^T \\mathbf{x}$. We choose $\\mathbf{x}$ to be of unit norm with nonnegative entries. The plasticity rules are the same as for the 2D case (eq.3): $\\frac{\\partial w_i(t)}{\\partial t}=(y(t)-y^*)(\\theta_0+\\theta_1 x_i(t))$.  \nThis is an affine system of ODEs, which can be written in vector form as $\\mathbf{\\dot w} = A\\mathbf{w} + \\mathbf{b}$ with $A = \\mathbf{x}(\\theta_0 \\mathbf{1} + \\theta_1 \\mathbf{x})^T$ and $\\mathbf{b}=-y^{\\ast}(\\theta_0 \\mathbf{1} + \\theta_1 \\mathbf{x})$, and $\\mathbf{1}$ a $N_{input}$-dimensional vector of ones.  \nFor a constant input $\\mathbf{x}$, this system is rank one, and the non-zero eigenvalue $\\lambda = \\theta_0 \\sum_{i<N_{input}}{x_i} + \\theta_1 \\sum_{i<N_{input}}{x_i^2}=\\theta_0 \\sum_{i<N_{input}}{x_i} + \\theta_1 $ for unitary norm inputs. This is a generalization of the derivation presented in Supplementary (eq.17, L807-828). \n\nEdge of stability: For the system to be stable, we need $\\lambda<0$, which translates for unit-norm, entrywise nonnegative inputs to $\\theta_0 \\sqrt{N_{input}} + \\theta_1 <0$ and $\\theta_0 + \\theta_1 <0$. Note that this is a generalization of the condition given L823 in Supplementary. We added this analysis in Supplementary, as well as a supplementary figure reproducing Fig 3D&E for 10 input neurons.\n\n*$N_{input}>2$ toy model, implicit parameterization*:  \nThis toy model has the same network architecture as above, but the number of input neurons does change the number of plasticity parameters, as the rules are this time parameterized by a distance metric of the Mahalanobis family (with covariance $\\Sigma \\in \\mathbb{R}^{N_{input} \\times N_{input}}$, see sections 4.2 in the main paper and A.4 in Supplementary). This leaves us with $\\frac{N_{input}(N_{input}+1)}{2}$ plasticity parameters. We added in supplementary the 3D and 10D equivalent of Fig.4B, more precisely some 2D slices of these cases for visualization.\n\nEdge of stability: This corresponds to $\\Sigma$ losing its positive semi-definite property (i.e. at least one eigenvalue becomes 0). In 2D, as pointed out by the reviewer, though the \"best\" rules (with strong memories) are close to instability, there are also “bad” rules (rules with weak memories) close to instability (Fig.4B). We verified that this property scales to higher dimensions (new supplementary figure), and clarified in the revised paper that this is a necessary though not sufficient condition for maximal memory. This may explain the initially inconclusive verification at scale of this prediction in Fig.5D.\n\nOverall, increasing the dimensionality of the toy models did not change qualitatively the findings reported in the paper.\n\n**Weight drift once on the solution manifold, similar to Blanc et al, COLT 2020; Li, Wang, Arora ICLR 2022**:  \nWe see evidence of weight drift for the spiking networks (Fig.2 and 5, perhaps most clearly visible in Fig.2D). However, it is unclear whether it is purely noise-driven perturbations or if there is a general direction to the drift. As far as we know, this is an open question in the field. We would guess that there is a direction and that the reviewer's hypothesis is true, as we notice most weights ultimately revert back to homogenous values, but we are not sure to what extent this is rule-dependent. Whether this corresponds to a flatter region of the loss, we don’t know for sure, but this is definitely a very interesting avenue for future research.\n\n**What is the justification for freezing plasticity after the first stimuli in the familiarity task?**  \nThis is a clarity mistake on our end which we have addressed in the revised version: weights are always plastic unless we are probing the familiar or the novel stimulus during a test session. The results don’t change too much if we let weights be plastic also during the test sessions, as the rules are overall slow. We chose to freeze the weights during the test session to be able to precisely monitor the forgetting of the familiar stimulus (memory timescale), for instance in Fig.2 and 5.\n\n**Minor comments and typos**:  \nWe adapted the description of the co-active spiking simulation (L102-104) to describe more precisely the effect on each synapse type.\n\n**Reference**  \n[1] Confavreux, Harrington, Kania, Ramesh, Krouglova, Bozelos, Macke, Saxe, Goncalves, and Vogels, Memory by a thousand rules: Automated discovery of multi-type plasticity rules reveals variety degeneracy at the heart of learning, bioRxiv, 2025"}, {"review_id": "sx3IXkxe2w", "reviewer": "Reviewer_ZVuV", "summary": "In a preprint (cited) the group of Tim Vogels in Austria has reported a large simulation stuy where many combinations of STDP rules (E to E, E to I, I to I, I to E) yield network dynamics that are stable (in the sense that the  mean rate of excitatory neurons stays at, say 3Hz)  despite ongoing plasticity.\n\nIn the NeurIPS contribution, the authors develop a toy model to better understand this phenomenon. The toy model is a linear feedforward network with STDP rules translated into a rate model with Hebbian  term (preXpost)  as well as a pre-only and a post-only term.\n\nThe toy model works like this. Since the network is linear and (by construction) stable,  the postsynaptic neuron has a fixed point of the firing rate (say 3 Hz)  which translates into a linear manifold of weights compatible with that fixed point.\nIf a constant stimulus is presented, the weights move to a new linear manifold (consistent with the the same fixed point of the firing rate of 3Hz, but in presence of the stimulus). Once the stimulation is removed, the weights move back to the original manifold but end up at a different position. Therefore the weight vector contains information about the past stimulus that is not visible in the firing rate pattern.", "questions": "1. The authors illustrate the main idea for a toy model in two dimensions, but the math of linear manifolds could be and should be generalized to the case of N>2 weights, where some belong to the stimulated inputs part and others not. There should still be one Eigenvalue with a large negative real part (related to the stability of the fixed point), but within the group of stimulated neurons more eigenvalues will show up. What can we learn from these?\n\n2. Kempter et al. 1999 have shown that in STDP rule where the Hebbian parameter gamma is nonzero, then even a purely Hebbian STDP rule induces indirectly a pre-only term (in the equivalent rate model) since every presynaptic spike influences the firing probability of the postsynaptic neuron proportionally to the weight: spike arrival at an excitatory/inhibitory synapse increase/decreases the postsynaptic rate which gives a correction term of order 1/N that should be included in the rate description that is used as rate equivalent of the original STDP model. This should be discussed.\n\n\n3. Unfortunately,  we only learn at the very end (Fig 5B) that the strongest familiarization effect is the one with post-only (beta>0) in I to E connections. Moreover Fig S6 shows large heterogeneity of the weights. Both are interesting observation but deserve an explanation.\n\n4. In what sense is the half-Hebbian rule FigS4D unstable? The firing rates show small fluctations but are stable over an hour. Weights transiently oscillate, but that does not indicate instability.\n\n5. The analysis assumes that the fixed point is reached during the stimulation time. But obviously, if we turn down the learning rate a lot, this assumption is no longer correct. Also, given the rate stabilization dynamics, it should be straightforward to calculate the rate of convergence (i.e., time constant of stabiity eigenvalue) as a function of eta, and I would expect the authors to do this for both regimes: convergence time of the rate dynamics during stimulus presentation and without a stimulus.\nMy main concern here is that the statement that small eta helps to keep memory is both trivial and wrong. Trivial because weights will move slower. Wrong, since nothing will happen for eta too small. (Main prediction 2).\n\n6. Main prediction 1: Is this related to a general result of dynamics and bifurcation theory? It is well known that at the transition from stable to unstable parameter regimes the (real part of) one of the Eigenvalues of the fixed point goes through zero.", "limitations": "yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n\nThe paper is clearly written, starts with an interesting observation, and turns their observations into  a toy model. I like the analysis of the toy model. Even though the methodology is not novel (see Kempter et al. 1999, PRE, Kempter et al. 2001, Neural Computation) the application of the mathematical framework to the case of a familiarity memory after a short stimulion is new and interesting.\n\n- Weaknesses\n\nThe abstract (as well as line 265 of the main text) seems to hint at a result that increasing from one plasticity rule to four plasticity rules would increase the dimension of the manifold of parameter combinations consistent with the stable firing rate fixed point. Unfortunately, this is not shown (or not developped in the main text), even though a linear toy model for, say a recurrent network with 3 neurons (2 excitatory, 1 inhibitory neurons) driven by  3 further input neurons  could be analyzed in the same way as the toy model. I see steps in this direction in the SIMULATIONS of the toy model in supplementary, but it should be possible to perform mathematical ANALYSIS for these cases. The analysis should get give you a spectrum of Eigenvalues for convergence to the fixed points in both stimulated and unstimulated case - and these eigenvalues will tell you a lot. One can then translate each of the 3 input neurons into a group of 10 input neurons with the same stability and still do the analysis.\n\nFor the toy model it remains unclear to the reader whether it is meant to be valid for E-E connections as well as EI/IE connections or not. It seems that Eq. (3) is generic and applicable to all cases. However, a lot of the later analysis and simulations seems to be focused on variations of iSTDP (which is the case that has already been studied by Vogels et al. cited ref 19). The results could have a bigger impact if they are shown to  more generally applicable.\n\nTwo of the three predictions look kind of obvious for somebody who is familiar with linear stability analysis.\n\nOverall, my impression is that the paper contains cool ideas that could be explored further to make a convincing package of results. As it stands, the paper does not exploit the full potential of its ideas. May be good strategy would be to work on the topic a bit longer and then submit to another upcoming venue.", "was_revised": true, "final_justification": "I appreciate the good discussion with the authors. I feel that my concerns  regarding the limitations continue to be justified and I do not change my overall rating.", "rebuttal": "We thank the reviewer for their thorough and overall positive review of our manuscript. We are glad that they appreciated the toy models developed and the general direction of the project. Below are detailed answers to the points raised in the review.\n\n**1. Extending the model to higher dimensions**:  \nWe chose the smallest model possible for ease of visualization, but as pointed out by the reviewer, we do not mention higher input dimensions. We propose to add two supplementary figures on the toy models with N>2 input neurons and mention this generalization of the toy model in the main text. In our hands, the N>2 model does not change the predictions reported in the paper. Overall, we observe stronger memories as there are more degrees of freedom (weights) for the same (scalar) postsynaptic neuron activity that is brought to a setpoint by a single plasticity rule. This suggests that the larger the network, the stronger the memory-by-accident phenomenon (currently discussed L265, but we added a pointer to the N>2 model there). For more details on the N>2 case and for lack of space here, we refer the reviewer to our rebuttal to reviewer wTCy.\n\n**2. Citing and discussing Kempter et al. 1999 and 2001**:  \nWe apologize for this oversight, and now cite these papers. Concerning the indirect pre-only term in their derivation, we chose a simpler route for the rate equivalent of the spike-based rules which neglects the effect of all correlations between pre- and postsynaptic neurons. This assumption has been made in other studies about plasticity in spiking networks [1-3], and provides a reasonable prediction of the network activity [1]. We added a discussion point about this.  \nWe are currently working on additional corrections to this spike-to-rate derivation beyond those performed in Kempter et al. 1999 that may explain the stability through co-activity phenomena and other mismatches between theory and simulations noted in several previous studies. We believe this is most appropriate for a separate, dedicated technical paper, as the paper at hand has already been flagged as \"confusing\" by other reviewers due to the many models and rules considered.\n\n**3. “Unfortunately, we only learn at the very end (Fig 5B) that the strongest familiarization effect is the one with post-only in I to E connections. Moreover Fig S6 shows large heterogeneity of the weights. Both are interesting observations but deserve an explanation”**.  \nThis hypothesis-testing of spike-based rules resulted from our analysis of the toy model, hence its appearance only at the end of the paper. These non-zero pre-only rules are not stable in isolation in recurrent spiking networks [3]. Thus the idea of testing all the rules in isolation in the feedforward spiking model was not clear to us initially, but now appears somewhat trivial following the development of the memory by accident framework.  \nWe thank the reviewer for the interesting observation that weight heterogeneity appears to depend on the rule used in the feedforward networks. We have run additional simulations to make sure this observation is robust (it is!) and have added it in the revised paper. However, these findings do not translate easily to the recurrent spiking network with co-active rules. As most of these IE rules with non-zero pre-only terms are unstable in isolation, we remain cautious in linking this observation to memory-by-accident. Overall, we agree that at present we spend too little time analysing and discussing the results of Fig.5, which we correct in the revised version of the paper.\n\n**4. “In what sense is the half-Hebbian rule FigS4D unstable?\"**  \nThe simulation in FigS4D is unstable because the weights slowly ramp up despite being exposed to constant inputs. They eventually reach extreme values if simulated for longer than 1h. We chose to keep the plotting consistent across all spiking simulations, but in that case we agree that it makes the weight instability tricky to see. We fixed this in the revised version.\n\n**5. Concerns on main prediction 2: learning rate and memory**  \nWe agree that our prediction on learning rates in the toy model (L186) is valid for small but non-zero learning rates and requires waiting long enough for the steady state to be reached. We have clarified this in the revised version. We also added a theory vs simulations time-to-convergence plot (Fig 3E, which is currently only simulations as rightfully pointed out). Moreover, we have added a supplementary control figure about time-to-convergence for stimulus vs background inputs. However, because of the definition of “stimulus” and “background” inputs (two vectors of unit norm, which one is “stimulus” and “background” is arbitrary), the results are similar. We have rephrased the verification of this prediction by checking that indeed when the learning rate is too small, learning stops, to show in the end more of a U-shaped curve (memory timescale as a function of learning rate value).\n\nWhile we agree with the reviewer that the current prediction in the toy model is quite simple as it stands (see below for a proposed extension), we don’t think this is the case for the spiking models, which is why we kept this prediction in the paper. For instance, without the weight degeneracy (or a rule that cannot exploit it such as the rule in fig 5C), the timescales of learning and forgetting are identical, and thus the learning rate doesn’t change the duration of the memory much. However, what we see in practice in the spiking models are two timescales for forgetting (L119-123). The weight degeneracy allows the network to land in a state where the postsynaptic activity is close to the target and the weights have the imprint of the familiar stimuli. The memory then seems to be forgotten by noise-driven fluctuations (fig 2C&D). These noise-driven fluctuations also depend on the learning rate of the rule, but can be of much longer timescale than learning driven by the activity mismatch of the postsynaptic neuron compared to the activity setpoint. We have added a discussion point of this interesting effect that will need to be studied more in future work.\n\nHowever, we agree that the prediction made by the toy model does not reflect this more interesting phenomenon, which is why we propose a simple extension to the toy model to be able to formulate the same prediction in a more grounded way: the addition of a diffusive process (noise). This allows to recast the relative improvement RI (a distance between starting and final weights) as a duration for the diffusive process to erase the memory, thus capturing the two-timescale-process seen in spiking networks. We thank the reviewer for pointing this out and believe that prediction 2 is now more legitimate and interesting.\n\n**6. Concerns on main prediction 1: memory strength and distance to instability**  \nAs pointed out by reviewer wTCy, not all the rules bordering instability are good at making memories (for instance only the top ones, b>0, in fig 4B). This means the relationship between RI and the eigenvalues of the system (neuron activities) is not direct. As such we don’t think this prediction is a trivial byproduct from bifurcation theory. But this is a very interesting idea that we look forward to exploring further in future work.\n\n**Additional points**:  \n\n- *Absence of a toy model for co-activity of plasticity rules*:  \nWe agree that a toy model for co-activity is another standout from [3]. We have attempted —and so far mostly failed at— producing a satisfying recurrent toy model with co-active rules, as documented in Fig S1. The recurrent model we show in Supplementary with the mean-field version of the spiking rules (eq.1) does not reproduce any behaviors observed in the spiking networks. This is mainly due to stability mismatches: a vast majority co-active rule quadruplets that were stable in recurrent spiking models in [3] lead to runaway weight dynamics in their mean-field (rate) version for over 95% of rules (Fig.S1, L861), thus preventing us from drawing any conclusions on their memory abilities. We have developed a feedforward linear model with 2 co-active rules. This model shows that memory strength (RI) generally increases compared to the single-rule case. However, this model does not capture any of the stability phenomena (rules unstable in isolation can be stable through co-activity). Overall, we fully agree that this is a great avenue for future work. However, since memory by accident doesn’t require co-activity to function and is a key phenomenon in its own right, we believe it is worthy of being told to the community on its own. Besides, a toy model able to capture co-activity likely requires us to delve deeper into spike timing effects (see point above about Kempter et al.). We have thus chosen to keep this out of the current paper, which was already flagged by other reviewers as already having too many rules and models in its current state.\n\n- *Overfocus on IE plasticity and iSTDP when testing predictions from the toy model:*  \nWe used iSTDP as it is convenient to simulate (robust and stable) and commonly used in the field for network stabilization, but typically not for learning itself (e.g. Litwin-Kumar and Doiron 2014 in which it is used in tandem with EE plasticity). But it is true that our predictions are not specific to IE synapses, and the paper was unnecessarily focused on iSTDP, thank you for rightfully pointing this out, we have addressed this in the revised version. Ensuring stability with the other rules in isolation is a little more tricky, though we can test the predictions in the feedforward spiking model. We have added a panel in Fig.5 testing the predictions with EE (or EI/II) plasticity in the feedforward spiking case, and find that predictions hold for all synapse types.\n\n**References**  \n[1] Vogels, Sprekeler et al. Science 2011  \n[2] Confavreux et al. NeurIPS 2020  \n[3] Confavreux et al. bioRxiv 2025"}, {"review_id": "zwyEsobep7", "reviewer": "Reviewer_jDbn", "summary": "In this study, authors address a specific type of memory which consists in distinguishing familiar and novel stimuli. Results suggest that training recurrent E-I networks with unsupervised plasticity rules that enforce stability of the network, such simple form of memory emerges a byproduct of learning. Authors showcase this phenomenon using different architectures of networks that, while not designed to have memory, they have them nonetheless when subject to unsupervised learning enforcing stability. A variety of network architectures is discussed, going from larger recurrent spiking networks, networks with inhibitory plasticity, and analytically tractable toy models which are small feedforward networks. Results suggest an interesting relationship between memory time scale and learning rate, or memory time scale and stability.", "questions": "1) Why would stability be required for memory formation? If we have a recurrent network that invariably goes into one single attractor state and its firing rate diverges there, this seems like a very strong memory in an unstable network. Can authors comment on that?\n\n2) How exactly is degeneracy related to stability and memory formation?\n\n3) What do coefficients \\theta_j in Eq. 2 stand for?\n\n4) What is the meaning of the result on Fig 5D, bottom?\n\n5) Why using a feedforward toy model and not a recurrently connected one?", "limitations": "Limitations are discussed with some clarity, however, there are limitations beyond those discussed in the Limitation section. One limitation that is not described is that the models only address similarity / novelty detection.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "_Strengths_:\n1) The research question is well motivated in the introduction, and the discussion is appealing to read.  \n2) Several ideas presented in the paper are really interesting and worth further investigation.\n\n_Weaknesses_:\n1) To some degree this paper still reads as a work in progress and requires careful clarification of main concepts as well as a clarification and straightforward description of methods. Most main concepts rely on a previous (unpublished and not peer-reviewed) work, and they need to be clarified in revision.\n2) The paper uses several model architectures (large recurrent spiking networks, feedforward networks, minimal toy models) and it remains to some degree unclear and unproven how well results transfer from one architecture to another. \n\n3) The definition of the problem has to be carefully revised. This namely includes specifying and emphasising that the narrative is about one specific and type of memory (novelty / familiarity).  \n\n4) Previous related results could be better recapitulated (in the introduction) and current paper's results could be better discussed with respect to relevant related literature (in discussion). The writing should reflect what is demonstrated by the current results, what is strongly suggested by the current results and what is merely suggested. Overgeneralising the findings that are mere observations from specific cases should not be presented as proven.\n \n5) The degeneracy of connectivity weights is mentioned several times (including in the abstract) as something that warrants memory formation, but how degeneracy is related to memory formation is not clearly explained nor demonstrated. The information the reader gets is that the weights of the toy model are assumed to be degenerate. This seems insufficient evidence for systematically linking degeneracy of weights and memory formation.", "was_revised": true, "final_justification": "A number of clarifications were provided during the discussion period and some of the discussed issues were addressed in revision, which allowed me to increase the overall recommendation. In spite of these improvements, the take-home message of the paper is still confusing, as the presentation of results is confounding proofs with observations. For the sake of scientific integrity, it is essential that claims presented in the paper are proven or convincingly demonstrated, which, in my opinion, is not yet achieved in the present version of the paper.", "rebuttal": "We thank the reviewer for the detailed feedback. We would first like to clear a potential misunderstanding when reading the reviewer’s summary: “Authors suggest that enforcing network stability gives networks that are also capable of memory, without explicitly training networks with any specific plasticity rule.” In this paper, though we consider a variety of local unsupervised plasticity rules, the network models always evolve with one instance of these rules (1 plasticity rule for the the toy models and feedforward spiking model, and 4 co-active rules—one per synapse type— in the recurrent spiking network model). Since the rules are unsupervised, we cannot write that the networks were “trained” on the familiarity task, but the plastic networks were shown stimuli as part of the familiarity detection task, such that their weights could keep an imprint (or not) of these specific stimuli.\n\n**Definitions and clarity, “the paper is overall confusing”**:  \nWe acknowledge that the various models and plasticity rules used in the study, used to convey the generality of our findings, may have obscured its message. We have rewritten parts of the introduction and results to define more clearly what we mean by “memory” and “stability” in this paper.\n\n- *Definition of memory*: Following previous work (Confavreux et al. 2025 [1]), we consider simple forms of memory: familiarity and novelty detection, which ask whether a network can remember whether an external stimulus has already been encountered in the past. In the spiking models, the familiarity detection task was taken from [1]. We consider a network with weights plastic following a given plasticity rule or quadruplet of rules. We present the network once with a few stimuli (the will-be familiar stimuli). After the stimulus presentation is over, the network continues to evolve with its own internal dynamics while receiving background inputs. We wait for various durations ranging from 1s to 4h before doing a “test session” that aims to probe whether the network remembers the familiar stimulus or not. That means freezing the weights, and showing the network the familiar stimuli and novel stimuli in succession to obtain the mean firing rate of excitatory neurons during familiar stimulus presentation $r_{fam}$ and likewise for novel stimulus presentation ($r_{nov}$). We then define memory as the relative preference of the network for familiar or novel, $\\Delta r_{mem} = 2(r_{nov} − r_{fam})/(r_{nov} + r_{fam})$. We hope this clarifies the current definition of memory that was L67-76 in the manuscript. We would be keen to hear any other suggestions the review may have as to what else we should measure to complement this definition. For the toy models, we adapted this definition due to the simplification of the network and of the task (described L159-175).  \n*Why is this definition of memory relevant?* For the spiking network models, we refer the reviewer to the methods of [1] which builds on previous studies and provides numerous controls that this task does provide a simple and efficient definition of memory. First, the familiar stimuli are always compared to novel stimuli, such that the metric for memory reflects how different the network response is for familiar vs novel. This metric is shown to correctly flag networks without ongoing plasticity as not having memorized the familiar stimuli. Moreover, most rule quadruplets obtained in [1] forget after various timescales, which suggest that the metric is correctly flagging an effect specific to the familiar stimulus that is ultimately forgotten by ongoing network activity and plasticity.\n\n- *Definition of stability*: In the recurrent spiking model, the definition of stability comes again from the paper that inspired this work [1]: a network is stable if its activity and weight dynamics remain \"biologically plausible\" for a wide range of inputs, despite the ongoing plasticity in the recurrent synapses. I.e., activities remain within plausible ranges (non-silent network, not too-high firing rates, asynchronous irregular activity), as well as slowly changing weights that don’t reach extreme values.\nIn the toy models, we somewhat narrow this description (L154-158) as both theoretical analysis (eq. 8) and simulations [1] show that the plasticity rules from eq.1 can be seen as establishing a setpoint on the network activity. We thus only consider rules with the property of bringing the neuronal activity to a setpoint $y^{*}$ and call them “rules that enforce network stability”.\n\n**1. Why would stability be required for memory formation?**     \nWe don’t see stability as a requirement for memory formation, but as a general observation in non-epileptic subjects and thus as a general requirement for models of the brain circuits and synaptic plasticity. The main conclusion of this paper is that stability enforced by simple local unsupervised rules tends to generate simple forms of memory as a byproduct. In the example provided by the reviewer, the divergence of the firing rate would preclude that network from being considered stable. Also, because the attractor state is not stimulus-specific, that network would not be flagged as memorizing the familiar stimulus.\n\n**2. How exactly is degeneracy related to stability and memory formation?**   \nWe observe that degeneracy is a consequence of stability (defined as bringing neuron activities to a setpoint): there are more weights than neurons in the network, so many weight combinations that will bring neurons to a setpoint value (L104-108, L140-142). We also observe that this degeneracy of weight matrices that can stabilize a network means that which weight matrix is currently implemented in the network can reflect previous inputs to the network which is a potential form of memory. In practice we verify this is the case for most local unsupervised rules we test in this study, at least transiently. These observations in the spiking models (fig 2B&C) become mathematically tractable and easy to visualize in the toy model (fig 3C for instance).\n\n**3. What do coefficients $\\theta_j$ in Eq. 2 stand for?**   \nThe four thetas are four scalar parameters that determine the plasticity rule in the first toy model (L156). They determine to what extent weight updates depend neither on the pre- or the postsynaptic activity $\\theta_0$, the presynaptic activity only $\\theta_1$, postsynaptic activity only $\\theta_2$ or both $\\theta_3$ (i.e. a Hebbian term).\n\n**4. What is the meaning of the result on Fig 5D, bottom?**   \nIn Fig. 5D we take 20 “rule quadruplets” from [1], i.e. 20 vectors of plasticity parameters that each determine four plasticity rules of type eq.1. We know from [1] that these 20 quadruplets all create recurrent spiking networks with stable activity and weight dynamics (which we verify again in simulations). 10 of these quadruplets create memories with short memory lifetimes, $\\Delta r_{mem} \\neq 0$ only for a few seconds, whereas the other 10 have longer memory lifetimes (>4h). We ask how close these 20 quadruplets are to instability, i.e. what is the nearest quadruplet of rules that creates an unstable network. To do so numerically, we perturb the plasticity parameters for each quadruplet across random dimensions, with perturbations of increasing magnitudes (from 1 to 5). We report which perturbations generated stable networks and which ones did not, as a function of the magnitude of the perturbation, and plot for each quadruplet what was the closest unstable perturbation. However, the results do not show a significant difference between the quadruplets with short memories and those with long memories (L222-230).\n\n**5. Why using a feedforward toy model and not a recurrently connected one?**   \nIn this paper, we want to show a very general finding about memory and stability, and thus a result applicable both in feedforward and recurrent settings. We designed the toy model as the simplest possible model able to exhibit memory by accident: two weights, one postsynaptic neuron. We have also made a few recurrent toy models, mainly to study how the co-activity of several plasticity rules in the same network affects memory by accident (see fig S1). We refer the reviewer to our rebuttal for review zVuV, which discusses co-activity. In short, the problem is that many rules create unstable networks when embedded in isolation in the recurrent models, and require other rules to be stable. This complicates testing what each rule is good at. The feedforward models, both rate and spiking, do not have that problem and were thus a great way to decouple the stability problems and the memory capabilities of each rule. We mention this problem only briefly currently (L235-236), we have changed the text to elaborate on this more in parts 3.1 and 3.2, as well as in the discussion.\n\n**Reference**  \n[1] Confavreux, Harrington, Kania, Ramesh, Krouglova, Bozelos, Macke, Saxe, Goncalves, and Vogels, Memory by a thousand rules: Automated discovery of multi-type plasticity rules reveals variety\ndegeneracy at the heart of learning, bioRxiv, 2025"}, {"review_id": "qof2PVcQMg", "reviewer": "Reviewer_cFwp", "summary": "This paper formalizes and analyzes the intriguing observation that stabilization of neural networks leads to memory, even in the absence of explicit memory mechanisms. This 'memory-by-accident' phenomenon is widespread and arises from the degeneracy of stabilizing weight matrices (i.e., the network state is history-dependent).", "questions": "The paper doesn't define stability or how the networks are optimized for stability.\n\nRelatedly, one point I'm confused about is the relationship between stability optimization and the familiarity detection task. Were the networks trained to do the task, or optimize stability, or both?\n\nThe paper mentions a firing rate set point, but I didn't see where that parameter shows up in the model (sorry if I missed it).\n\np. 6: \"the model predicted that the only rules that do not elicit memory are exclusively non-Hebbian\". It's not clear to me that such a strong statement is warranted. I believe this is true for the toy model, but I don't think the authors have shown this in a more general form. There could exist non-Hebbian rules that elicit memory under other modeling assumptions.\n\nI'd like to understand better what this means for neuroscience. Are there testable predictions? Does it change how we think about memory systems in the brain?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper explores an interesting and provocative idea.\n- Overall well-written.\n- Mathematically rigorous.\n- Opens up a new area of research.\n\nWeaknesses:\n- A few aspects need further explanation.\n- Not entirely clear what the implications are for neuroscience.", "was_revised": true, "final_justification": "The authors have done a good job addressing my questions. I'm going to maintain my score of 5, because I think that this accurately reflects my current evaluation of the paper.", "rebuttal": "We thank the reviewer for their positive review and interesting suggestions, please find below detailed answers to all the points raised above.\n\n**1. “The paper doesn't define stability or how the networks are optimized for stability.”**   \nThis is indeed an oversight on our end, also raised by reviewer jDbn, which we fix in the revised version of the paper. \nIn the recurrent spiking model, the definition of stability comes from the paper that inspired this work [1]: a network is stable if its activity and weight dynamics are biologically plausible for a wide range of inputs. I.e., activities remain within plausible ranges (non-silent network, not too-high firing rates, asynchronous irregular activity), as well as slowly changing weights that don’t reach extreme values. In the toy models, we somewhat narrow this description (L154-158) as both theoretical analysis (eqs.8 & 24) and simulations [1] show that the plasticity rules from eq.1 can be seen as establishing a setpoint on the network activity. We thus only consider rules with the property of bringing the neuronal activity to a setpoint and call them “rules that enforce network stability”. In this paper, saying that “networks are optimized for stability” is an abuse of notation on our end, we mean that the unsupervised plasticity rules embedded in the network can be seen as stabilizing network dynamics, i.e. bring the neuron activities to a firing rate setpoint. We have fixed and clarified these points in the revised paper.\n\n**2. Were the networks trained to do the task, or optimize stability, or both?**  \nSince we are using unsupervised plasticity rules, the networks are not explicitly trained for anything, they receive inputs as part of the familiarity task and react to these inputs depending on the plasticity rule(s) embedded in the network. However, analysis performed on such rules here and in previous work [2, 3] shows these unsupervised rules can be understood as an approximation of gradient descent on a stability objective (forcing the network activity to a setpoint, see Supp L849). We have clarified this point in the revised version of the paper.\n\n**3. The paper mentions a firing rate set point, but I didn't see where that parameter shows up in the model**:   \nWe acknowledge that the several similar yet slightly different plasticity parameterizations introduced in the paper can be somewhat confusing. We have attempted to clarify and contrast these parameterizations in the revised manuscript. \nFor the plasticity rules in the spiking network models (eq.1), the firing rate setpoint can be computed for each synapse type independently (eqs.8 & 24, L233) and depends on the plasticity parameters and the network activity. For the toy model with the explicit parameterization, the firing rate set point $ y^{\\ast} $ is imposed by construction of the plasticity search space, we only consider rules of the shape eq.2 that admit $ y^{\\ast} $ as a fixed point for all inputs of unit norm. For the implicit parameterization, the setpoint $ y^{\\ast} $ is also imposed by construction, and we define the rules by the distance metric that they will minimize when bringing network activity to that setpoint.\n\n**4. p. 6: \"the model predicted that the only rules that do not elicit memory are exclusively non-Hebbian. It's not clear to me that such a strong statement is warranted. I believe this is true for the toy model, but I don't think the authors have shown this in a more general form.\"**   \nThe reviewer rightfully points out that we don’t consider all possible rules, and as such we cannot prove that statement conclusively. Hence, we have rephrased this sentence to dial down and clarify this point, and added a point in discussion highlighting the limitation that there are many Hebbian/non-Hebbian rules that exist and that were not considered in our analysis. Nevertheless, we think that the evidence presented here across a variety of rule types/network models is sufficient to propose the theory of memory by accident despite this limitation.\n\n**5. “I'd like to understand better what this means for neuroscience. Are there testable predictions? Does it change how we think about memory systems in the brain?”**  \nBesides the points raised in the discussion which are not necessarily directly testable predictions, we believe the impact of this work is two-fold:\n- 1/ Reinterpreting old data: Numerous evidence exists for Hebbian/anti-Hebbian plasticity in the brain [4,5]. However, previous theoretical work suggested that such simple rules are not enough to understand the memory abilities of brains [6,7]. Here, we propose that the evidence required to understand basic memory abilities in the brain may have been available for nearly 30 years. And we speculate in discussion that more complex memory abilities could be compositions using these simple building blocks. As such, we believe that memory-by-accident has interesting implications for neuroscience.  \n- 2/ Testable predictions: our toy models formulated a few predictions that we tested *in silico* in the final figure, that link properties of the plasticity rule (synaptic mechanisms) and corresponding memory features (behaviour/cognitive features). For *in vivo* predictions, we could speculate that brain regions endowed with long-term storage might possess different learning rules (the ones suited for long memories, for instance with pre-only terms) than regions usually associated with faster forgetting such as potentially working memory, though all these rules may belong to the same Hebbian/non-Hebbian family of rules like the ones studied in this paper.\n\n**References**  \n[1] Confavreux et al. biorXiv 2025  \n[2] Vogels, Sprekeler et al. Science 2011  \n[3] Kempter et al. PRE 1999  \n[4] Markram et al. Science 1997  \n[5] Bi and Poo, Journal of neuroscience 1998   \n[6] Morrison, Diesmann and Gerstner Biological cybernetics 2008   \n[7] Zenke, Hennequin and Gerstner PLoS Comp Bio 2013"}]

Author rebuttal:

Meta-review: While one reviewer was uncertain about acceptance, most reviewers agreed that the finding was interesting and worthy of publication, provided the paper was duly clarified.

Personally, I would also note that the proposed "inevitability activity-silent memory" has clear implications for neuroscience, considering the debate about how much working memory and short-term memory rely on such activity silent, synaptic mechanisms (https://pubmed.ncbi.nlm.nih.gov/26051384/).

After discussion with reviewers, I recommend acceptance, provided that the necessary clarifications and explanations are actually incorporated in the final paper.
Decision: Accept (poster)