Discovering plasticity rules that organize and maintain neural circuits

Authors: david  bell
alison  duffy
adrienne  fairhall
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:45:37

Abstract: Intrinsic dynamics within the brain can accelerate learning by providing a prior scaffolding for dynamics aligned with task objectives. Such intrinsic dynamics would ideally self-organize and self-sustain in the face of biological noise including synaptic turnover and cell death. An example of such dynamics is the formation of sequences, a ubiquitous motif in neural activity. The sequence-generating circuit in zebra finch HVC provides a reliable timing scaffold for motor output in song and demonstrates a remarkable capacity for unsupervised recovery following perturbation. Inspired by HVC, we seek a local plasticity rule capable of organizing and maintaining sequence-generating dynamics despite continual network perturbations. We adopt a meta-learning approach introduced by Confavreux et al, which parameterizes a learning rule using basis functions constructed from pre- and postsynaptic activity and synapse size, with tunable time constants. Candidate rules are simulated within initially random networks, and their fitness is evaluated according to a loss function that measures the fidelity with which the resulting dynamics encode time. We use this approach to introduce biological noise, forcing meta-learning to find robust solutions. We first show that, in the absence of perturbations, meta-learning identifies a temporally asymmetric generalization of Oja's rule that reliably organizes sparse sequential activity. When synaptic turnover is introduced, the learned rule incorporates a form of homeostasis, better maintaining robust sequential dynamics relative to other previously proposed rules. Additionally, inspired by recent findings demonstrating that the strength of projections from inhibitory interneurons in HVC also dynamically responds to perturbations, we explore the role of inhibitory plasticity in sequence-generating circuits. We find that learned plasticity adjusts both excitation and inhibition in response to manipulations, outperforming rules applied only to excitatory connections. We demonstrate how plasticity acting on both excitatory and inhibitory synapses can better shape excitatory cell dynamics to scaffold timing representations.

Main paper: 
 introduction and related work: 
 How computational structures are organized and maintained within the brain is a central question within neuroscience. While feedback is clearly essential for learning, self-organization of neural circuits can unfold without feedback, e.g. during development. Brains have evolved specific cell types with nonrandom spatial organization, plasticity rules, and connectivity that likely introduce a Figure 1 : Meta-learning approach to discovering plasticity rules that organize sequences (a) In zebra finch song learning, a neural representation of time (left) in HVC simplifies the sequential motor learning task of producing the correct spectral output. (b) Putative network structure of zebra finch HVC: a feed-forward, excitatory network with recurrent inhibition (left). HVC excitatory neurons (red) fire sparsely in time while interneurons (blue) fire tonically (right). (c) Strategy for learning plasticity underlying sequence organization: candidate plasticity rules, parameterized by a set of coefficients and time constants, are simulated. A loss function is evaluated on the resulting dynamics, and new candidate rules are generated. (d) Test procedure for representation of time. Networks are activated 400 times (blue bars). From the final 50 activations, 6 are chosen to train a decoder and 6 to test the representation by decoding time from neural activity. (e) The robustness of discovered plasticity rules can be tested by introducing synaptic turnover, in which synapses are stochastically removed and added, into simulations.strong set of inductive biases on the information processing they perform. How might organization of useful circuit dynamics be established and maintained throughout life without the need for feedback? Recent work suggests self-organized computations, once established, can accelerate learning and improve performance when experience is limited: Nicola and Clopath [1] demonstrated that a stable high dimensional time signal could improve a network's performance on sequential motor tasks (Fig. 1a ). Najarro and Risi [2] learned Hebbian plasticity that orchestrated spontaneous walking behavior in quadruped agents; similar work has shown architectural priors increase the sample efficiency and generalization of RL approaches to locomotion [3, 4] . Additionally, in RL settings, supplying agents with a time input permits them to adopt time-dependent policies [5] . The ability of computational primitives, such as timing representations, to self-organize is challenged by the shifting structure of neural circuits. Synaptic loss, synaptogenesis, cell death, and neurogenesis pose challenges for all learning algorithms, but particularly for self-organization which must be based solely on local information rather than global task performance.Here, we aim to find plasticity rules that self-organize and maintain one useful computational primitive: sparse, sequential activity. Such activity is widely seen in many areas of the brain including hippocampus [6] , cortex [7] , and basal ganglia [8] . In the songbird zebra finch, area HVC (used as a proper noun), a cortical-like region, displays sequential activity representing time [9] , reducing the problem of motor learning to driving the correct motor neuron at the correct moment [10] . Extensive literature has explored how such sequence-generating circuits could emerge in the absence of feedback [11] [12] [13] [14] [15] [16] [17] , but has largely focused on either how these structures organize or how they self-maintain, using guessed plasticity rules, and neglecting the effects of ongoing synaptic noise. Further, previous work on sequence organization within HVC has focused on plasticity between excitatory (E) neurons. Recent experimental findings show unsupervised recovery of HVC dynamics is accompanied by changes in both E→E and also inhibitory-to-excitatory (I→E) synaptic strength [18] .Here, instead of imposing a guessed rule, we ask which self-supervised plasticity rules can organize and maintain sequential dynamics within a network. We employ meta-learning, a supervised method to learn learning rules [19] [20] [21] [22] [23] , in order to discover rules that self-organize a sequence. Our approach stems from a rich history of learning local plasticity rules, including rules that extract representations from data [24] , enhance artificial agent performance on familiarity and navigation tasks [2, 25] , and explore biologically-plausible replacements or complements to backpropagation [19, 20, 26] . In this study, we parameterize the space of plasticity rules with a basis of activity-and synapse size-dependent terms. The set of coefficients weighting these terms and associated time constants are adjusted to minimize a loss function. Inspired by the HVC context, we pose the loss in terms of the accuracy with which the time since an initial network input can be decoded from the circuit dynamics. We first consider E→E rules alone, and then add I→E and E→I plasticity. We then introduce perturbations to the circuit and investigate which learned plasticity rules promote circuit stability. We find that meta-learned rules for self-organized sequence generation and maintenance contain distinct forms of spike-timing dependent plasticity, homeostasis, and network bounds, which outperform previously proposed sequence-organizing rules in the presence of noise and that plasticity on reciprocal connectivity to inhibitory neurons confers additional stability on the network dynamics. Our main contribution is the exploration of unsupervised and unrewarded plasticity via meta-learning that organizes and maintains a specific and biologically relevant computational motif, a sequence.In the zebra finch, nucleus HVC contains excitatory neurons that fire sparsely (typically in one burst of spikes) during song and are purportedly arranged in a feed-forward structure (Fig. 1b ) [27] . A subset of these cells, known as HVC (RA) neurons, project to downstream nucleus RA (robust nucleus of the archistriatum), which in turn projects to vocal neurons of the syrinx and to the brainstem, which regulates respiration [9] . HVC receives excitatory projections from nucleus Uva, which controls the onset of song syllables [28] and provides input for the duration of song [29] . HVC (RA) neurons inhibit each other disynaptically via a population of inhibitory interneurons [30] . Remarkably, singing behavior can persist when the nucleus is transected [31] , demonstrating its resilience.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The primary claims of the paper are addressed computationally in Section 2, results, and are expanded upon in the supplement.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
learning biologically plausible plasticity on e→e synapses that organizes a sequence-generating circuit: 
 We first learned a plasticity rule on E→E synapses that organizes a randomly connected network into a sequence-generating circuit in the absence of any perturbation. We initially constrained plasticity to E→E synapses to compare with previously proposed rules, which have largely only considered E→E plasticity. We did not include rules that imposed hard bounds on the size of individual synapses or on the collective strength of all synapses onto a neuron, as we aimed to learn plasticity rules that could permit flexible rescaling of connections in response to perturbations, as has been observed experimentally [18] .We use a network of 25 E and 8 I threshold-linear neurons. Each neuron fired according tox j (t) = [V j (t) -b] + , where V j (t) evolves via τ m Vj (t) = -V j (t) + i w ij x i (t).Here, w ij is the weight of the synapse i → j, τ m is the membrane time constant, and b is the bias. Initial connectivity (Fig. 2d ) was random, but contained no I→I connectivity as is the case in HVC [30, 32] (see Supp. Sec. 1 for all network model details).We adopt a meta-learning approach pioneered by Bengio et al. [19] and extended by Confavreux et al [33] . We parameterize a set of plasticity rules with coefficients c k and time constants τ k , such that individual synapses, w ij , evolve according toEQUATIONwhere x i and x j are pre-and postsynaptic activities, respectively, Θ is the Heaviside function, and F k is the k th term in the plasticity rule (see Fig. 3a or Supp. Sec. 2 for all terms). The basis includes terms that filter pre-and postsynaptic activities with decaying exponentials (denoted e.g. xi ) as these terms convey information about the durations of activations and relative ordering, not just their instantaneous rates. Synapses evolving under Eq. 1 were bounded so that they obeyed Dale's law.We next define a loss function that evaluates the quality of the sequential dynamics organized under a chosen {c k , τ k } and attempt to minimize it using an evolutionary strategy, Covariance Matrix Adaptation (CMA-ES) [34] . We use CMA-ES to sample from the space of possible {c k , τ k } and evaluate the loss at each point by simulating 10 randomly initialized networks under the given rule and evaluating the resultant dynamics at the end of the simulation. Each simulation is divided into 400 activations of 110 ms. At t = 10 ms of each activation, a single fixed neuron is driven by a strong kick of excitation [17] . Following this, all other neurons in the network receive Poisson distributed input for a period of 65 ms. A fraction of this Poisson input is held fixed from trial to trial, mirroring input to HVC from the nucleus Uva, which likely does not provide a fully stochastic signal to the downstream area, [28] . The total loss for a given rule is simply the sum of losses across each of the 10 networks.To learn sequences, we define a loss function based on three principles: (1) elapsed time since initial network activation should be readily decoded from network activity, (2) total network activity should be sparse, and (3) total synaptic change in the network should be minimized. The latter two principles impose the assumption that effective plasticity does not require excess network activity or synaptic change to stabilize network function. Our loss isEQUATIONwhereEQUATIONP(c, τ ) penalizes the all synaptic changes due to each component of F. This L 1 -like penalty on F penalizes each term not by the size of the term's coefficient, c k , but by the quantity of synaptic change it evokes. While penalizing the magnitude of c k is standard [20, 33] , we take this approach to compare different terms on a common scale, as each component of F has differing dependence on x i , x j , and w ij . λ a and λ s are positive constants weighting the activity and synaptic change penalties. To determine the decoder loss L dec (c, τ ), the activity of networks was sampled at 500 time points of six activations of the network. From these, a linear decoder was constructed and used to decode the activity at 200 time points of six subsequent activations. Note that since the initial kick of excitation was time-locked to beginning of each activation, constructing a decoder to read out time elapsed since the beginning of the activation was equivalent to decoding time elapsed since the initial excitatory kick was presented.Meta-learning reliably found plasticity rules that organized random E synaptic connectivity into feed-forward structures that generated sequences when activated (Fig. 2d-e ). The organized structures were not grouped into links, as in a synfire chain, but were better described by a kernel in which the strength of a synapse between two cells depended on the lag between their mean firing times (Fig. 2d , right). The set of rules discovered by meta-learning are not sparse in the space of {c k , τ k } (Fig. 3a ). To test whether learned plasticity rules truly required all terms with nonzero coefficients, we compared 'dropout' variants of the discovered rule, in which one coefficient within {c k } was set to zero, to its unaltered form. We computed the loss of these variants on test networks to determine if a term's absence impacted the loss (Fig. 3 )b). Computing the change in the median loss of networks organized by the learned solution and a dropout variant across 14 discovered rules revealed reliable trends in the importance of various terms (Fig. 3c ). We refit these terms progressively in order of the impact on loss across all training runs and found a sharp elbow at 3 terms: a temporally asymmetric Hebbian learning term, xi x j (blue pentagon in Fig. 3 ), its complement x i xj (orange square), and a term second order in the postsynaptic activity multiplied by the synapse size, w ij xj x j (purple triangle). The first term was consistently learned with a positive coefficient while the latter two were always almost negative (Fig. 3a ), rendering the effective learning ruleEQUATIONwhere all {c i } are non-negative and time constants are different for each term. The 3 most important terms can be interpreted as a temporally asymmetric generalization of Oja's rule in that the Hebbian learning term, x i x j , is replaced by the first two terms in Eq. 4, which depend on the relative timing of x i and x j . The time constant of the third term was on average short (∼1 ms), making this term akin to w ij x 2 j , the normalizing term of Oja's rule.We next asked whether ongoing disruptions to network structure alter which plasticity rules are metalearned. To explore this, we introduced synaptic turnover to the simulation phase of the meta-learning loop. Synaptic turnover is a stochastic process by which existing synapses disappear and new, small synapses emerge (Fig. 1e ). Prior to each network activation, all connections were updated according toEQUATIONwhere x ST ∼ U [0, 1], p ST is the probability of single synapse turnover per activation, and ϵ is a small positive (negative) constant if the presynaptic cell is excitatory (inhibitory). Since plasticity is unable to act on connections of size 0 (see Eq. 1), synaptic turnover determines the set of synapses available to the plasticity rule. To ensure learned rules were robust to a spectrum of rates of synaptic turnover, only half the networks used to evaluate the batch loss underwent this process.We found that meta-learned rules were able to organize persistent representations of time despite synaptic turnover, with performance near that of rules learned on unperturbed networks. Our termsensitivity analysis (Fig 4a ) showed that solutions again heavily depended on temporally asymmetric Hebbian learning, i.e. xi x j (light blue hexagon), and the bound on postsynaptic activity, w ij xj x j (purple triangle); however, we frequently found dependence on two additional terms: one that constantly strengthened all synapses (dark blue pentagon) and an activity bound independent of synapse size (orange square). Refitting the plasticity rule in order of impact on loss demonstrated that these 4 terms recapitulated most of the success of the learned solutions. The effective rule may be written asEQUATIONwhere again all {c i } are non-negative. Synapses for which postsynaptic activity, x j , remains chronically small will be potentiated by c 0 ; however, adding this potentiating term also necessitates an activity bound that does not scale with synapse size, such as the term with prefactor c 2 in Eq. 6. To understand this impact of this term, consider a synapse between two neurons whose typical firing times are far apart, i.e. xi x j is nearly zero, the fixed point under this rule isEQUATIONif ⟨x j x j ⟩ > 0, where ⟨•⟩ denotes the time average. Thus, a large enough choice of c 2 prevents every synapse in the network from growing, enforcing sparsity and decreasing the risk of a neuron changing its firing time upon loss of its original inputs. We additionally note that Eq. 6 does not contain -x i xj , which appeared in the reduced rule learned in the absence of synaptic turnover (Eq. 4). This may be because the roles of -x i xj and -x j x j are partially redundant: both terms can suppress synapses that run counter to the sequential dynamics in the network. When constant potentiation of all synapses occurs, the term in xj x j preferable as it offsets constant potentiation of all synapses. In the unperturbed context, where constant synaptic growth is unnecessary, the term in xj x j is problematic in that it can set all afferent synapses to a driven neuron to zero (whereas -w ij xj x j cannot). Thus, the term in x i xj becomes preferable in the unperturbed context.Do these discovered learning rules more robustly encode timing than previously proposed rules when the circuit is disrupted with biologically relevant noise? We hypothesized this would be true given that discovered rules do not impose hard bounds on the size of single synapses, total synaptic strength onto a neuron, nor number of synapses, as other models of sequence formation have [11] [12] [13] 17] . The absence of these constraints permits compensatory rescaling of synapses in response to disruptions. We compared meta-learned rules trained with and without synaptic turnover to a previously proposed sequence learning rule that used multiplicative asymmetric Hebbian learning, a single synapse bound, and bounds on the total strength of synapses onto and out of individual neurons [11] . Each rule was applied to 100 test networks for 400 activations without disruption. Following this, a decoder was constructed to read out time from the neural activity, and then synapses in the networks were turned over during an additional 150 activations, after which the loss was evaluated using the constructed decoder (Fig. 4c ). We compared the best versions of each rule when synapses were turned over (with probability p ST = 0.00072) at each activation, equivalent to a 90% probability of individual synapse survival during the disruption period (Fig. 4d , light purple points), and when they were not turned over (Fig. 4d , grey points; see Supp. Sec. 4). We used the Kruskal-Wallis H test to test for equality of medians. We found that meta-learned rules trained with and without synaptic turnover outperformed the rule based based on rigid synapse constraints (p = 2.5 x 10 -8 , Cohen's d = -0.45 and p = 7.2 x 10 -7 , Cohen's d = -0.44, respectively), while in the absence of perturbation, medians were not distinct after 4-fold Bonferroni correction (p = 0.016, Cohen's d = -0.29 and p = 0.029, Cohen's d = -0.30, respectively). When we studied the connectivity structure of networks organized by meta-learned rules, we found that the discovered rule generated denser feedforward connectivity in comparison to other plasticity rules with alternative forms of Hebbian learning and heterosynaptic competition (see Supp. Sec. 5).Meta-learning allows the exploration of multiple plasticity rules operating on distinct sets of synapses within the same circuit, as might arise if there are multiple cell types [35, 36] . In particular, I→E plasticity has been the focus of much recent work [37] [38] [39] [40] [41] [42] . The interaction of many plasticity rules is challenging to analyze theoretically, but meta-learning allows exploration of these interactions [43, 44] . Recent evidence suggests there may be multiple forms of plasticity within sequencegenerating circuits: Wang et al. [18] increased the intrinsic excitability of in vivo HVC (RA) neurons and found that the strength of both E→I and I→E connections could dynamically shift in response. Targeted cells received increased total inhibitory synaptic strength and decreased excitatory strength. To understand the role of this additional plasticity, we meta-learned plasticity on all sets of synapses within the circuit (Fig. 5a ) both with and without synaptic turnover. Specifically, we attempted to learn three independent plasticity rules that operated on three distinct groups of synapses (E→E, E→I, and I→E; see Supplementary Section 6).Meta-learning uncovered triplets of plasticity rules that successfully organized initially random networks into sequence generators (Fig. 5b ). When trained with turnover in E→E synapses, solutions that acted upon E→I and I→E synapses in addition to E→E outperformed solutions that only acted upon E→E connections, particularly when the rate of synaptic turnover was high (Fig. 5c ), suggesting this additional plasticity played an important role in maintaining network dynamics through perturbation.To investigate how rules acting on all synapses generated improved time representations, we repeated the dropout analysis. We found that E→E plasticity within these learned triples was largely similar to the rules previously learned on E→E synapses alone (Eqs. 4 and 6): solutions were consistently sensitive to the removal of xi x j and w ij xj x j , which appeared consistently with positive and negative coefficients, respectively. Further, dependence on these terms persisted when we trained networks with turnover on E→E synapses or I→E synapses (Supp. Fig. 6 ). As expected, we also found that solutions depended heavily upon terms that acted upon E→I and I→E synapses, but this plasticity was more difficult to interpret due to increased trial to trial variability in the discovered rules. We found, however, that the E→I plasticity rule consistently depended on the second order presynaptic term xi x i , which always appeared with a positive coefficient, suggesting that E cells project to inhibitory counterparts with a strength that increases with the E cell level of activity. An implication of dependence on this term is that the strength of an E neuron's recurrent inhibition, defined as|W rec i | = | k w E→I i,k w I→E k,i |, where i is the index of the E cell and k indexes the I cells to which it projects, might depend on its level of activity. Thus, ablation of excitatory inputs to an E cell might cause its recurrent, disynaptic inhibition to lower in a manner that homeostatically restores its firing.As the role of E→I and I→E plasticity was not completely clarified by our perturbations so far, we next investigated how this plasticity adjusted synapses coupled to an individual E cell when its typical input was manipulated. Noting the dependence of the E→I plasticity rule on xi x i , we hypothesized that a targeted neuron's recurrent inhibition and excitatory afferents might be adjusted in concert to restore its typical firing pattern. We used discovered learning rules to organize sequences and then performed three varieties of in silico manipulations of individual E cells within these networks. In the first, we scaled down the excitatory afferents to the targeted E cell by 50% (Fig. 6a, diagram ). In the second, we scaled up the same connections by 50% (Fig. 6b ). In the last, we increased the rate of the targeted cell's input Poisson process by a factor of 10 (Fig. 6c ). This final manipulation mirrored the viral insertion of NaChBac to HVC RA neurons in Wang et al., which causes these cells to become hyper-excitable [18, 45, 46] . Wang et al. found that manipulated cells recruited additional inhibition and weakened excitatory afferents.We found that scaling down a targeted E cell's excitatory afferents resulted in a rescaling of those excitatory weights and an accompanying decrease in |W rec i | (Fig. 6a ). We further found that these synaptic changes restore the initial firing pattern of the E cell: in Fig 6d , we plot the initial responses of targeted E cells across 20 self-organized sequences in the space of duration and peak amplitude of response (black points). Scaling down the excitatory afferents initially causes both the peak amplitude and duration of response to decrease (Fig. 6d ; light purple points), but these responses largely recover after ∼25 activations of the network (Fig. 6d ; blue points). When we instead strengthened E afferents to the targeted cell, we found these connections weakened and recurrent inhibition strengthened in compensatory fashion (Fig. 6b ). In these networks, we found that targeted E cell responses that were initially lengthened with increased peak amplitude (Fig. 6e ; light purple points) were reduced back to their pre-perturbation values of duration and amplitude (Fig. 6e ; black points show initial responses; blue points, responses after 25 activations). Finally, rendering the targeted E cell hyper-excitable caused it to scale down its excitatory afferents and increase its recurrent inhibition in a manner that restored its typical firing pattern. In summary, plasticity rules on excitatory afferents and recurrent inhibition operate in tandem to maintain the firing pattern of the neuron (see Supp. Sec. 7 for additional details).How might modulation of recurrent inhibition contribute differently to the modulation of excitation in preserving network dynamics? We hypothesized that distinct plasticity on different sets of synapses might confer a robust representation of time if these rules governed distinct aspects of the desired network function. For instance, within our HVC-like circuit, peak firing of E neurons might be controlled by E→E plasticity while total activity might be controlled by I→E plasticity. Since excitatory and inhibitory inputs to E cells differ in their timescales (E inputs are transient while I inputs are relatively tonic), we reasoned plasticity on both sets of synapses should increase the control of the postsynaptic response. While it is already known that multiple plasticity mechanisms can sharpen the responses of neurons to stimuli [37] , prior work has not addressed whether similar plasticity might be leveraged to preserve the timing of such responses, which is crucial to timing representations. To explore this, we constructed a simplified model of a single neuron responding to a broad range of excitatory inputs, characterized by varying peak amplitudes and durations, and a tonic inhibitory input. We compared the responses of the neuron when both the excitatory and inhibitory input synapses (Fig. 6h ) evolved under plasticity rules to responses produced when only the excitatory synapse evolved under a plasticity rule (Fig. 6g ) and found the two rule model was able to better constrain the duration, peak amplitude, and timing of the postsynaptic response the E neuron (Fig. 6i-j ; see Supp. Sec. 8 for full description of reduced model).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the findings and methods are discussed extensively in the Discussion and Limitations section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
discussion and limitations: 
 Meta-learning plasticity rules via stochastic optimization is a promising technique, but suffers a number of limitations. One, the optimization process becomes expensive as the size of the rule basis, the number of neurons in the network, and amount of simulation time required grows. Further, CMA-ES may require many epochs to converge on good solutions. Training E→E plasticity (plasticity on all synapses) across 10 networks in batch typically required 24 (72) hours of compute on 30 Cascade Lake or Ice Lake Intel CPU cores to yield reasonable solutions. Two, meta-learning tends to generate different solutions based on the seed; due to the expensive nature of each trial, we did not carry out enough trials to claim full knowledge of the solution space. Three, the plasticity rules learned were quite dense in our choice of basis, limiting interpretation, and we ultimately employed perturbations to better understand the critical terms. Four, the choice of basis limits the space of discoverable rules; for instance, we did not include feedback-modulated plasticity in this study. Five, the initial connectivity of the circuit likely has a strong bearing on the sort of plasticity that successfully can leverage it [47, 48] .Prior to any experience, intrinsic, self-organized dynamics within the brain can serve as powerful priors that can accelerate and shape successful, feedback driven learning. In this work, we study how one such computational primitive could emerge by adapting a meta-learning procedure to learn the learning rules that self-organize and maintain robust representations of time in neural dynamics. Metalearning discovers a temporally asymmetric (STDP-like) generalization of Oja's rule that organizes and maintains sparse, sequential activity out of initially random connectivity, which outperforms other models of sequence generation in the presence of synaptic turnover by permitting flexible rescaling of inputs to restore dynamics. Additionally we found that plasticity rules learned on all sets of synapses outperform plasticity rules applied only to E connections. Through a toy model, we show how plasticity on all synapses could confer extra timing stability if the plasticity in distinct sets of synapses act on different moments of an E neuron's activity.In this work, we have developed a paradigm to understand how computational primitives might self-organize within neural circuits and selected sequences as our test example. Future work could study the emergence of other canonical forms of neural dynamics that have been widely identified in brain activity and serve as fundamental components of computation, such as line attractors or limit cycles, or how plasticity rules generating such components interact with rules requiring feedback.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Code referenced in supplementary section, Code Availability.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper clearly outlines all methods. See section: Results, and also supplement.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Section 2, results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?Answer: [Yes]Justification: The computational resources used are detailed in the Discussion section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: The NeurIPS code of ethics is fully abided by in this work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Societal impacts are expounded upon in the discussion section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Code, data, and models used in the study belong to the authors or are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This paper proposes a procedure for discovering the synaptic plasticity rules by which a neural network can learn sequences. There were some questions as to the novelty and generalizability of this work; however, there was broad agreement that this work represents an interesting contribution to the field of theoretical neuroscience. Therefore, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a conceptually simple set-up to learn plasticity rules that result in neural networks that perform a specific task. The set-up of the paper is elegant and made up of simple but effective elements, such as linear-threshold neurons and evolutionary optimisation. The authors obtain some intriguing results, e.g. that their method discovers plasticity rules well-known in experimental neuroscience.
strength_weakness: strength_weakness: One of the main limitations of the paper is that this very promising and flexible set-up is only applied to one very idiosyncratic task of elapsed time decoding. This is a very specific task, and not one that brains do spontaneously (see next paragraph), so although it yields interesting results it seems a bit lacking. It would be tremendously useful to see the results in a broad range of tasks, to see e.g. under what conditions other common plasticity rules (Hebbian, anti-Hebbian, perhaps some dopamine-like reward-modulated plasticity) emerge.

The motivation of the system set-up is somewhat dubious. The authors point out the importance of "intrinsic, self-organised dynamics" but then include as the primary component of their loss function the performance of an external decoder that predicts simulation time. Neither the decoder nor the ground truth time values are in any way intrinsic to neural activity, so I would argue this is not representative of self-organisation.

On a separate note, the paper sorely lacks any demonstration of the "downstream" effects of these plasticity rules. For example, the author's motivation is that "self-organized computations provide scaffolds for solving difficult tasks". However, the authors don't actually show that the networks evolved with their discovered plasticity rules can actually solve any difficult task.

The paper gets really hard to follow towards the end, particularly Sections 2.3.2 and 2.3.3. The intervention to increase or decrease E afferents in Sec. 2.3.2, as well as the analysis metrics in Fig. 6a-c aren't very clear. It also surprising to see Sec. 2.3.3, the final results subsection of the paper, introducing a whole new model and a different plasticity rule, which are only partially related to the model and plasticity rules studied elsewhere. Overall, I could not understand well the set-up or results in either of these subsections.

The details of the meta-learning procedure should be explained more thoroughly in Sec. 2.1.1. Figures are quite busy and in general hard to parse.
questions: questions: The p-values in Sec. 2.2.1 are somewhat meaningless, because one can always run more simulations to decrease the p-value. I would recommend quantifying effect size (with e.g. Cohen's $d$) instead.

Please define important terms at first use (e.g. HVC).

What is $\mathcal{P}(\textbf{C})$ in L. 103?

References 4, 5, 11 and 28 are misformatted.
limitations: limitations: The "Discussion and limitations" section is very short and should be expanded.
paper_summary: paper_summary: This paper proposes a method for discovering plasticity rules in spiking neural networks to achieve sequence generation using both excitatory and inhibitory dynamics. Rules are parametrized with basis function. The biological approach in the model involves also considerations homeostasis and robustness to perturbation. The HVC region of the zebra finch was used as inspiration. The study shows interesting finding, including a self-discovered generalisation of the Oja rule, robustness to perturbation and enhanced stability, including homeostatic mechanisms.
strength_weakness: strength_weakness: - Unclear novelty: there have been many studies that evolved plasticity rules using some form of evolutionary computation and biological plausible rules. In fact, evolution of Hebbian-like rules has been a popular area of research for decades. While this study advances many previous studies, I feel that the novelty and contributions in this area are not well highlighted with respect to existing studies. 
- It is unclear to me how the findings can be used more broadly. I agree that it is interesting the CMA-ES and meta-learning can optimize plasticity rules to have interesting biological properties, but how can this finding be used? This is the most important concern I have: what impact can this paper and its discoveries have in the field? 
- Figures' clarity: there are a lot of assumption on the meaning of symbols, colours, etc, in the figure. E.g., Fig 1c, unless one is very familiar with the zebra fish, I don't think these graphs very useful. What is the big blue dot on top of the red ones? what do the colour in the cell-index vs. time mean? what does the graph mean? 
- The manuscript describes synaptic turnover starting from the abstract, but a search of the term to find out what that is and how it is implemented did not provide satisfactory results. 
- There may be an issue with the submission as the Supplementary information is not included, resulted also in broken references (e.g. line 197, page 7)
questions: questions: - How many fitness evaluation through CMA-ES were performed? How much time did one evaluation take? How did the fitness evolve over time? 
- Following up from a point in the weakness. How can these findings be used? Is there a recommendation that we should use evolution to implement and tune artificial spiking neural networks?
limitations: limitations: The paper addresses the limitations, in particular in relation to computational demands.
paper_summary: paper_summary: This work applies a meta-learning procedure for plasticity rule discovery to a neural network model of sequence generation. Plasticity rules are constructed from parameterized basis functions, and the parameters are found through evolutionary search based on a fitness function quantifying how accurately the plasticity rules train the network to represent elapsed time within episodic rollouts. The work analyzes the plasticity rules discovered in a variety of conditions: with noise, with synaptic turnover, and with plasticity enabled on excitatory and/or inhibitory neurons.
strength_weakness: strength_weakness: **Wiring vs plasticity**: It seems like an assumption of this work is that the sequence generating structure of HVC neural circuits emerges largely due to activity-dependent plasticity in a network with initially random connectivity. What is the role of activity-independent wiring that takes place during development? And how much of an effect does more structured initialization have on the discovered plasticity rules?

**HVC predictions**: As a main contribution of this work is improving a model of HVC, what experimental predictions will result from this computational analysis?

**Figures improvements**: The figures in general have pretty small text, inconsistency in formatting, and subpanels that are cramped. Fig 1 especially has a lot going on, uses a variety of fonts, and adopts a colorblind-unfriendly palette. Fig 2c, I didn't see why the 3 trials were needed; could it just show one and put other trials into the Supplementary? Fig 4d, use 45deg or 90deg xlabel angles instead?

**Text improvements** (minor): (1) The neuron model notation on Line 77 took me a while to parse and seems non-standard according to colleagues; ultimately, it is a simple exponential decay and such neural dynamics are usually expressed in dv/dt notation. (2) The spacing on Page 11 is messed up due to the type of latex linebreak used.
questions: questions: See above.
limitations: limitations: Yes, the authors adequately address the limitations in the Discussion section.
paper_summary: paper_summary: The authors investigate the development and persistence of intrinsic connectivity structures within a
neural region without external input. They propose that a local plasticity rule can lead to self-organising connectivity motifs, creating inductive biases beneficial for subsequent information processing. The study focuses on sparse and sequential dynamics, as observed in the HVC region of zebra finches. The paper explores which plasticity rules can organise and maintain these sequential dynamics using meta-learning.The space of plasticity rules is parameterised using terms based on activations and synapse size.
The coefficients and their time constants are adjusted to minimise the loss, which comprises three
components: encouraging sparsity in activation, synaptic change, and improving the accuracy of time decoding. Initially, excitatory-excitatory rules are considered, followed by inhibitory-excitatory and excitatory-inhibitory rules. The study evaluates the efficiency of these rules under synaptic turnover and compares them to established Hebbian plasticity rules. Evolutionary algorithms identified a rule resembling a temporally generalised Oja’s rule, which
outperformed a multiplicative asymmetric Hebbian rule in the context of synaptic turnover.
strength_weakness: strength_weakness: The manuscript occasionally feels unfinished and rushed, particularly in section 2.3.1.
The references to Figure 3 (blue circles, green stars?) and the lack of references to Figure 4
in the text were often frustrating. Additionally, I was keen to understand why the second term
estimated without noise lost its significance when noise was introduced, but the authors did
not address this point.
Furthermore, it was somewhat anticlimactic that the authors did not explicitly explore the
interaction of the estimated plasticity rule with subsequent supervised learning, as this was
strongly hinted at in the introduction. For instance, are networks with the estimated rule better
at reproducing songs than models with a traditional Hebbian rule?
Lastly, I found it odd that the decoding of time was not reported throughout the paper. Since
the reduction of loss can be attributed to three factors, this omission is noteworthy.
questions: questions: - Why does the second term, which was estimated without noise, lose its importance when noise
is introduced? The authors did not address this point.
- Why did the authors not explicitly explore the interaction between the estimated plasticity rule and
a subsequent supervised learning procedure, despite strongly hinting at this in the introduction?
- Are networks using the estimated rule more effective at reproducing songs than those employing a
traditional Hebbian rule?
- Why was the decoding of time not reported throughout the paper, given that the reduction of loss
could be influenced by three different factors?
- Could you clarify why the estimated rule is considered a temporally generalised Oja’s rule?
limitations: limitations: The authors were aware of the limitations of the methods used in the paper and addressed
them accordingly.
paper_summary: paper_summary: This paper introduces meta-learned plasticity rules for sequence generating neural circuits. While existing works demonstrated that the dynamics in HVC is generated by both excitatory and inhibitory synaptic updates, their approaches are based on a guessed rule. Motivated from this, the authors experimentally demonstrate that synapses with meta-learned plasticity rules introduce enhanced stability on the network dynamics under noisy environments, which is also biologically plausible (e.g., homeostasis)
strength_weakness: strength_weakness: - Though I appreciate the novel problem statement using meta-learned plasticity rules, I could not find technical novelties in the paper. In other words, my primary concern is if this paper presents any novel insights in machine learning perspective. 

- For example, meta-learning of plasticity rules has been widely explored in various tasks [1,2,3]. Some of these papers have already employed the similar optimization techniques such as CMA-ES to quickly optimize the meta-learning parameters. Hence, I think the authors should have discussed these methods in the paper. 

[1] Miconi, Thomas, Kenneth Stanley, and Jeff Clune. "Differentiable plasticity: training plastic neural networks with backpropagation." International Conference on Machine Learning. PMLR, 2018.

[2] Najarro, Elias, and Sebastian Risi. "Meta-learning through hebbian plasticity in random networks." Advances in Neural Information Processing Systems 33 (2020): 20719-20731.

[3] Rodriguez, Hector Garcia, Qinghai Guo, and Timoleon Moraitis. "Short-term plasticity neurons learning to learn and forget." International Conference on Machine Learning. PMLR, 2022.

- Also, the idea of homeostasis using inhibitory and excitatory synapses are not new in machine learning perspectives [1,2]. I believe the main contribution in this paper is to understand the meta-learned plasticity rules in modeling neuronal sequences. If so, it seems like applying existing methods to understand another problem. Please correct me if I’m wrong.

[1] Yoshida, Naoto, et al. "Embodiment perspective of reward definition for behavioural homeostasis." Deep RL Workshop NeurIPS 2021. 2021.

[2] Kang, Beomseok, Biswadeep Chakraborty, and Saibal Mukhopadhyay. "Unsupervised 3D Object Learning through Neuron Activity aware Plasticity." arXiv preprint arXiv:2302.11622 (2023).

- It would be nice to have a background section for readers who are not familiar with neuronal sequence modeling and zebra finch HVC.

- The current reference (citation) style looks different to NeurIPS format. I think the references should be written in [x] or (x).
questions: questions: N/A
limitations: limitations: N/A

Comments: title: Reply
comment: Thank you for addressing my concerns and clarifying the contribution of this work. I understand that the primary contribution at the perspective of neuroscience. To be honest, I'm not much familiar with neuroscience and still not sure on how significant contribution this paper has made. Considering the answer and evaluation from other reviewers, I would like to increase my final score to 5 but decrease the confidence to 2.
Thank you for addressing all points raised. We will keep our score.
Thank you for answering my questions and addressing the suggested manuscript improvements. I am happy to maintain my original high score.
Many thanks to the authors for their rebuttal. I remain unconvinced that the method is in any way intrinsic (time-since-input is functionally the same as time-since-simulation-start), but I still think it's an interesting set of experiments.
The authors made significant efforts to address my comments, and have also improved the paper according to the other reviewers' comment. I'm therefore happy to increase my evaluation.