Structured flexibility in recurrent neural networks via neuromodulation

Authors: julia c costacurta
shaunak  bhandarkar
david  zoltowski
scott w linderman
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:27:09

Abstract: A core aim in theoretical and systems neuroscience is to develop models which help us better understand biological intelligence. 
Such models range broadly in both complexity and biological plausibility. 
One widely-adopted example is task-optimized recurrent neural networks (RNNs), which have been used to generate hypotheses about how the brain’s neural dynamics may organize to accomplish tasks. 
However, task-optimized RNNs typically have a fixed weight matrix representing the synaptic connectivity between neurons. From decades of neuroscience research, we know that synaptic weights are constantly changing, controlled in part by chemicals such as neuromodulators. 
In this work we explore the computational implications of synaptic gain scaling, a form of neuromodulation, using task-optimized low-rank RNNs.
In our neuromodulated RNN (NM-RNN) model, a neuromodulatory subnetwork outputs a low-dimensional neuromodulatory signal that dynamically scales the low-rank recurrent weights of an output-generating RNN. 
In empirical experiments, we find that the structured flexibility in the NM-RNN allows it to both train and generalize with a higher degree of accuracy than low-rank RNNs on a set of canonical tasks.
Additionally, via theoretical analyses we show how neuromodulatory gain scaling endows networks with gating mechanisms commonly found in artificial RNNs. 
We end by analyzing the low-rank dynamics of trained NM-RNNs, to show how task computations are distributed.

Main paper: 
 introduction: 
 Humans and animals show an innate ability to adapt and generalize their behavior across various environments and contexts. This suggests that the neural computations producing these behaviors must have flexible dynamics that are able to adjust to these novel conditions. Given the popularity of recurrent neural networks (RNNs) in studying such neural computations, a key question is whether this flexibility is (1) adequately and (2) accurately portrayed in RNN models of computation.Traditional RNN models have fixed input, recurrent, and output weight matrices. Thus, the only way for an input to impact a dynamical computation is via the static input weight matrix. Prior work has shown that flexible, modular computation is possible with these models [1] , but looking to the biology suggests alternative ways of modeling. In particular, experimental neuroscience research has shown that synaptic strengths in the brain (akin to weight matrix entries in RNNs) are constantly changing -in part due to the influence of neuromodulatory signals [2] .Neuromodulatory signals are powerful and prevalent influences on neural activity and subsequent behavior. Dopamine, a well-known example, is implicated in motor deficits resulting from Parkinson's disease and has been the subject of extensive study by neuroscientists [3] . For computational study, neuromodulators are especially interesting because of their effects on synaptic connections and learning [4] . In particular, neuromodulators have been shown to alter synaptic strength between neurons, effectively reconfiguring circuit dynamics [5] .In this work, we seek to incorporate a neuromodulatory signal into task-trained RNN models. We propose a model consisting of a pair of RNNs: a small "neuromodulatory" RNN and a larger, low-rank "output-generating" RNN. The neuromodulatory RNN controls the weights of the outputgenerating RNN by scaling each rank-1 component of its recurrent weight matrix. This allows the network to produce flexible, yet structured dynamics that unfold over the course of a task. We first review background work in both the machine learning and computational neuroscience literature. Next, we introduce the model and provide some intuition for the impact of neuromodulation on the model's dynamics, relating our model to the canonical long short-term memory (LSTM) network. We end by presenting the performance and generalization capabilities of neuromodulated RNNs on a variety of neuroscience and machine learning tasks, showcasing the ability of a relatively simple, biologically-motivated augmentation to enhance the capacity of RNN models.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We believe that the claims made in the introduction (i.e. that the NM-RNN performs and generalizes better than the LR-RNN) are adequately shown by the results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 First, we review related work in the theoretical neuroscience and machine learning literature.Our work builds on a body of literature dating back to the 1980s, when pioneering computational neuroscientists added neuromodulation to small biophysical circuit models (for reviews, see [6] [7] [8] ). These models consist of coupled differential equations whose biophysical parameters are carefully specified to simulate biologically-accurate spiking activity. As Marder relates in her retrospective review [5] , such neuromodulatory models were created in response to neuronal circuit models which viewed circuit dynamics as "hard-wired". Neuromodulatory mechanisms offered an answer to experimental observations that anatomically fixed biological circuits were capable of producing variable outputs [9, 10] . Of particular relevance to this work, in his 1990 paper Abbott [11] showed that adding a neuromodulatory parameter to an ODE model of spiking activity allows a network of neurons to display capacity for both long-and short-term memory and gate the learning process, anticipating the LSTMs that would become prominent a few years later. We also draw comparisons between our model and the LSTM in the sections that follow. However, our work does not aim to model any specific biophysical system; rather, it aims to begin to bridge the gap between these highly biologically-accurate models and general network models (i.e. RNNs) of neuronal activity by adding a biologically-motivated form of structured flexibility.More recent attempts to model neuromodulation have taken advantage of increased computational power. Kennedy et al. [12] modeled modulatory influence in spiking RNNs by linearly scaling the firing rates of a subset of neurons. Papadopoulos et al. [13] also use spiking RNNs, and incorporate arousal-mediated modulatory signals to induce phase transitions. Stroud et al. [14] use a balanced excitatory/inhibitory RNN to model motor cortex, and incorporate modulatory signals as constant multipliers on each neuron's activity. Liu et al. [15] study how neuromodulatory signals facilitate credit assignment during learning in spiking neural networks. Duong et al. [16] study multi-timescale synaptic plasticity and gain modulation in the context of adaptive whitening in early sensory processing. In addition to RNN models, gain modulation has been explored in the context of neural ODEs [17] and feedforward networks [18] .The work of Tsuda et al. [19] is most similar to what we present here. The authors train RNNs using constant, multiplicative neuromodulatory signals applied to pre-specified subsets of the recurrent weights. They show that these neuromodulatory signals allow an otherwise fixed network to perform variations of a task. In contrast, we employ time-dependent neuromodulatory signals that allow dynamics to evolve throughout tasks. Instead of pre-specifying the values and regions of impact of neuromodulators, we allow the model to learn the time-varying neuromodulatory signal and what segment of the neuronal population it impacts.Our approach is closely related to recent work using hypernetworks to enhance model capacity. Ha et al. [20] use small networks (termed hypernetworks) to generate parameters for layers of larger networks. In their HyperRNN, a hypernetwork generates the weight matrix of an RNN as the linear combination of a learned set of matrices. We also allow our neuromodulatory network to specify the weight matrix of a larger RNN as a linear combination of a learned set of matrices; however, our learned matrices are rank-1 to facilitate easier dynamical analysis and faster training. It is also worth noting that in practice, Ha et al. [20] simplify their HyperRNN so that the hypernetwork scales the rows of a learned weight matrix, which could be seen as postsynaptic scaling in our model. Similarly, von Oswald et al. [21] study the ability of hypernetworks to learn in the multitask and continual learning setting. They find that hypernetworks trained to produce task-specific weight realizations achieve high performance on continual learning benchmarks. In exploring potential neuroscience applications of their work, they remark that while their approach might be unrealistic, a hypernetwork that outputs lower-dimensional modulatory signals could assist in implementing task-specific mode-switching. We seek to obtain similar performance gains with a biologically plausible model of neuromodulation.For a variety of tasks of interest, measured neural recordings are often well-described by a set of lower dimensional latent variables [22, 23] (although see [24, 25] for an alternative view). Likewise, artificial neural networks trained to solve tasks that mimic those found in neural experiments also often exhibit low-rank structure [26] . Based on these findings, recurrent neural networks with low-rank weight matrices (also called low-rank RNNs, LR-RNNs) have emerged as a popular class of models for studying neural dynamics [27] [28] [29] . Importantly, low-rank RNNs are able to model nonlinear dynamics which evolve in a low-rank subspace, offering potential for visualization and interpretation. Here, we leverage low-rank RNNs as ideal candidates for neuromodulation, with each factor of the low-rank recurrence matrix becoming a possible target for synaptic scaling.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We mention limitations of our approach in the Discussion section of the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
neuromodulated recurrent neural networks: 
 Motivated by the appealing dynamical structure of low-rank RNNs and the ability of neuromodulation to add structured flexibility, we propose the neuromodulated RNN (NM-RNN). The NM-RNN consists of two linked subnetworks corresponding to neuromodulation and output generation. The output generating subnetwork is a low-rank RNN, which admits a natural way to implement neuromodulation. We allow the output of the neuromodulatory subnetwork to scale the low-rank factors of the output generating subnetwork's weight matrix. In particular, we propose incorporating neuromodulatory drive via a coupled ODE with neuromodulatory subnetwork state z(t) ∈ R M and output-generating state x(t) ∈ R N :EQUATIONEQUATIONEQUATIONwhere the dynamics matrix W x z(t) is a function of the neuromodulatory subnetwork state z(t) via a neuromodulatory signal s(z(t)) ∈ R K which scales each low-rank component of W x :EQUATIONThe output-generating subnetwork is modeled as a size-N low-rank RNN where u(t) ∈ R P are the inputs, B x ∈ R N ×P are the input weights, and ϕ(•) is the tanh nonlinearity. The neuromodulatory subnetwork is modeled as a small vanilla RNN with its own time-constant τ z , recurrence weights W z ∈ R M ×M , and input weights B z ∈ R M ×P . To limit the capacity of the neuromodulatory subnetwork, we take its dimension M to be smaller than the output-generating subnetwork's dimension N . We set τ z ≫ τ x since neuromodulatory signals are believed to evolve relatively slowly [30] .The neuromodulatory subnetwork state z(t) alters the rank-K dynamics matrix W x ∈ R N ×N via a K-dimensional linear readout s(z(t)), where σ(•) is the sigmoid nonlinearity. The components of s act as linear scaling factors on each rank-1 componentℓ k r ⊤ k ∈ R M ×M of W x .For ease of notation, in the rest of the text we write s(t) to mean s(z(t)). The output y(t) ∈ R O of the paired networks is a linear readout of x(t).This augmentation to the RNN framework allows for structured flexibility in computation. In traditional RNNs, the recurrent weight matrix is fixed, and thus the inputs to the system can only perturb the state of the network. In the NM-RNN, the neuromodulatory subnetwork can use information from the inputs to dynamically up-and down-weight different low-rank components of the recurrent weight matrix, offering greater computational flexibility. As we will see below, this flexibility also allows the network to reuse dynamical components across different tasks and task conditions.To gain some intuition for the potential impacts of neuromodulation on RNN dynamics, first consider the case where W x is symmetric (i.e., ℓ k = r k ∀k), where {ℓ k } 1≤k≤K forms an orthonormal set, where the nonlinearity is removed (i.e., ϕ(x) = x), and where there are no inputs (i.e., u(t) = 0 ∀t). We can then reparameterize the system with a new hidden state w(t) = L ⊤ x(t), where L ∈ R N ×K is the matrix whose columns are ℓ k (so that L T L = I). This produces decoupled dynamics:EQUATIONwhere S(t) = diag (s(t)). Solving this ODE gives an equation for the components of w(t):w k (t) = w k (0) exp - t 0 (1 -s k (t ′ )) τ x dt ′From this equation and the visualization in Fig. 2A , we see that the decay rate of each component w k (t) is governed by its corresponding neuromodulatory signal s k (t). In this way, s(t) can effectively speed up or slow down decay of dynamic modes, similar to gating in an LSTM.Having observed that neuromodulation can alter the timescales of dynamics, note further that the low-rank update for the linearized NM-RNN in eq. ( 5) mirrors the cell-state update equation for a long short-term memory (LSTM) cell [31] . Specifically, the neuromodulatory signal s(t) resembles the forget gate of an LSTM (fig. 2B ). Indeed, as in eq. ( 5), if we linearize the output-generating subnetwork of the NM-RNN and assume that L = R (so that W x is symmetric) and L T L = I, then for w(t) = L T x(t) and τ x = 1, the discretized low-rank dynamics are given byEQUATIONLSTMs have two states that recurrently update across each timestep t: a hidden state h t ∈ R N LSTM and a cell state c t ∈ R N LSTM . Equation ( 6) closely mirrors the cell-state update of the LSTM:EQUATIONHere, the forget gate f t is a form of modulation that depends on the LSTM hidden state h t , much like the NM-RNN's neuromodulatory signal s(t) is a form of modulation depending on the NM-RNN's neuromodulatory subnetwork state z(t). The second term i t ⊙ ct can be viewed as a gated transformation of the input signal u(t). In fact, under suitable assumptions, we show that the dynamics of an NM-RNN can be reproduced by those of an LSTM (see Supplementary Material, Proposition 1).As a gated analog of the RNN, the LSTM has enjoyed greater success than ordinary RNNs in performing tasks that involve keeping track of long-distance dependencies in the input signal [32] . Thus, highlighting the connection between the NM-RNN and LSTM suggests the NM-RNN's ability to successfully model long-timescale dependencies, unlike regular RNNs (see Section 6).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All proofs and derivations are included in full in the Supplementary Material.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
time interval reproduction: 
 To evaluate the potential of neuromodulated RNNs to add structured flexibility, we first consider a timing task since neuromodulators such as dopamine are implicated in time measurement and perception [33] . In the Measure-Wait-Go (MWG) task (Fig. 3A ) [34] , the network receives a 3channel input containing the measure, wait, and go cues. The network must measure the interval between the measure and wait cues, and reproduce it at the go cue by outputting a linear ramp of the same duration. Tasks such as this one are commonly used to study the neural underpinnings of timing perception in humans and non-human primates [35, 36] .To investigate how the NM-RNN's neuromodulatory signal is constrained by the task requirements, we first consider a class of analytically tractable NM-RNNs: networks for which the output-generating subnetwork is linear (i.e., the tanh nonlinearity is replaced with the identity function) and rank-1. In this case, there is one pair of row and column factors, ℓ and r, respectively. If the target output signal is given by f (t) and there are no inputs, then an NM-RNN that successfully produces this output signal will precisely have the neuromodulatory signal,s(t) = f (t) + τ x f ′ (t) -d (ℓ ⊤ r) f (t) -c T w ⊥ (0)e -t/τx -d + (c ⊤ ℓ) r ⊤ w ⊥ (0)e -t/τx, where our readout is y= c ⊤ x + d and w ⊥ (t) := x(t) -1 ||ℓ|| 2 2 ℓℓ ⊤ x(t) is the component of x(t)evolving outside of the column space of ℓ (viewed as a matrix in R N ×1 ). For the full derivation, see the Supplementary Material, Section B. If we assume further that w ⊥ (0) is sufficiently small and τ x is also sufficiently small, then we may make the approximation,EQUATIONIn the MWG task, no inputs are provided to the network during the ramping period following the go cue, so eq. ( 8) applies. In fig. 3B , we show that the neuromodulatory signal of a trained rank-1 NM-RNN during the ramping period matches closely with the theoretical prediction made by eq. ( 8) for both trained and extrapolated target intervals.To continue our analysis of NM-RNNs on the MWG task, we increase the rank of the outputgenerating subnetwork to three. We do this to compare to the networks shown in Beiran et al. [34] and to showcase networks with more degrees of structured flexibility. In Beiran et al. [34] , the authors show that rank-3 low-rank RNNs perform and extrapolate better on this task when provided a tonic context-dependent input, which varies depending on the length of the desired interval. As we have mentioned, such sensory inputs to the network may only alter the resulting dynamics by being passed through the input weight matrix. We propose the NM-RNN as an alternative mechanism by which inputs may alter the network dynamics. to reproduce four intervals, then tested their extrapolation to longer and shorter intervals. In the LR-RNN and NM-RNN, the low-rank matrix was chosen to have rank 3, as in Beiran et al. [34] . In fig. 3C , we plot the L2 losses for ten instances of each model. We see that although the vanilla RNN, LSTM, and NM-RNN are all able to train accurately, the NM-RNN consistently achieves a lower loss on the extrapolated intervals. In fig. 3D , we then show outputs for a typical LR-RNN and NM-RNN (performance of these visualized networks indicated by stars in fig. 3C , visualizations of vanilla RNN and LSTM shown in Supp. Fig. 1 ). The outputs of the NM-RNN have more accurate slope and shape for both trained and extrapolated intervals, with the LR-RNN failing to reproduce shorter and longer extrapolated intervals.We then investigate how the neuromodulatory signal s(t) contributes to the task computation. Figure 3E shows the three dimensions of s(t) plotted over the full range of trained and extrapolated intervals. Each dimension shows activity correlated to particular stages of the task. In fig. 3E (right), we see that s 1 (t) and s 3 (t) have activity highly correlated to the measured interval. In particular, we can see that between the wait and go cues, s 1 (t) separates shorter intervals from longer ones, setting up initial dynamics for the go period when the ramp is generated. The third component s 3 (t) appears to be involved with ending the output ramp, since it saturates first for the shorter intervals and then the longer ones. Figure 3F shows the result of ablating each dimension of s(t) by keeping that component fixed around its initial value. We see that performance suffers in all cases, especially when ablating the effect of s 1 (t) and s 3 (t). Most dramatically, ablating s 3 (t) destroys the ability of the network to change the slope of the output ramp appropriately. These results show that the network uses its neuromodulatory signal to process timing information and generalize across task conditions.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe the methods by which we generate synthetic data trials in the main text and Supplementary Material. We also include specific training details and hyperparameters in the Supplementary Material.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
reusing dynamics for multitask learning: 
 Next, we move beyond generalization within a single task to investigate the capabilites of the NM-RNN when switching between tasks. There has been recent interest in studying how neural network models might reassemble learned dynamical motifs to accomplish multiple tasks [1, 37] . Driscoll et al [1] showed that an RNN trained to perform an array of tasks shares modular dynamical motifs across task periods and between tasks. With this result in mind, we were curious how the NM-RNN might use its neuromodulatory signal to flexibly reconfigure dynamics across tasks.We performed our analysis using the four-task set from Duncker et al. [38] , which includes the tasks DelayPro, DelayAnti, MemoryPro, and MemoryAnti illustrated in fig. 4A . In the DelayPro task, the network receives a three-channel input consisting of a fixation input and two sensory inputs which encode an angle θ ∈ [0, 2π) as (sin(θ), cos(θ)). The fixation input starts and remains at 1, then drops to 0 to signal the start of the readout period, when the network must generate its response. The sensory inputs appear after a delay, and persist throughout the trial. The goal of the network is to produce a three-channel output which reproduces the fixation and sensory inputs. In the MemoryPro task, the sensory inputs disappear before the readout period, requiring the network to store θ. In the 'Anti' tasks, the networks must instead produce the opposite sensory outputs, (sin(θ + π), cos(θ + π)), during the readout period. The task context is fed in as an additional one-hot input. These tasks are analogous to variants of the center-out reaching task, which has been used to study the neural mechanisms of motion in non-human primates [39] .To study the potential of NM-RNNs to flexibly reconfigure dynamics to perform a new task, we only fed the contextual inputs to the neuromodulatory subnetwork, and not to the output-generating subnetwork. This required the model to reuse the output-generating subnetwork's weights when adding a new task. We trained an NM-RNN to perform the first three tasks in the set (DelayPro, DelayAnti, MemoryPro), then froze the weights of the output-generating subnetwork and retrained only the neuromodulatory subnetwork's weights on the fourth task, MemoryAnti. We compared this to retraining the input weights of LR-RNNs, vanilla RNNs, and LSTMs, to investigate two strategies of processing context.We trained parameter-matched NM-RNNs (N = 100, M = 20, K = 3, τ x = 10, τ z = 100), LR-RNNs (N = 100, K = 3, τ = 10), vanilla RNNs (N = 18, τ = 10), and LSTMs (N = 8) in this training/retraining framework. Figure 4B shows performance of example networks on the trained and retrained tasks, using the percent correct metric from Driscoll et al. [1] . The NM-RNN matches the performance of the LSTM and higher-rank vanilla RNN, and considerably outperforms the LR-RNN with no modulation. This performance gain over LR-RNNs is not the result of retraining more parameters; in fact, due to the contrasting sizes of the neuromodulatory and low-rank subnetworks, the input weight matrix of the comparison LR-RNN contains more parameters than the entire neuromodulatory subnetwork, since it must process all inputs (context, sensory, and fixation). To see exactly how the recurrent dynamics were rearranged for this new task, we plotted the neuromodulatory signal of an example network for learned and extrapolated tasks in fig. 4C .We then analyzed the dynamical structure of one of the NM-RNNs by performing PCA on the output-generating subnetwork's state x(t) for a variety of input angles. Figure 4D shows the first two PCs of the neural activity during the stimulus presentation period (before the stimulus shut off for Memory trials). During this period, the neural activity spreads out to arrange on a ring according to the measured angle. After the stimulus disappears in the MemoryPro/Anti tasks, the neural activity decays back along these axes, but it is still decodable based on its angle from the origin (see Supp. Fig. 4 ). To find how this model encoded Pro/Anti versions of tasks, we performed another PCA on the neural activity during the readout period. As shown in fig. 4E , the sign of PC5 during this period is correlated with whether the task is Pro or Anti. Curiously, the positive/negative relationship flips for θ ∈ (π, 2π), likely relating to the symmetric structure of sine and cosine. These results show the ability of the NM-RNN to flexibly reconfigure the dynamics of the output-generating subnetwork, both to solve multiple tasks simultaneously and to generalize to a novel task.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
capturing long-term dependencies via neuromodulation: 
 Inspired by the similarity between the coupled NM-RNN and the LSTM (see section 3.2), we designed a sequence-related task with long-term dependencies, called the Element Finder Task (EFT). On this task, gated models like the NM-RNN outperform ordinary RNNs. When endowed with suitable feedback coupling from the output-generating subnetwork to the neuromodulatory subnetwork, the NM-RNN demonstrates LSTM-like performance on the EFT, while vanilla RNNs (with matched parameter count) fail to solve this task.In the EFT (fig. 5A ), the input stream consists of a query index, q ∈ {0, 1, . . . , T -1} followed by a sequence of T randomly chosen integers. The goal of the model is to recover the value of the element at index q from the sequence of integers. At each time for t ≥ 1, the tth element of the sequence is passed as a one-dimensional input to the model. At time t = T , the model must output the value of the element at index q in the sequence. For our results (shown below), we took T = 25.We trained several NM-RNNs, LR-RNNs, full-rank RNNs, and LSTMs on the EFT, conserving the total parameter count across networks. To emphasize its connection to the LSTM, each NM-RNN included an additional feedback coupling from x(t) to z(t):EQUATIONEach model used a linear readout with no readout bias. The resulting performances of each model tested are shown in fig. 5B . Figure 5C moreover illustrates the learning dynamics (as measured by MSE loss) for a single run of selected networks. Like LSTMs, NM-RNNs successfully perform the task, whereas low-and full-rank RNNs largely fail to do so.To understand how a particular NM-RNN (N = 18, M = 5, K = 8, τ x = 2, τ z = 10) uses neuromodulatory gating to solve the EFT, we visualize the trial-averaged behavior of different components of s(t) across query indices (q = 5, 10, 15, 20), revealing that certain components of s(t) transition between 0 and 1 on a timescale correlated to the query index q (fig. 5D ; left and right); while other components zero out (fig. 5 ; middle). Visualizing a low-dimensional projection of z(t) across different query indices reveals that z(t) settles to a fixed point on an approximate line attractor encoding query index q (fig. 5E ). These findings show that z(t) attends to the query index, facilitating gate-switching behavior in s(t) upon arrival of the queried element.Next, we analyze x(t) by visualizing its top 2 principal components across each combination of the query indices q = 5, 10, and 15 and the target element values -10, -5, 0, 5, and 10 (fig. 5F ).Trajectories with different element values but the same query index start at the same location. Each trajectory converges towards the origin, and upon arrival of the query timestep, rapidly moves to a fixed point on an approximate line attractor that encodes element value. The arrangement of fixed points along this line moreover preserves the ordering of their corresponding element values. In summary, these results show that the NM-RNN solves the EFT by distributing its computations across the neuromodulatory subnetwork, which attends to the query index, and the output-generating subnetwork, which retrieves the target element value. 
discussion: 
 As we have shown, neuromodulated RNNs display an increased ability to both perform and generalize on tasks, demonstrating an important computational implication of synaptic gain scaling. This enhanced performance is enabled by the structured flexibility neuromodulation adds to the dynamics of the network, via modulation of the singular values of the low-rank recurrent weight matrix. As we have shown both theoretically and in practice, this flexibility lends itself particularly well to tasks with timing-related variability. In addition, we saw performance gains over the LR-RNN for the multitask paradigm. Curiously, the gating-like dynamics introduced by adding neuromodulation are reminiscent of the canonical LSTM, and we can prove equivalence under certain conditions.Limitations. One limitation of this work relates to the scale of the networks tested. Our networks were on the scale of N ≈ 100 neurons at their largest, as opposed to other related works which use neuron counts in the thousands. However, we found that this number of neurons was adequate to perform the tasks we presented. We also have yet to compare our results to neural data, limiting our ability to draw biological conclusions.Future Work. We are excited at the potential of future work to further bridge the gap between biophysical and recurrent neural network models. To expand on the NM-RNN model, we aim to embrace the broad range of roles neuromodulation can play in neural circuits. Potential future avenues include: (1) sparsifying the rank-1 components of the recurrent weight matrix to better imitate the ability of neuromodulators to act on spatially localized subpopulations of cells; (2) changing the readout function of s(t) to enable it to take both negative and positive values, in line with the ability of neuromodulators to act as both excitatory and inhibitory; and (3) investigating how different neuromodulatory effects may act on different timescales, both during task completion and learning over longer timescales [4, 5] . More generally, each neuron (or synapse) could have an internal state beyond its firing rate which is manipulated by neuromodulators, as in recent work investigating the role of modulation in generating novel dynamical patterns [40, 41] . Beyond neuromodulators, there exist a multitude of extrasynaptic signaling mechanisms in the brain, such as neuropeptides and hormones, each with their own computational and modeling implications.In this work, we only analyzed networks post-training. We are also curious how our computational mechanism of neuromodulation impacts the network during learning. Prior work has modeled the role of neuromodulation in learning, for example, by augmenting the traditional Hebbian learning rule with neuromodulation to implement a three-factor learning rule [42] , and by using neuromodulation to create a more biologically plausible learning rule for RNNs [43] . Our neuromodulatory signal could induce similar mechanisms during learning.NeurIPS Paper Checklist 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Descriptions of the required compute resources used are included in the Supplementary Material.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We have reviewed the Code of Ethics and believe our paper conforms to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper dealt with modeling synthetic data, with no current real-world use cases.While there are no immediate societal impacts that we anticipate, we believe this kind of work could contribute to better understanding the human brain, leading to improved treatment for disorders.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: This paper proposes a low-rank recurrent neural network model with neuromodulation that modulates the strength of the low-rank factors, and it develops analytical connections with LSTMs. Overall, the reviewers agreed this paper presents a novel solution to an important problem in theoretical neuroscience, while establishing theoretically sound connections with LSTMs. Therefore, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This work mimic the synaptic plasticity observed in brain which is driven by neuromodulators to develop neuro-inspired artificial neural networks. It proposes the neuromodulated NM-RNN, it has a neuromodulatory subnetwork that outputs a low-dim output that will scale the synaptic weights of low-rank RNN. It has connection to LSTMs with similar gating mechanisms, or even the dynamics of NM-RNN could be reproduced with a LSTM. The model is better capturing long-term dependencies. The work also demonstrate how this framework is applicable in multitask learning.
strength_weakness: strength_weakness: **Method**

1. The design choice of rank-1 in modulatory subnetwork should be described, more hyperparameters should be explored.

**Baselines**

1. The proposed NM-RNN models has multiple connections with LSTM, the novelty is therefore limited. Meanwhile, its performance is not comparable to LSTM as shown in Fig5 B. 
2. The baselines are limited, i.e. comparisons with transformers which also has input-dependent attention weights which also model the nonstationarity, and relevant to neural plasticity.

***Evaluation**

1. The tasks and  LSTM is not evaluated in measure-wait-go, multitask learning tasks. Adding more tasks to demonstrate the effectiveness of NM-RNN to captures long-term dependencies.
questions: questions: 1. Why choose rank-1 for modulatory subnetwork?

2. Why the training dynamics (MSE loss) for all models are noisy and fluctuated, any optimization techniques (regularization, normalization, scheduler might help with the training dynamics), how to guarantee the convergence of the model?

3. Any computational efficiency that NM-RNN might bring in?
limitations: limitations: No potential negative societal impact. The limited scale of the network has been discussed.
paper_summary: paper_summary: This work studies the effects of synaptic gain scaling, a neuromodulatory mechanism, on the performance of task-trained low-rank RNNs – which have been used to understand the dynamics and other finer details of neural computation. Specifically, it introduces a simple time-varying neuromodulatory mechanism implemented as a hypernetwork to modulate the weights of low-rank RNNs trained on multiple tasks. The authors draw connections between the proposed model and LSTMs by theoretically demonstrating that the neuromodulation provides for a form of gating. Finally, the authors show that the proposed model outperforms low-rank RNNs and is comparable to LSTMs in multi-task settings and tasks involving long-term dependencies.
strength_weakness: strength_weakness: 1. The model is limited in its bio-realism, so it is unclear how well this would match to computations performed in the brain. Specifically, there are no testable predictions for this model of neuromodulation and no comparison with neural data (as acknowledged by the authors in the limitations), so this limits the significance of the work.
2. The model does not seem to perform as well as LSTMs in the long-term dependencies task (the loss attained by LSTMs is far lower). Furthermore, the authors do not show comparisons with LSTMs and vanilla RNNs for the timing task and the multi-task setting. It would be important to show how the NM-RNN performs in comparison to at least the LSTMs as well in these settings.
3. On a related note to the previous point, the authors have not compared their model to existing implementations of neuromodulation-infused RNNs. Given the similarity of this work to Tsuda et al., 2021 [1] (which also seems to explain observations in neural data from Drosophila), is it possible to compare the NM-RNN to this model, and perhaps other approaches such as those introduced by Liu et al., 2022 [2]?
4. Most importantly, it seems like the scalability of this model is not well-studied and this is also related to the fact that the tasks considered are very simplistic. While these simple tasks allow for some interpretability, I think it would also be important to benchmark the NM-RNN on more complex tasks. For example, studying how it adapts under perturbation in a reaching/control task – this would be a more complex task and could also help understand the links between neuromodulation and learning. In general, moving beyond toy-ish tasks would strongly improve the experimental section of the paper.

(References provided in Questions section.)
questions: questions: See the Weaknesses section. Some additional questions:
1. Have the authors tried training the neuromodulatory subnetwork alone while treating the task subnetwork as a reservoir? This could potentially provide for a parameter-efficient multi-task learning framework (loosely related, see the recent paper by Williams et al., 2024 [3]).
2. Could the authors clarify what the three different NM-RNN curves are in Fig. 5C? If they are different seeds or configurations, this should be specified.
3. Have the authors tried using other activation functions in calculating $\mathbf{s}(\mathbf{z}(t))$, i.e., not restricting the neuromodulatory signal to values between 0 and 1 (and allowing negative values)? Could this lead to improved performance?

**References:**
1. Tsuda et al. "Neuromodulators generate multiple context-relevant behaviors in a recurrent neural network by shifting activity hypertubes." bioRxiv (2021): 2021-05.
2. Liu et al. "Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators." Advances in neural information processing systems 35 (2022): 17528-17542.
3. Williams et al. "Expressivity of Neural Networks with Random Weights and Learned Biases." arXiv preprint arXiv:2407.00957 (2024).

**Rebuttal update:** Confidence increased from 3 to 4. I am in favor of accepting the paper.
limitations: limitations: The limitations have been adequately discussed in the paper (specifically, biological realism, comparison to neural data, and scalability).
paper_summary: paper_summary: The authors introduce and implement a novel biologically-inspired variant of standard recurrent neural networks (RNNs), which they evaluate on a number of tasks that require dynamics to generalize across task conditions (Measure-Wait-Go), switch between tasks/task contexts (four-task set from Duncker et al.), and to capture long-term dependencies (Element Finder Task). Their variant, the neuromodulated RNN (NM-RNN), derives from gated coupling between two networks, where the so-called neuromodulation network dynamically scales the recurrent weights of the output-generating network. The authors provide mathematical intuitions about NM-RNNs and how they relate to LSTMs, compare their performance on above-mentioned tasks, and analyse and visualize the networks' learned dynamics.
strength_weakness: strength_weakness: I find the main weakness of the paper to be a lack of clarity in motivation and terminology. I gather the main motivation of the paper is to "bridge the gap between [...] highly biologically-accurate models and general network models (i.e. RNNs) of neuronal activity by adding a biologically-motivated form of structured flexibility" (lines 64-65) - in order to better be able to study flexibility and generalization capabilities observed in biological neural networks.The latter is not very clear from the paper. 

The link to neuromodulation features prominently in the Introduction, but should be spelled out more clearly, especially how exactly and strongly it links to the suggested architecture (or it should be clearly stated that the inspiration is only of loose nature). The suggested architecture effectively performs synaptic gain scaling, which is only one of many effects of neuromodulation. Streamlining terminology here might help to improve the clarity of the paper and to guide the reader. As an example, the authors mention dopamine as "a well-known example [...] implicated in motor deficits resulting from Parkinson's disease", but that statement needs to be better connected to the presented study and why it motivates the study. 

Both Motivation and Discussion would benefit from an explicit positioning of the approach within the field of NeuroAI: how much does this work contribute to bringing artificial NNs closer to biological NNs, and how much does it help in using ANNs to study BNNs?

In a similar vein, authors may want to discuss their work in relation to:
 - Auzina, I. A., Yıldız, Ç., Magliacane, S., Bethge, M., & Gavves, E. (2023). Modulated Neural ODEs. Proceedings of the 37th Conference on Neural Information Processing Systems. Retrieved from https://github.com/IlzeAmandaA/MoNODE.
- Naumann, L. B., Keijser, J., & Sprekeler, H. (2022). Invariant neural subspaces maintained by feedback modulation. ELife, 11, 76096. https://doi.org/10.7554/eLife.76096
questions: questions: Conceptual:

Can you clearly define what exactly is meant by **structured flexibility**? Is flexibility in standard RNNs unstructured? Specifically, the EFT analysis and visualization convincingly demonstrate structure in the sense that computations are divided between neuromodulatory and output-generating network; in the Memory/Delay/Anti/Pro, this structure is imposed; but for the MWG task, the authors do not seem to analyse structure. 

What have we learned about the computational implications of synaptic gain scaling?

**EFT**

How do the authors predict the performance of the model will change with increasing T? How did the inputs at test time differ from the inputs during training? Is the zeroing out of s(t) components a signature of overparameterization relative to task complexity? 


Technical:

Fig. 2A: The corresponding text talks about different components k, but it is not clear what is shown in the figure (a single component across time, or different components? How do changes in s(t) come about?)

Could this be a plug-in replacement for GRUs/LSTMs? If so, how would the model perform standard sequence modeling tasks, such as sequence classification?

The authors don't mention the vanishing gradient issue that RNNs severely suffer from. Did the not have this problem?

Fig 3. E: please offer more interpretation for what is shown.

Minor:

ln. 104: "Likewise, artificial neural networks trained to solve tasks that mimic those found in neural experi-
ments also often exhibit low-rank structure." Please provide a reference.

Fig 4 A: This panel is not clear. What is shown? Please add labels to all traces and to the axes, and make clear how the subpanels’ axes relate to each other. 

The Memory/Anti task is not readily understandable from the description and visualization in the paper. Please explain this better to make the paper self-contained.

Fig 5C what are the squiggly lines? Fig E not clear where trajectories begin and end, black line not visible. Fig F similarly hard to see/interpret.

Eq. 4: Please specify what \sigma is. 

Fig 3 references mixed up in ln.208.

ln. 267: "Curiously, the positive/negative relationship flips for θ ∈ (π, 2π), likely relating to the symmetric structure of sine and cosine." where can we see this?
limitations: limitations: The authors adequately address the limitations of the work.
paper_summary: paper_summary: The paper proposes an RNN architecture that include synaptic modulation, motivated by neuromodulatory factors in the brain. In essence, the paper shows it is possible to linearly influence the connectivity matrix of a low-rank RNN by scaling it with the output of smaller RNN, the latter nominally describing neuromodulation. The achieved composition of networks outperforms classical RNNs on tasks with timing and history dependence. The presence of the neuromodulatory signal also allow for multitask learning and reuse of previously learnt dynamics to learn a new task. The paper also compares the performance of the NM-RNN with those of a LSTM, both trained on a digit recall task, where LSTMs are known to outperform RNN. Instead, the result show the added flexibility of NM-RNN.
strength_weakness: strength_weakness: While I generally enjoyed the paper, there are a few issues that impact my initial score.

The domain of impact of the paper is mostly in theoretical neuroscience, since it is not clear from the results that the absolute performance of the NM-RNNs actually exceeds popular constructs such as the LSTM (e.g., F5B). Because a lot of the paper is spent discussing the parallels of the NM-RNN gating mechanism and the LSTM, this becomes important from a mechanistic perspective. Can we gain insight into why there is a plateau in performance, despite the argued parallels in architecture? This could be useful from a fundamental ML perspective.

Related to the above, it would be good to understand actual performance of these networks, in addition to the loss (since there is not always a 1-1 correspondence between the loss and performance).

It is also important to discuss the computational implications of the architecture from a trainability standpoint (see Question below).

From the theoretical neuroscience standpoint, the assumption of neuromodulation acting in this way (weighting low rank components) seems quite abstract and strong. I can understand the argument that networks are low rank, but the idea that modulation effectively selects or weights these components would seemingly require a high degree of precision and some ‘knowledge’ of these components at the level of the modulation generation (I note some discussion of this in the Future Work). Is this realistic, and/or are there ways that the developed models could be substantiated in actual experiments? Are there other ways of neuromodulation entering the model that would be similar effective, or does it really need to be as this sum of low rank components?
questions: questions: The architecture seems to require an a prior specification of the maximum rank of the connectivity matrix. From an application standpoint, how would this be determined?

Is the proposed method amenable to batch training? Can the authors characterize the trainability/computational implementation considerations of the proposed architecture, relative to say an LSTM? Certainly, there would seem to be more overhead relative to a vanilla RNN.
limitations: limitations: The discussion contains several limitations and future work directions, which are largely appropriate and appreciated.

Comments: comment: I just wanted to step in and provide my perspective here as another reviewer, because I believe that the papers this reviewer mentions are orthogonal to what I believe is the goal of this paper.

The goal here is to propose a neuromodulatory mechanism inspired by synaptic gain scaling, as a circuit-level mechanistic model of how modulatory signals could reconfigure the dynamics of a recurrent network to perform different tasks. I view it as a mechanistic model rather than a predictive one, i.e., it describes how a mechanism that is computationally similar to what is observed in the brain enables multi-task flexibility, rather than serving as a better predictive model or advancing performance from a deep learning perspective. In short, it is trying to computationally analyze the effect of synaptic gain scaling on multi-task performance.

Several works have used recurrent neural networks to describe how the brain could be performing certain task-related computations, including influential work by Yang et al. [1] and Driscoll et al. [2]. While I'm not arguing here that somehow vanilla RNNs are more bio-plausible or better brain models than LSTMs, I believe there are different levels of modeling and biological realism in the computational neuroscience literature. Circuit-level computational models such as this work eschew synapse-level biological plausibility and aim to study computations through the lens of dynamics [3], task performance, or other hallmarks of neural computation like flexibility and generalization.

I would thus argue that RNNs are good and simple/minimal models to use here – we are aware of recurrent circuits in various brain regions [4,5,6,7], but there is no similar support for transformer-like architectures or self-attention, to my knowledge (although on the other hand, these models are, recently, used at the cognitive level of modeling [8,9]). And given that the kind of tasks being performed here are related to working memory, which relies on the prefrontal cortex [10], which is further known to contain several recurrent microcircuits [6,7], I'd argue that RNNs are better computational models to use here than transformers if the goal is to model neural computation (which I strongly believe to be the case here).

Finally, and perhaps most importantly, the papers that the reviewer refers to here are not trying to model how the brain could compute, or how mechanisms in the brain could enable multi-task flexibility. Works such as [11,12,13], using transformers to model neural population activity, are harnessing advances in deep learning to serve as better representation learning methods or predictive models of behavior from neural activity, for e.g., towards the goal of building better decoders from brain-computer interfaces – they are not meant to serve as models of how the brain could compute.

Overall, I just wanted to provide my views here and I hope to engage in further discussion with this reviewer and/or the authors on this.

**References:**
1. Yang, Guangyu Robert et al. “Task representations in neural networks trained to perform many cognitive tasks.” Nature neuroscience vol. 22,2 (2019): 297-306.
2. Driscoll, Laura N et al. “Flexible multitask computation in recurrent networks utilizes shared dynamical motifs.” Nature neuroscience vol. 27,7 (2024): 1349-1363.
3. Driscoll, Laura N et al. “Computation through Cortical Dynamics.” Neuron vol. 98,5 (2018): 873-875.
4. Douglas, Rodney J, and Kevan A C Martin. “Recurrent neuronal circuits in the neocortex.” Current biology : CB vol. 17,13 (2007): R496-500.
5. Wang, Xiao-Jing. “Decision making in recurrent neuronal circuits.” Neuron vol. 60,2 (2008): 215-34.
6. Mante, Valerio et al. “Context-dependent computation by recurrent dynamics in prefrontal cortex.” Nature vol. 503,7474 (2013): 78-84.
7. Fuster, J M. “Memory networks in the prefrontal cortex.” Progress in brain research vol. 122 (2000): 309-16.
8. Didolkar, Aniket, et al. "Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving." arXiv preprint arXiv:2405.12205 (2024).
9. Webb, Taylor, et al. "A Prefrontal Cortex-inspired Architecture for Planning in Large Language Models." arXiv preprint arXiv:2310.00194 (2023).
10. Funahashi, Shintaro. “Working Memory in the Prefrontal Cortex.” Brain sciences vol. 7,5 49 (2017).
11. Ye, Joel, and Chethan Pandarinath. "Representation learning for neural population activity with Neural Data Transformers." arXiv preprint arXiv:2108.01210 (2021).
12. Le, Trung, and Eli Shlizerman. "Stndt: Modeling neural population activity with spatiotemporal transformers." Advances in Neural Information Processing Systems 35 (2022): 17926-17939.
13. Antoniades, Antonis, et al. "Neuroformer: Multimodal and multitask generative pretraining for brain data." arXiv preprint arXiv:2311.00136 (2023).
Thank you for your response. We would like to address your remaining concerns:

To retrain our networks on the new task, we retrained weights which we viewed as "processing" the input for each model. For the NM-RNN, we retrained the parameters of the neuromodulatory subnetwork (input, recurrent, and output weights) and froze the parameters of the larger output-generating subnetwork (input, low-rank recurrent component, and output weights). In essence, we learned a new neuromodulatory signal s(t) to use with existing low-rank recurrent weights in the output-generating subnetwork. We then sought to come up with a fair comparison to the other models (RNNs/LSTMs). Since these models do not have a separate neuromodulatory subnetwork, allowing them to retrain all of their weights seemed like a strong baseline. Instead, we retrained the input weight matrices for the low-rank/vanilla RNNs and LSTMs. In this case the models have to produce new task outputs using fixed recurrent weight matrices (like the NM-RNN) and produce different behaviors via the input weights.

Our reason for studying this multitask setting was not to test potential for continual learning, but rather to see if the model can solve a new task when it's constrained to re-use prior dynamical motifs. It's true that after retraining there is likely to be overwriting/forgetting in the retrained weights, however we did not attempt to limit this. We think it's interesting that the NM-RNN can solve the new task using low-rank components learned for a prior set of tasks, just weighted differently via the new neuromodulatory signal s(t). Notably, using the previously learned s(t) signals would allow the NM-RNN to immediately switch back to solving old tasks. Although we haven't yet explored whether those signals are maintained or attempted to maintain them, this would be an interesting future direction.

Re: batch training, thank you for the suggestion. To offer some more concrete details, for the rebuttal figures we trained parameter-matched NM-RNNs (N = 100, M = 20, R = 4) and LSTMs (N = 8)  in the multitask setting using batching (1000 samples for each task, batch size 100). We first trained on the original 3 tasks for 100k iterations of gradient descent, then 50k iterations retraining on the new task. The NM-RNN took about 1.5 hours to train, and the LSTM took about 15 minutes. We believe this discrepancy is due to the large difference in internal state sizes. For instance, the NM-RNN’s neuromodulatory subnetwork alone is 20-dimensional while the highest dimension of any matrix in the LSTM is 8. Additionally, we want to emphasize that we did not focus on optimizing the NM-RNN for speed, but rather wanted to demonstrate its task performance compared to the LSTM. We will ensure these details are made more clear in the final version.
Thanks for the authors' response. While I am not convinced that LSTM and transformers are not "brain-like", while RNNs are. I view them all more as predictive models instead of mechanistic models (i.e. hodgkin huxley model). And a lot of recent works have started to use transformers model neural population dynamics [1][2][3], and outperforms RNNs. I think adding more comparisons with baselines, and improving the optimization will be helpful. I decided to maintain my original score.

[1] Representation learning for neural population activity with Neural Data Transformers.

[2] STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer.

[3] Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data.
I thank the reviewers for the rebuttal.  While most of my points have been addressed, I still have a few remaining concerns, namely:
- the implementation of transfer learning is still opaque from my perspective. "we froze all recurrent and output weights and retrained only the weights which directly receive the input" is unclear; unless the input of the modulating network is fixed, but not the task-solving network, my impression is that the setup may lead to substantial overwriting/forgetting, and providing information on re-test of prior tasks is crucial in this regard.
- Re: batch training. The differences in training time should be described transparently, esp. in the context of the ML impacts of this work.

Overall, I am comfortable with my existing score.
I thank the authors for their detailed reply to my review and the improved figures. I'd like to ask the authors to include their clarifications in the final version of the paper as they deem helpful for future readers; especially the positioning within the field of NeuroAI is important. I support acceptance of the paper.
I thank the authors for the rebuttal and appreciate the additional simulations with LSTMs. The authors' reasoning behind not comparing their method with Tsuda et al. and Liu et al. sounds reasonable to me. I am mostly satisfied with the other responses, and while I'd have liked to see more complex tasks/multi-task experiments, I completely understand not being able to do so due to the limited time available, and this is not a reason for me to reject the paper.

Overall, I quite like the straightforwardness of this paper, the theoretical results connecting the proposed mechanism to LSTMs, and the general presentation, figures and interpretable experiments. I maintain my positive opinion and score, and I think this will be a good contribution to computational neuroscience and the audience at NeurIPS.