Structured flexibility in recurrent neural networks via neuromodulation
Authors: Julia C Costacurta
Shaunak Bhandarkar
David M. Zoltowski
Scott Linderman
Venue: NeurIPS 2024 poster (2024)
Keywords: recurrent neural networks
neuromodulation
low-rank recurrent neural networks
timing
biological computation
Primary area: neuroscience_and_cognitive_science
TL;DR: We add a neuromodulation-inspired signal to a low-rank RNN and show that it enhances performance and generalization on neuroscience and machine learning tasks.

Abstract: A core aim in theoretical and systems neuroscience is to develop models which help us better understand biological intelligence. 
Such models range broadly in both complexity and biological plausibility. 
One widely-adopted example is task-optimized recurrent neural networks (RNNs), which have been used to generate hypotheses about how the brain’s neural dynamics may organize to accomplish tasks. 
However, task-optimized RNNs typically have a fixed weight matrix representing the synaptic connectivity between neurons. From decades of neuroscience research, we know that synaptic weights are constantly changing, controlled in part by chemicals such as neuromodulators. 
In this work we explore the computational implications of synaptic gain scaling, a form of neuromodulation, using task-optimized low-rank RNNs.
In our neuromodulated RNN (NM-RNN) model, a neuromodulatory subnetwork outputs a low-dimensional neuromodulatory signal that dynamically scales the low-rank recurrent weights of an output-generating RNN. 
In empirical experiments, we find that the structured flexibility in the NM-RNN allows it to both train and generalize with a higher degree of accuracy than low-rank RNNs on a set of canonical tasks.
Additionally, via theoretical analyses we show how neuromodulatory gain scaling endows networks with gating mechanisms commonly found in artificial RNNs. 
We end by analyzing the low-rank dynamics of trained NM-RNNs, to show how task computations are distributed.

Structured flexibility in recurrent neural networks via neuromodulation
Julia C. Costacurta* Stanford University
Shaunak Bhandarkar*
Stanford University
j costac@stanford. edu
ShaumakbBstanford. edu
David Zoltowski Stanford University dzolton@stanford.cc
Scott Linderman
Stanford University scott. boott.:inderran@@taafooo......
Abstract
A core aim in theoretical and systems neuroscience is to develop models that help better understand biological intelligence. Such models range broadly in both com- plexity and biological plausibility. One widely-adopted example is task-optimized recurrent neural networks (RNNs), which have been used to generate hypotheses about how the brain's 's neural dynamics may organize to accomplish tasks. However, task-optimized RNNs typically have a fixed weight matrix representing the synaptic connectivity between neurons. From decades of neuroscience research, know that synaptic weights are constantly changing, controlled in part by chemicals such as neuromodulators. In this work we explore the computational implications of synaptic gain scaling, a form of neuromodulation, using lask-optimized low- rank RNNs. In our neuromodulated RNN (NM-RNN) model, a neuromodulatory subnetwork outputs a www-dimensional neuromodulatory signal that dynamically scales the low-rank recurrent weights of an output- generating RNN. In empirical experiments, we find that the structured flexibility in the NM-RNN allows it to both train and generalize with higher degree of accuracy than low-rank RNNs on set of canonical tasks. Additionally, via theoretical analyses we show how neu- romodulatory gain scaling endows networks with gating mechanisms commonly found in artificial RNNs. We end by analyzing the low-rank dynamics of trained NM-RNNs, to show how task computations are distributed.
Introduction
Humans and animals show innate ability adapt and generalize their behavior across various environments and contexts. This suggests that the neural computations producing these behaviors must have flexible dynamics that are able to adjust these novel conditions. Given the popularity of recurrent neural networks (RNNs) in studying such neural computations. key question is whether this flexibility is (1) adequately and (2) accurately portrayed in RNN models of computation.
Traditional RNN models have fixed input, recurrent, and output weight matrices. Thus, the only way for an input to impact a dynamical computation is via the static input weight matrix. Prior work has shown that flexible, modular computation is possible with these models [1], but looking to the biology suggests alternative ways of modeling. In particular, experimental neuroscience research has shown that synaptic strengths in the brain (akin to weight matrix entries in RNNs) are constantly changing - in part due to the influence of neuromodulatory signals [2].
Neuromodulatory signals powerful and prevalent influences neural activity and subsequent behavior. Dopamine, well-known example, implicated in motor deficits resulting from Parkinson's disease and has been the subject of extensive study by neuroscientists [3]. For computational study,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

neuromodulators are especially interesting because of their effects on synaptic connections and learning [4]. In particular, neuromodulators have been shown to alter synaptic strength between neurons, effectively reconfiguring circuit dynamics [5].
In this work, we seek incorporate a neuromodulatory signal into task-trained RNN models. We propose a model consisting of a pair of RNNs: a small "neuromodulatory RNN and a larger, low-rank coutput-generating
RNN. The neuromodulatory RNN controls the weights of the output- generating RNN by scaling each rank-1 component of its recurrent weight matrix. This allows the network to produce flexible, yet structured dynamics that unfold over the course of task. We first review background work in both the machine learning and computational neuroscience literature. Next, we introduce the model and provide some intuition for the impact of neuromodulation on the model's dynamics, relating our model to the canonical long short-term memory (LSTM) network. We end by presenting the performance and generalization capabilities of ncuromodulated RNNs on variety of neuroscience and machine learning tasks, showcasing the ability of a relatively simple, biologically motivated augmentation to enhance the capacity of RNN models.
aolloically-mooivateed
2 Background
First, we review related work in the theoretical neuroscience and machine learning literature.
2.1 Modeling neuromodulatory signals
Our work builds on body of literature dating back to the 1980s, when pioneering computational neuroscientists added neuromodulation to small biophysical eircuit models (for reviews, see [6-8]). These models consist of coupled differential equations whose biophysical parameters are carefully specified to simulate biologically-accuree spiking activity. As Marder relates in her retrospective review [5], such neuromodulatory models were created in response to neuronal circuit models which viewed circuit dynamics as "hard-wired". Neuromodulatory mechanisms offered an answer to experimental observations that anatomically fixed biological circuits were capable of producing variable outputs [9, 10]. Of particular relevance to this work, in his 1990 paper Abbott [11] showed that adding neuromodulatory parameter to an ODE model of spiking activity allows a network of neurons to display capacity for both long- and short-term memory and gate the learning process, anticipating the LSTMs that would become prominent a few years later. We also draw comparisons between our model and the LSTM in the sections that follow. However, our work does not aim to model any specific biophysical system; rather, it aims to begin to bridge the gap between these highly waagicalll-acccratt models and general network models (i.e. RNNs) of neuronal activity by adding biologically- motivated eioooiee form of structured flexibility.
More recent attempts to model neuromodulation have taken advantage of increased computational power. Prior work has investigated modulatory influence in spiking neural networks, for example by linearly scaling the firing rates of a subset of neurons [12], incorporating arousal- modulatory signals induce phase transitions [13], and facilitating credit assignment during learning [14]. Stroud et al. [15] use a balanced excitatoryinhiiitory RNN to model motor cortex, and incorporate modulatory signals as constant multipliers on each neuron's activity. Our work similarly proposes a modulatory signal that impacts network activity, but we instead allow this signal to scale factors of the recurrence matrix. Duong al. [16] use a similar factor scaling approach to model adaptive whitening in early sensory processing, Our approach differs by focusing on low-rank recurrence matrices in task-oriented setting. In addition RNN models, gain modulation has been explored in neural ODEs [17] and feedforward networks [18].
The works of Tsuda al. [19] and Vecoven et al. [20] are most similar what we present here. In Tsuda et al. [19], the authors train RNNs using constant, multiplicative neuromodulatory signals applied to pre-specified subsets of the recurrent weights. They show that these neuromodulatory signals allow an otherwise fixed network to perform variations of a task. In contrast, we employ time-dependent neuromodulatory signals that allow dynamies evolve throughout tasks. Instead of pre-specifying the values and regions of impact of neuromodulators, we allow the model to learn the time-varying neuromodulatory signal and what segment of the neuronal population it impacts. Vecoven et al. [20] use a two-network approach in which a neuromodulatory network processes contextual information to alter the activation functions of "main" feedforward deep neural network. They show that this "Neuro-Mon Modulated Network" outperforms RNNs on meta-RI. benchmarks. We also use two-network approach which one network modulates the parameters of the other, but instead applied to recurrent neural networks in RL paradigm.
2

---

2.2 Hypernetworks
Our approach is closely related to recent work using hypernetworks to enhance model capacity. Ha al. [21] use small networks (termed hypernetworks) to generate parameters for layers of larger networks. In their HyperRNN, hypernetwork generates the weight matrix of an RNN as the linear combination of a learned set of matrices. We also allow our neuromodulatory network to specify the weight matrix of a larger RNN as linear combination of learned set of matrices; however, our learned matrices are rank-1 to facilitate easier dynamical analysis and faster training. is also worth noting that in practice, Ha et al. [21] simplify their HyperRNN so that the hypernetwork scales the rows of learned weight matrix, which could be seen as postsynaptic scaling in our model. Similarly, von Oswald et al. [22] study the ability of hyper- networks to learn in the multitask and continual learning setting. They find that hypernetworks trained to produce task-specific weight realizations achieve high performance continual learning benchmarks. In exploring potential neuroscience applications of their work, they remark that while their approach might be unrealistic, hypernetwork that outputs lower-dimensional modulatory signals could assist in implementing task-specific mode-switching. We seek to obtain similar performance gains with a biologi- cally plausible model of neuromodulation.
neuromodulatory subnetwork uft
sift)
sk(t)
dme-varying neuramodulstary signal
st(t)
sx(t)
2.3 Low-rank recurrent neural networks
neuromodulsted low-rank recurrent weights
For a variety tasks of interest, measured neural record- ings often well-described by set of lower dimensional latent variables [23, 24J (although see [25, 26] for an alter- native view). Likewise, artificial neural networks trained to solve tasks that mimic those found in neural experi- ments also often exhibit low-rank structure [27]. Based on these findings, recurrent neural networks with low-rank weight matrices (also called low-rank RNNs, LR-RNNS) have emerged as popular class of models for studying neural dynamics [28-30]. Importantly, low-rank RNNs are able to model nonlinear dynamics which evolve in low-rank subspace, offering potential for visualization and interpretation. Here, we leverage low-rank RNNs as ideal candidates for neuromodulation, with each factor of the low-rank recurrence matrix becoming a possible target for synaptic scaling.
utt)
yft)
output generating subnetwork
The NM-RNN consists of two subnetworks: a low-rank subnetwork that generates the output (bottom) and smaller, full-rank neuromodulatory sub- network (top).
Figure 1:
3 Neuromodulated recurrent neural networks
Motivated by the appealing dynamical structure of low-rank RNNs and the ability of neuromodulation to add structured flexibility, we propose the neuromodulated RNN (NM-RNN). The NM-RNN consists of two linked subnetworks corresponding to neuromodulation and output generation. The output generating subnetwork is a low-rank RNN, which admits natural way implement neuromodulation. We allow the output the neuromodulatory subnetwork to scale the low-rank factors of the output generating subnetwork 's weight matrix. In particular, propose incorporating neuromodulatory drive via a coupled ODE with neuromodulatory subnetwork state xtt) E R and output-generating state x (t) E R
dz(() x(t) W- d(=(())) Bt
dt
dx(()
-x(1)= + W (=(((!) ((x(1) + B,u())
(2)
di
y(t) Cxx(t) + +d,
(3)
3

---

A
Impact of neuromodulation on witt
B
Schematic LSTM
Schematic the NM-RNN
sit)
C-11
C2 1/-1
host
-4
B
$
B,
Ioch
h4
2 1-
B,
0
time
Figure 2: A. Illustration of how neuromodulatory signals s(t) affect decay rates the state w(t) in I-D, simplified model (derived in Section 3.1). calculated with prespecifted s(t) signal. As st)) approaches O, we() decays more rapidly. B. & C. Visual comparison of an LSTM and an NM-RNN. Corresponding parts of the networks are highlighted with shaded rectangles. Blue: forget gate computation. Green: an input gate to recurrent dynamics. Purple: recurrent feedback onto the modulatory state variable.
where the dynamics matrix W (z(ttt) is function of the neuromodulatory subnetwork state x(t) via neuromodulatory signal s(x(()) E RK which scales each low-rank component of W.:
s(z(0) - =A, + b.)
W (=()()) = Σ (x())))) ler..
The oututt-gnnrratnng subnetwork is modeled as size-N low-rank RNN where uit) R are the inputs, Bg E NR are the input weights, and 6(.) is the tanh nonlinearity. The neuromodulatory subnetwork is modeled as a small vanilla RNN with its own time-constant тN, recurrence weights W- E IR MM and input weights B, E RMXP To limit the capacity of the neuromodulatory subnetwork, we take its dimension M to be smaller than the sion N. We set T3 >> Tu since neuromodulatory signals are believed to evolve relatively slowly [31]. The neuromodulatory subnetwork state ztt) alters the rank-K dynamics matrix Wa N via K ---dimensional linear readout s( (z(()), where a(.) is the sigmoid nonlinearity. The components of act as linear scaling factors on each rank-1 component CrT, E XMMX M of W For ease of notation, in the rest of the text we write s()) to mean s(2()). The output y() E RRO of the paired networks is linear readout of (t)..
subnetwork's 's dimen-
This augmentation to the RNN framework allows for structured flexibility in computation. In tradi- tional RNNs, the recurrent weight matrix is fixed, and thus the inputs to the system can only perturb the state of the network. In the NM-RNN, the neuromodulatory subnetwork can use information from the inputs to dynamically up- and down-weight different low-rank components of the recurrent weight matrix, offering greater computational flexibility. As we will see below, this flexibility also allows the network reuse dynamical components across different tasks and task conditions.
3.1 Mathematical intuition
To gain some intuition for the potential impacts neuromodulation on RNN dynamics, first consider the case where Wa is symmetric (i.e., Eg = F Vk), where (Lx (CaJik forms an orthonormal set, where the nonlinearity is removed (i.e., ((z) = 2), and where there are no inputs (i.e., u(t) = 0 Vt). We can then reparameterize the system with a new hidden state wtl) = L ax()), where L E N
is the matrix whose columns are Ly (so that L L 1). This produces decoupled dynamics:
Tr dw((t = wt) + (())
(5)
dt
where St)) diag (s())). Solving this ODE gives an equation for the components of w(t):
8u (t')) (-('(1-a())) di
(1-18x(tt)) Tx
wa (1) = wn) exp
From this equation and the visualization in Fig. 2A, we see that the decay rate of each component wr wx(() (f) governed by its corresponding neuromodulatory signal st.(). this way, 8 st) can effectively speed up or slow down decay of dynamic modes, similar gating in an LSTM.

---

3.2 Connection to LSTMs
Having observed that neuromodulation can alter the timescales of dynamics, note further that the low-rank update for the linearized NM-RNN in eq. (5) mirrors the cell-state update equation for long short-term memory (LSTM) cell [32]. Specifically, the neuromodulatory signal s(1) resembles the forget gate of an LSTM (fig. 2B). Indeed, in eq. (5), if we linearize the outut-generating subnetwork of the NM-RNN and assume that L = R (so that Wa is symmetric) and L L I, then for (l)) L" ax)) and Ta 1, the discretized low-rank dynamies are given by
Wt 8, C w1-1 + L' u
(6)
LSTMs have two states that recurrently update across each timestep t: a hidden state h2 E R.NISTM and cell state E I MNSTTM Equation (6) closely mirrors the cell-state update of the LSTM:
= f C Ct-1 & O Ć₁
Here, the forget gate f, is a form of modulation that depends on the LSTM hidden state h. much like the NM-RNN's neuromodulatory signal s(t) a form of modulation depending on the NM- RNN's neuromodulatory subnetwork state zt). The second term é @ è₂ can be viewed as gated
transformation of the input signal utt). In fact, under suitable assumptions, we show that the dynamics of an NM-RNN can be reproduced by those of an LSTM (see Supplementary Material, Proposition l).
As a gated analog of the RNN, the LSTM has enjoyed greater success than ordinary RNNs in performing tasks that involve keeping track of long fong-distance dependencies in the input signal [33]. Thus, highlighting the connection between the NM-RNN and LSTM suggests the NM-RNN's ability to successfully model Long-timescale dependencies, unlike regular RNNs (see Section 6).
4 Time interval reproduction
To evaluate the potential of neuromodulated RNNs to add structured flexibility, first consider a timing task since neuromodulatorss such as dopamine are implicated in time measurement and perception [34]. In the Measure- Wait--o (MWG) task (Fig. 3A) [35], the network receives 3- channel input containing the measure, wait, and go cues. The network must measure the interval between the measure and wait cues, and reproduce it at the go cue by outputting a linear ramp of the same duration. Tasks such as this one are commonly used to study the neural underpinnings of timing perception in humans and non-human primates [36, 37].
4.1 Experiment matches theory for rank-1 networks
To investigate how the NM-RNN's neuromodulatory signal is constrained by the task requirements, we first consider class of analytically tractable NM-RNNs: networks for which the output-generating subnetwork is linear (i.e., the tanh nonlinearity is replaced with the identity function) and rank-1. In this case, there is one pair of row and column factors, l and respectively. If the target output signal is given by ft) and there are no inputs, then an NM-RNN that successfully produces this output signal will precisely have the neuromodulatory signal,
s(1) + (()) d s(1) (e r) (f(t) cTw (0)e-t/T. - d) + (c (r (0)e-t/r.))
where our readout is = c x + d and w (t) = xt) || ee xt) is the component of zxf) evolving outside of the column space of l (viewed as a matrix in Nx1). For the full derivation, see the Supplementary Material, Section B. If assume further that w (0) is sufficiently small and T is also sufficiently small, then we may make the approximation,
a(1) ~ f(1) + TTf(() - d 1 ET (1+
T2t(() 5(1)
(8)
('r( d)
In the MWG task, no inputs are provided to the network during the ramping period following the go cue, so eq. (8) applies. In fig. 3B, we show that the neuromodulatory signal of trained rank-1 NM-RNN during the ramping period matches closely with the theoretical prediction made by eq. (8) for both trained and extrapolated target intervals.
5

---

A
B
D
Rank-1 Predictions Empirical Results
Low-Rank RNN
measure walt go
vt (output)
Input
spike
0.0
sequence
desired
-0.5
output
ramp
0.8
L2 loss comparison
0.4
NM-RNN
20
0.6
15
vanila
outoout
0.3
12
-0.3
-1.0
80
-0.6
time
extr:
trained
oxtr.
Interval
up
extrapolated
trained
extrapolated up
Neuromodulatory Signal
Neuromodulatory Signal
F
Aligned Measure
Aligned
Ablations
meassre
wait
0.5
5,0
0.7
each
channel constant
00.5
0.5
0.8
0.3
5y0
-0.5
time
time
time
Figure 3: A. Visualization of Measure- Maansure-Wai-GGe Wait-Go task. B. Theoretical predictions (dashed lines) match closely with empirical results (solid lines) for rank-1 network. C. L2 loss comparsion of four model types on MWG task. We trained 10 randomly initialized parameter-matched low-rank RNNs, NM-RNNs, vanilla RNNs, and LSTMs. Median losses shown by bars. Performance of visualized LR-RNN and NM-RNN are starred (example visualizations for all four models shown in Supp. Fig. 1). D. Comparison of model-generated output ramps for both trained (purple) and extrapolated (red, blue) intervals. Dashed lines show target outputs. F.
neuromodulatory signal s()) for traineddextrapolatee intervals. Left, traces are aligned to start of trial. Right, traces are aligned to 'go' cue. F. Resulting output traces when ablating each component of st)). In all panels, colors reflect trained/extrapolltee intervals (see legend). For output plots, dashed grey lines are targets. Additional model visualizations in Supp. Fig. 1-2.
4.2 Improved generalization and interpretability timing task for rank-3 networks
To continue analysis of NM-RNNs on the MWG task, increase the rank of the output- generating subnetwork to three. We do this to compare to the networks shown in Beiran et al. [35] and to showcase networks with more degrees of structured flexibility. In Beiran et al. [35], the authors show that rank-3 low-rank RNNs perform and extrapolate better on this task when provided a tonic context-dependen input, which varies depending on the length of the desired interval. As have mentioned, such sensory inputs to the network may only alter the resulting dynamics by being passed through the input weight matrix. We propose the NM-RNN as an alternative mechanism by which inputs may alter the network dynamics.
We trained parameter- matched NM-RNNs (N = 100, M - = T. = 10, T, = 100). LR-RNNs (N 106, K 3, 10). vanilla RNNs (N 33,, == 10), and LSTMs (N 15) to reproduce four intervals, then tested their extrapolation to longer and shorter intervals. In the LR-RNN and NM-RNN,
low-rank matrix was chosen to have rank 3, as in Beiran al. [35]. In fig. 3C, we plot the L.2 losses for ten instances of each model. We see that although the vanilla RNN, I.TM, and NM-RNN are able to train accurately, the NM-RNN consistently achieves a lower loss on the extrapolated intervals. In fig. 3D, then show outputs for typical LR-RNN and NM- RM-RNN

---

(performance of these visualized networks indicated by stars in fig. 3C, visualizations of vanilla RNN and LSTM shown in Supp. Fig. 1). The outputs of the NM-RNN have more accurate slope and shape for both trained and extrapolated intervals, with the LR-RNN failing to reproduce shorter and longer extrapolated intervals.
We then investigate how the neuromodulatory signal s()) contributes the task computation. Fig- ure 3E shows the three dimensions of 8(4) plotted over the full range of trained and extrapolated intervals. Each dimension shows activity correlated to particular stages of the task. In fig. 3E (right), we see that 81 (1) and 6y(() have activity highly correlated to the measured interval. In particular, we can see that between the wait and go cues, (1( separates shorter intervals from longer ones, setting up initial dynamics for the go period when the ramp is generated. The third component (t) appears be involved with ending the output ramp, since it saturates first for the shorter intervals and then the longer ones. Figure 3F shows the result of ablating each dimension of s(t) by keeping that component fixed around its initial value. We see that performance suffers in all cases, especially when ablating the effect of $1 (t) and (t). Most dramatically, ablating sy(t) destroys the ability of the network to change the slope of the output ramp appropriately. These results show that the
network uses its neuromodulatory signal to process timing information and generalize across task conditions.
5 Reusing dynamics for multitask learning
Next, we move beyond generalization within single task to investigate the capabilites of the NM- RNN when switching between tasks. There has been recent interest in studying how neural network models might reassemble learned dynamical motifs to accomplish multiple tasks [1, 38]. Driscoll et al [1] showed that an RNN trained to perform an array of tasks shares modular dynamical motifs across task periods and between tasks. With this result in mind, we were curious how the NM-RNN might use its neuromodulatory signal to flexibly reconfigure dynamics across tasks.
We performed our analysis using the four-task set from Duncker et al. [39], which includes the tasks DelayPro, DelayAnti, MemoryPro, and Memory Anti illustrated in fig. 4A. In the DelayPro task, the network receives a three-channe input consisting of a fixation input and two sensory inputs which encode an angle E [0, 2x) as (sin(f), c0)( The fixation input starts and remains at 1, then drops to 0 to signal the start of the readout period, when the network must generate its response. The sensory inputs appear after delay, and persist throughout the trial. The goal of the network is to produce a three-channel output which reproduces the fixation and sensory inputs. In the MemoryPro task, the sensory inputs disappear before the readout period, requiring the network to store O. In the 'Anti' tasks, the networks must instead produce the opposite sensory outputs, (sin(0 + rT), cos(O + r)), during the readout period. The task context is fed in as an additional one-hot input. These tasks analogous to variants of the center-out reaching task, which has been used to study the neural mechanisms of motion in non-human primates [40].
To study the potential of NM-RNNs to flexibly reconfigure dynamics to perform a new task, we only fed the contextual inputs to the neuromodulatory subnetwork. and not to the output-generating subnetwork. This required the model to reuse the output-generating subnetwork's 'ss weights when adding a new task. We trained an NM-RNN to perform the first three tasks in the set (DelayPro, AelayAnti, MemoryPro), then froze the weights of the output-genee generating subnetwork and retrained only the neuromodulatory subnetwork's 's weights on the fourth task, MemoryAnti Anti. We compared this to retraining the input weights of LR-NNNs, vanilla RNNs, and LSTMs, to investigate two strategies of processing context.
We trained marhheder-matched NM-RNNs (N- = 100, M = 20, K = To = 10, T2 = 100), LR-RNNs (N = 100, K 3, 7 = 10), vanilla RNNs (N = 18, 10), and LSTMs (NV = 8) in this training/retraining framework. Figure 4B shows performance of example networks on the trained and retrained tasks, using the percent correct metric from Driscoll et al. [1]. The NM-RNN matches the performance of the LSTM and higher-rank vanilla RNN, and considerably outperforms the LR- RNN with no modulation. This performance gain over LR-RNNs is not the result of retraining more parameters; in fact, due to the contrasting sizes of the neuromodulatory and low-rank subnetworks, the input weight matrix of the comparison LR-RNN contains more parameters than the entire neuromodulatory subnetwork, since it must process all inputs (context, sensory, and fixation). To see exactly how the recurrent dynamies were rearranged for this new task, we plotted the neuromodulatory signal of an example network for learned and extrapolated tasks in fig. 4C.
7

---

A
Task
Inputs
Outputs
Multitask Neuromoduultorry Signal
Fixation
Contect
Sensory
Target
0.5
Initially Trained
movn siy0 cooup)))))
stmm (memory)
DelayAnti
nown
sXtP e coupf
sytt
MemoryPro sroP).coatt
Retrained
MemoryAnti
seport sigh
100
time
Performance Comparison
DelayPro
MemoryPro
100
DeleyAnti
MemaryAnti
Readout PCs 4 &
D
E
8 (0,
B (n. 2ml
perceet correct
0.4
2
0.2
8C5
40-
I-mr
mmmm
-0.2
-0.2
vocills
bit
-0.4
trainod
rctrained
4
task
PCI
PC4
PC4
Figure 4: A. Depiction of inputs and targets for four tasks. Networks were trained on the first three tasks, then context-procee processing weights were retrained on the fourth task (final row). B. Performance comparison for four model types in multitask setting, for three initially trained tasks and retrained MemoryAnti task. Median performance shown by bars, with color of bar split case of overlapping. We trained 10 randomly initialized martheter-matched low-rank RNNs, NM-RNNs, vanilla RNNs, and LSTMs. C. Neuromodulatory signal for an example network. Dynamical analysis of network activity during different stages of the tasks. (Left) PCs 1&2 during the stimulus period show ring attractor which stores the measured angle. (Right) Sign of PCS during readout period corresponds to Pro/Anti. Additional model visualizations in Supp. Fig. 3-4.
We then analyzed the dynamical structure of one of the NM-RNNs by performing PCA on the subnetwork's 's state x=t) for variety of input angles. Figure 4D shows the first two PCs of the neural activity during the stimulus presentation period (before the stimulus shut off for arrange
Memory trials). During this period, the neural activity spreads the measured angle. After the stimulus disappears in the MemoryPro/AAe Anti tasks, the neural activity decays back along these axes, but it is still decodable based its angle from the origin (see Supp. Fig. 4). To find how this model encoded Pro/Anti versions of tasks, we performed another PCA on the neural activity during the readout period. As shown in fig. 4E, the sign of PC5 during this period is correlated with whether the task is Pro or Anti. Curiously, the positive/negative relationship flips for 0 (H, 2T), likely relating the symmetric structure of sine and cosine. These results show the ability of the NM-RNN MN- to flexibly reconfigure the dynamics of the output-generating subnetwork, both solve multiple tasks simultaneously and generalize to a novel task.
ring according
Capturing long-term dependencies via neuromodulation
Inspired by the similarity between the coupled NM-RNN and the LSTM (see section 3.2), designed a sequence-relate task with long-term dependencies, called the Element Finder Task (EFT). On this task, gated models like the NM-RNN outperform ordinary RNNs. When endowed with suitable feedback coupling from the output-generatin subnetwork to the neuromodul latory subnetwork, the NM-RNN demonstrates LSTM-like performance the EFT, while vanilla RNNs (with matched parameter count) fail to solve this task.
In the EFT (fig. 5A), the input stream consists of a query index, y (0. 1...,...,7 T --13 1} followed by sequence of 7' randomly chosen integers. The goal of the model is to recover the value of the element at index g from the sequence of integers. At each time for t 2 1, the 4th clement of the sequence is

---

passed as a one one-dimensional input to the model. At time t = T. the model must output the value of the element at index q in the sequence. For our results (shown below), we took T 25.
We trained several NM-RNNs, LR- AR-RNNs, full-rank RNNs, and LSTMs on the EFT, conserving the total parameter count across networks. To emphasize its connection to the LSTM, each NM-RNN included an additional feedback coupling from xtt) to (t):
dz(1) di
=(1) W ((((()) + (B ((x(()(++++)))))) ban) B.u(t)
Each model used a linear readout with no readout bias. The resulting performances of each model tested are shown in fig. 5B. Figure 5C moreover illustrates the learning dynamics (as measured by MSE loss) for a single run of selected networks. Like LSTMs, NM-RNNs successfully perform the task, whereas low- and full-rank RNNs largely fail to do so.
To understand how a particular NM-RNN (N = 18, M = 5 neuromodulatory gating to solve the EFT, we visualize the trial-averaged behavior of different components of s(t) across query indices (q = 5. 10, 15, 20), revealing that certain components of s(t) transition between and 1 on timescale correlated to the query index q (fig. 5D; left and right); oow-dimessional projection of (t)
2, 10) uses
while other components zero out (fig. 5; middle). Visualizing across different query indices reveals that zt) settles to fixed point on an approximate line attractor encoding query index g (fig. 5E). These findings show that =t)) attends to the query index, facilitating gate-switching behavior st)) upon arrival of the queried element.
Next, we analyze ætt) by visualizing its top 2 principal components across each combination of the query indices g = 5, 10, and 15 and the target element values -00,- -5, 0, 5, and 10 (fig. 5F). Trajectories with different element values but the same query index start at the same location. Each trajectory converges towards the origin, and upon arrival of the query timestep, rapidly moves to a fixed point on an approximate line attractor that encodes element value. The arrangement of fixed points along this line moreover preserves the ordering of their corresponding element values. In summary, these results show that the NM-RNN solves the EFT by distributing its computations across the neuromodulatory subnetwork, which attends to the query index, and the output-generating subnetwork, which retrieves the target element value.
7 Discussion
As we have shown, neuromodulated RNNs display an increased ability to both perform and generalize on tasks, demonstrating an important computational implication of synaptic gain scaling. This enhanced performance is enabled by the structured flexibility neuromodulation adds to the dynamics of the network, via modulation of the singular values of the low-rank recurrent weight matrix. As we have shown both theoretically and in practice, this flexibility lends itself particularly well to tasks with timing-related variability. In addition, we saw performance gains over the LR- LR-NNN for the multitask paradigm. Curiously, the gating-like dynamics introduced by adding neuromodulation are reminiscent of the canonical LSTM, and we can prove equivalence under certain conditions.
Limitations. One limitation of this work relates to the scale of the networks tested. Our networks on the scale of N s 100 neurons at their largest, opposed other related works which use neuron counts in the thousands. However, we found that this number of neurons was adequate to perform the tasks presented. We also have yet to compare our results neural data, limiting our ability to draw biological conclusions.
Future Work. We are excited at the potential of future work to further bridge the gap between biophysical and recurrent neural network models. To expand on the NM-RNN model, we aim to embrace the broad range of roles neuromodulation can play in neural circuits. Potential future avenues include: (1) sparsifying the rank-1 components of the recurrent weight matrix to better imitate the ability of neuromodulators to act on spatially localized subpopulations of cells; (2) changing the readout function of s(t) to enable to take both negative and positive values, in line with the ability of neuromodulators to act as both excitatory and inhibitory; and (3) investigating how different neuromodulatory effects may act on different timescales, both during task completion and learning over longer timescales [4, 5]. More generally, each neuron (or synapse) could have internal state beyond its firing rate which is manipulated by neuromodulators, as in recent work investigating the role of modulation in generating novel dynamical patterns [41, 42]. Beyond neuromodulators, there

---

A
B
Model Performance
50
NM-RNN Training Dynamics
Minimum Loss. Attained
Element Finder Task
10"
LSTM
22 9 3 0
6 -3
10a
30
RNN
Quary indox
10-1
MSE
20
NM-RNN
NM-RNN
10-3
LSTM NM--AN LR-RNN
%
5000
10000
15000
20000
Model Tyge
Iterations
D 1.0
Selected Components of s(t)
E
PCs 2 of z(t)
F
PCs 1 & of x(()
10
ge10
15
ges
syltt
0.5
-1
Timesteg
Timestep
Timestep
Figure 5: A. Visualization of the Element Finder Task. B. MSE losses attained across multiple runs in different classes of models trained on the EFT (median indicated by black lines). C. Training loss curves for selected parameter-matched models. The NM-RNNs have hyperparameter counts 1: (M=5, N=18, R=8), (M-5, N=13, R=12), and (M=10, N=12, R=7). D. Visualization of selected components of s(t) for an example NM-RNN, shown across different query indices, E. Trajectories for the top two PCs of (t) across different query indices. The different trajectories converge to an approximate line attractor (black) encoding query index. The time at which the queried element arrives is marked in red. F. Top two PCs of att), visualized for different query indices and target element values. Each trajectory converges to a fixed point on approximate line attractor encoding element value. Each curve shown D, E, and is averaged over 100 trials. Additional visualizations in Supp. Fig. 5-6.
exist multitude of extrasynaptic signaling mechanisms in the brain, such as neuropeptides and hormones, each with their own computational and modeling implications.
In this work, only analyzed networks post-training. We also curious how our computational mechanism neuromodulation impacts the network during learning. Prior work has modeled the role of neuromodulation in learning, for example, by augmenting the traditional Hebbian learning rule with neuromodulation to implement& a three-factor learning rule [43], and by using neuromodulation create biologically plausible learning rule for RNNs [44]. Our neuromodulatory signal could induce similar mechanisms during learning.
Acknowledgements and Disclosure of Funding
We thank Laura Driscoll, Lea Duncker, Gyu Heo, Bernardo Sabatini, and the members of the Linderman Lab for helpful feedback throughout this project. This work was supported by grants from the NIH BRAIN Initiative (U19NS1 1
13201, R01NS131987, & RF1 CRCNS Program (R01NS130789). J.C.C. is funded by the NSF Graduate Research Fellowship, Stanford Graduate Fellowship, and Stanford Diversifying Academia, Recruiting Excellence (DARE) Fellowship. D.M.Z. funded by the Wu Tsai Interdisciplinary Postdoctoral Research Fellowship. S.W.L. is supported by fellowships from the Simons Collaboration the Global Brain, the Alfred Sloan Foundation, and the McKnight Foundation. The authors have no competing interests to declare.
and the NSF/NIH
10

---

References
[1] Laura N Driscoll. Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):339-1163, 2024.
[2] Ami Citri and Robert C Malenka, Synaptic plasticity: multiple forms, functions, and mechanisms. Neumpsychphammccoogy, 33(1):18841, 2008.
[3] Marjan Jahanshahi, Catherine RG Jones, Jan Zijlmans, Regina Katzenschlager, Lucy Lee, Niall Quian, Chris D Frith, and Andrew J Lees. Dopaminergic modulation of striato-frontal connectivity during motor timing in parkinson's 's disease, Brain, 133(3):727-744 2010.
[4] Peter Dayan. Twenty-five lessons from computational neuromodulation. Neuron, 76(1):240-256, 2012.
Eve Marder. Neuromodulation of neuronal circuits: back to future. Neuron, 76(1):1-11, 2012.
[6] Jean-Marc Fellous and Christiane Linster. Computational models of neuromodulation. Neural Computation. 10(4):771-805, 1998.
Ronald M Harris-Warrick and Eve Marder. Modulation of neural networks behavior. Annual Review of Neuroscience, 14(1):39-57, 1991.
[8] Alex H Williams, Albert W Hamood, and Eve Marder. Neuromodulation in small networks. In Encyclope- dia of Computational Neuroscience, pages 2300-2313. Springer, 2022.
[9] Eve Marder. Mechanisms underlying neurotransmitter modulation neuronal circuit. Trends in Neurosciences, 7(2):48-53, 1984.
[10] Eve Marder and Scott Hooper. Neurotransmitter modulation of the stomatogastric ganglion of decapod crustaccans. In Model Neural Networks and Behavior, pages 319-337. Springer, 1985.
[11] Larry F Abbott. Modulation of function and gated learning in a network memory. Proceedings of the National Academy of Sciences, $7(23):9241-9222 1990.
[12] Ann Kennedy, Prabhat S Kunwar, Ling-yun Li, Stefanos Stagkourakis, Daniel A Wagenaar, and David J Anderson. Stimulus-seccffic hypothalamic encoding of persistent defensive state. Nature, 586(7831); 730-734, 2020.
[13] Lia Papadopoulos, Suhyun Jo, Kevin Zumwalt, Michael Wehr, David A McCormick, and Luca Mazzucato. Modulation of metastable ensemble dynamics explains optimal coding moderate arousal in auditory cortex. bioRxiv, April 2024.
[14] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Chea-Brown, Uygar Sümbüi. Cell-type- specific neuromodulation guides synaptic credit assignment in a spiking neural network. Proceedings of the National Academy of Sciences, (18(5)))e22 :)22111)21111 1821118, 2021.
[15] Jake P Stroud, Mason A Porter, Guillaume Hennequin, and Tim P Vogels. Motor primitives in space and time via targeted gain modulation in cortical networks. Nature Neuroscience, 21(12):1744-1783 December 2018
[16] Lyndon Duong, David Lipshutz, David Heeger, Dmitri Chklovskii, and Fero P Simoncelli. Adaptive whitening in neural populations with modulatin interneurons. In International Conference on Machine Learning, pages 8902-8921 8921. PMLR, 2023.
[17] lize Amanda Auzina, Çagatay Yıldız, Sara Magliacane, Matthas Bethge, and Efstratios Gavves. Modulated neural odes. Advances in Neural Information Processing Syxtems, 36:44572-44594 2023.
[18] Laura Naumann, Joram Keijser, and Henning Sprekeler. Invariant neural subspaces maintained by feedback modulation. Elife, 11:77696,, 2022.
[19] Ben Tsuda, Stefan C Pate, Kay M Tye, Hava T Siegelmann, and Terrence J Sejnowski. Neuromodulators generate multiple context-relevant hehaviors in recurrent neural network by shifting activity hypertubes. bioRxiv, 2021.
[20] Nicolas Vecoven, Damien Emst, Antoine Wehenkel, and Guillaume Drion. Introducing neuromodulation in deep neural networks to learn adaptive behaviours. PloS one, 15())=e0277222, 2020.
[21] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017.

---

[22] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
[23] John P Cunningham and Byron M Yu. Dimensionality reduction for large-scale neural recordings. Nature Neuroscience, 17(11):1500 1509, 2014.
[24] Saurabh Vyas, Matthew D Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Annual Review of Neuroscience, 43:249-275, 2020.
[25] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz. Matteo Carandini, and Kenneth Harris. High- dimensional geometry of population responses in visual cortex. Nature, 51177e,571(7666:3311-333 2019.
[26] Dean Pospisil and Jonathan W Pillow, Revisiting the high-dimensional geometry of population responses in visual cortex. bioRxiv, 2024.
[27] Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randommess and structure during learning in rans. Advances in Neural Information Processing Systems, 13:13332-1111 13362, 2020.
[28] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3);609 2at1x6009-62..223 623.c29, August 2018.
[29] Alexis Duhreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamics. Nature Neunscience, 25(6))783-794 2022.
[30] Manuel Beiran, Alexis Dubreuil, Adrian Valente, Francesca Mastrogiuseppe, and Srdjan Ostojic. Shaping dynamics with multiple populations in low-rank recurrent networks. Neural Computation, 33(6):1572- 1615, 05 2021.
[31] Eve Marder and Ronald I. Calabrese. Principles of rhythmic motor pattern generation. Physiological Reviews, 76(3):687-717, 1996.
[32] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 983)1 November 1997.
[33] Yong Yu, Xiaosheng Si, Changhua Hu, and Jiamxun Zhang. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures. Neural Computation, 51(7):1235-1270 07 2019.
[34] Allison Hamilos, Giulia Spedicato, Ye Hong, Fangmiao Sun, Yulong Li, and John A Assad. Slowly evolving dopaminergic activity modulates the moment-to to-moment probability of reward-related self-timed movements. Elife. 10, December 2021.
[35] Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mchrdad Jazayeri, and Srdjan Ostojic. Parametric control of flexible timing through fow-dimensional neural manifolds. Neuron, 111(5):739-75 753, 2023.
[36] Wilhert Zarco, Hugo Merchant, Luis Prado, and Juan Carlos Mendez. Subsecond timing in primates: com- parison of interval production between human subjects and rhesus monkeys. Journal of Neurophysiology, 102(6):311132022
2009.
[37] Akihisa Mita, Hajime Mushiake, Keisetsu Shima, Yoshiya Matsuzaka, and Jun Tanji. Interval time coding by neurons in the presupplementary and supplementary motor areas. Nature Neuroscience, 12(4)5502 507, 2009.
[38] Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience, 22 (2):297-306, 2019.
[39] Lea Duncker, Laura Driscoll, Krishna V Shenoy, Maneesh Sahani, and David Sussillo. Organizing recurrent network dynamies by task-computation to enable continual learning. Advances in Neural Information Processing Systems, 33: 13:14387-1 2020.
[40] Cynthia Chestek, Aaron Batista, Gopal Santhanam, M Byron, Afsheen Afshar, John P Cunningham. Vikash Gilja, Stephen Ryu, Mark M Churchland, and Krishna V Shenoy. Single-neuron stability during repeated reaching in macaque premotor cortex. Journal of Neuroscience, 17744) 2007.
[41] Kyle Aitken and Stefan Mihalas. Neural population dynamics of computing with synaptic modulations. Elife, 12:e83035, 2023.
12

---

[42] Kyle Aitken, Luke Campagnola, Marina E Garrett, Shawn R Olsen, and Stefan Mihalas. Simple synaptic modulations implement diverse novelty computations. Cell Reports, 43(5), 2024.
[43] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-timing-depeedenn plasticity, and theory of three-factor learning rules. Frontiers in Neural Circuits, 9:85, 2016.
[44] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. Biologically- plausible backpropagation through arbitrary timespans via local neuromodulators. Advances in Neural Information Processing Systems, 35:17528 -5:2522--77542, 2022.
13

---

NeurIPS Paper Checklist
Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe that the claims made in the introduction (i.e. that the NM-RNN performs and generalizes better than the LR-RNN) adequately shown by the results.
Guidelines:
The answer means that abstract and introduction do not include the claims made in paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention limitations of our approach in the Discussion section the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (c.g., independence assumptions, noiseless settings, model well- specificaaion, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to penalize honesty concerning limitations.
Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
14

---

Justification:
All proof's and derivations are included in full in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include thcorctical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclosc all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether
Question:
and data are provided or not)?
Answer: [Yes]
Justification: We describe the methods by which we generate synthetic data trials in the main text and Supplementary Material. We also include specific training details and hyperparameters in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments,
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, the contribution is specific model and empirical evaluation, may be necessary to either make possible others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to hosted model (e.g.. in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurlPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproducc that algorithm.
(b) If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
(c) contribution a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the of closed-soue source models, may be that access registered users), hut should be possible for other researchers have some path to repraducing or verifying the results.
model is limited some way (e.g..
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
15

---

Answer: [Yes]
Justification:
The necessary code required to reproduce the trained models is included in the Supple- mentary Material. This code will be made available public repository upon publication.
Guidelines:
The answer NA means that paper does include experiments requiring code.
Please see NeurIPS code and data submission guidelines (https://nipp httphipp..cpppbii///// quides/CodeSuumieee www.immmisiooooooooo..oo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurIPS code Seaeee//ooouSbmiiiiccc essonnoliigg cy) for more details.
reproduce submission guidelines pattps://nipps cc/public/
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
training/test details are included in the Supplementary Material,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: For most of our results, all performance datapoints are presented, climinating the need for error bars. All other statistical information is included accurately in the text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should he explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation the standard enror of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error hars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For cach experiment, docs paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Descriptions of the required compute resources used are included in the Supplementary Material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't Itt make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code cc
Ethics
Answer:
[Yes]
Justification:
We have reviewed the Code of Ethics and believe our paper conforms to it.
Guidelines:
The answer means that authors have not reviewed NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
This paper dealt with modeling synthetic data, with current real-world use cases. While there are immediate societal impacts that anticipate, we believe this kind of work could contribute better understanding human brain, leading improved treatment disorders.
Guidelines:
answer means that is no societal impact of work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considcrations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point the quality generative models could used to generate deepfakes for disinformation. other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
For example,
legitimate to point out that an improvement
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Relcased models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided, For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if' it has changed) should be provided.
18

---

If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New Assets
Question: Are introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA that the paper does not release
Researchers should communicate the details of the dataset/coolemmodel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjeets
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity applica- ble), such as the institution conducting the review.
19


Reviews:
[{"review_id": "H78RsSCeRm", "reviewer": "Reviewer_JBgz", "summary": "This work mimic the synaptic plasticity observed in brain which is driven by neuromodulators to develop neuro-inspired artificial neural networks. It proposes the neuromodulated NM-RNN, it has a neuromodulatory subnetwork that outputs a low-dim output that will scale the synaptic weights of low-rank RNN. It has connection to LSTMs with similar gating mechanisms, or even the dynamics of NM-RNN could be reproduced with a LSTM. The model is better capturing long-term dependencies. The work also demonstrate how this framework is applicable in multitask learning.", "questions": "1. Why choose rank-1 for modulatory subnetwork?\n\n2. Why the training dynamics (MSE loss) for all models are noisy and fluctuated, any optimization techniques (regularization, normalization, scheduler might help with the training dynamics), how to guarantee the convergence of the model?\n\n3. Any computational efficiency that NM-RNN might bring in?", "limitations": "No potential negative societal impact. The limited scale of the network has been discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Methods**\n\nThis work focused on an important question to incorporate nonstationarity observed in biological network into artificial network. It implemented the potential mechanisms that synaptic plasticity controlled by the neural modulators. The subnetwork to outputs modulatory inputs for the scaling the low-rank RNN's weights is a novel design.\n\n**Evaluation**\n\nThe NM-RNN has been benchmarked with multiple classic artificial neural networks (RNN, LSTM) on multiple tasks including measure-wait-go, multitask learning, element finder task to demonstrate its capabilities of capturing long-term dependencies.", "weaknesses": "**Method**\n\n1. The design choice of rank-1 in modulatory subnetwork should be described, more hyperparameters should be explored.\n\n**Baselines**\n\n1. The proposed NM-RNN models has multiple connections with LSTM, the novelty is therefore limited. Meanwhile, its performance is not comparable to LSTM as shown in Fig5 B. \n2. The baselines are limited, i.e. comparisons with transformers which also has input-dependent attention weights which also model the nonstationarity, and relevant to neural plasticity.\n\n***Evaluation**\n\n1. The tasks and  LSTM is not evaluated in measure-wait-go, multitask learning tasks. Adding more tasks to demonstrate the effectiveness of NM-RNN to captures long-term dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback. We have addressed some of your comments from the Weaknesses section in the general rebuttal (in particular, concerns about Evaluation and Baselines). We would also like to clarify why we did not initially provide an LSTM baseline for the Measure-Wait-Go and multitask settings. The goal of this project is to offer a new model relevant to the field of computational neuroscience. Low-rank RNNs are commonly used tools in computational neuroscience due to their low-dimensional internal dynamics. We augmented the low-rank RNN with an interpretable modulatory parameter, and showed the comparative effectiveness of low-rank RNNs and NM-RNNs on tasks related to neuroscience. We also showed that NM-RNNs help to recover some of the performance gains of LSTMs. However, LSTMs are not commonly used as “brain-like” models when studying neural population dynamics. One of our contributions was to show the similarities between our more “brain-like” NM-RNN and the popular LSTM. Like LSTMs, transformers are not typically used to model neural population dynamics, so we did not include comparisons to these models.\n\nTo address your questions:\n\nWhy choose rank-1 for modulatory subnetwork? \n\nTo clarify, the modulatory subnetwork is full-rank and the output-generating subnetwork is low-rank. In our study of the Measure-Wait-Go task we chose to present results with rank-1 and rank-3 NM-RNNs. We analyzed rank-1 networks due to their theoretical tractability, and rank-3 networks in order to compare to existing work which used rank-3 low-rank RNNs on this task [1]. For the multitask setting, we chose rank-3 NM-RNNs by sweeping over ranks and determining the rank required to consistently train on the first three tasks. For the element-finder setting we tried many combinations of rank-size parameters to compare to the LSTM. \n\nWhy the training dynamics (MSE loss) for all models are noisy and fluctuated, any optimization techniques (regularization, normalization, scheduler might help with the training dynamics), how to guarantee the convergence of the model? \n\nWe agree that the training dynamics for the Element-Finder task (EFT) are quite noisy, and would likely become smoother with the use of regularization techniques. However, our main point with this figure was to show general trends, i.e. vanilla RNNs struggle to solve EFT while some LSTMs and NM-RNNs are able to learn it consistently. In addition, we did not show the training dynamics for any of the other tasks, but can share empirically that they were much smoother than in the EFT, suggesting that the noisy learning may also be due to the relative complexity of this task.\n\nAny computational efficiency that NM-RNN might bring in? \n\nAll models are run sequentially, so while there is a slight difference in computational efficiency between each individual step, the scaling laws likely don’t change. In general, you raise an interesting point that the matrix multiplication cost for running a step of a low-rank RNN is smaller than that of a vanilla RNN. So, low-rank RNNs and NM-RNNs will be faster to evaluate compared to a vanilla RNN with similar neuron count. We will add a discussion of computational implications to the final version.\n\nCitations \n[1] Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mehrdad Jazayeri, and Srdjan Ostojic. Parametric control of flexible timing through low-dimensional neural manifolds. Neuron, 111(5):739–753, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvUF0600vB", "reviewer": "Reviewer_Eb4N", "summary": "This work studies the effects of synaptic gain scaling, a neuromodulatory mechanism, on the performance of task-trained low-rank RNNs – which have been used to understand the dynamics and other finer details of neural computation. Specifically, it introduces a simple time-varying neuromodulatory mechanism implemented as a hypernetwork to modulate the weights of low-rank RNNs trained on multiple tasks. The authors draw connections between the proposed model and LSTMs by theoretically demonstrating that the neuromodulation provides for a form of gating. Finally, the authors show that the proposed model outperforms low-rank RNNs and is comparable to LSTMs in multi-task settings and tasks involving long-term dependencies.", "questions": "See the Weaknesses section. Some additional questions:\n1. Have the authors tried training the neuromodulatory subnetwork alone while treating the task subnetwork as a reservoir? This could potentially provide for a parameter-efficient multi-task learning framework (loosely related, see the recent paper by Williams et al., 2024 [3]).\n2. Could the authors clarify what the three different NM-RNN curves are in Fig. 5C? If they are different seeds or configurations, this should be specified.\n3. Have the authors tried using other activation functions in calculating $\\mathbf{s}(\\mathbf{z}(t))$, i.e., not restricting the neuromodulatory signal to values between 0 and 1 (and allowing negative values)? Could this lead to improved performance?\n\n**References:**\n1. Tsuda et al. \"Neuromodulators generate multiple context-relevant behaviors in a recurrent neural network by shifting activity hypertubes.\" bioRxiv (2021): 2021-05.\n2. Liu et al. \"Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators.\" Advances in neural information processing systems 35 (2022): 17528-17542.\n3. Williams et al. \"Expressivity of Neural Networks with Random Weights and Learned Biases.\" arXiv preprint arXiv:2407.00957 (2024).\n\n**Rebuttal update:** Confidence increased from 3 to 4. I am in favor of accepting the paper.", "limitations": "The limitations have been adequately discussed in the paper (specifically, biological realism, comparison to neural data, and scalability).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and the overall presentation of the paper is good – related work is adequately described, and the model formulation is clear and simple but effective.\n2. The authors theoretically study the links between the proposed NM-RNNs and LSTMs and show that, under certain assumptions, NM-RNNs learn gating mechanisms similar to LSTMs leading to their improved performance over LR-RNNs on certain tasks. To my knowledge, this theoretical analysis is novel and sound.\n3. The authors perform several clearly motivated numerical experiments to analyze the computational implications of the proposed model. Particularly, they demonstrate that their model outperforms ordinary LR-RNNs in multi-task settings and capturing long-term dependencies.", "weaknesses": "1. The model is limited in its bio-realism, so it is unclear how well this would match to computations performed in the brain. Specifically, there are no testable predictions for this model of neuromodulation and no comparison with neural data (as acknowledged by the authors in the limitations), so this limits the significance of the work.\n2. The model does not seem to perform as well as LSTMs in the long-term dependencies task (the loss attained by LSTMs is far lower). Furthermore, the authors do not show comparisons with LSTMs and vanilla RNNs for the timing task and the multi-task setting. It would be important to show how the NM-RNN performs in comparison to at least the LSTMs as well in these settings.\n3. On a related note to the previous point, the authors have not compared their model to existing implementations of neuromodulation-infused RNNs. Given the similarity of this work to Tsuda et al., 2021 [1] (which also seems to explain observations in neural data from Drosophila), is it possible to compare the NM-RNN to this model, and perhaps other approaches such as those introduced by Liu et al., 2022 [2]?\n4. Most importantly, it seems like the scalability of this model is not well-studied and this is also related to the fact that the tasks considered are very simplistic. While these simple tasks allow for some interpretability, I think it would also be important to benchmark the NM-RNN on more complex tasks. For example, studying how it adapts under perturbation in a reaching/control task – this would be a more complex task and could also help understand the links between neuromodulation and learning. In general, moving beyond toy-ish tasks would strongly improve the experimental section of the paper.\n\n(References provided in Questions section.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback. We have addressed some of your comments in the general rebuttal, in particular the second point under Weaknesses (regarding performance comparison to LSTMs and vanilla RNNs). We would also like to respond to the additional weaknesses noted. \n\nWhile we currently have no comparison with neural data, we believe this model can offer testable predictions of how diseases involving neuromodulator deficiencies and dysfunction impact movement and timing. As RNNs have been used to generate testable predictions of neural function, we believe our model offers a biologically meaningful extension by scaling synaptic weights as biological neuromodulation has been shown to do. \n\nWe did not compare our work with the prior methods mentioned in the Related Work section because of a few key differences in motivation and implementation. The work of Tsuda et al., while similar, requires prespecification of (1) which neurons are neuromodulated and (2) the constant value that the neuromodulation takes. Our paradigm allows both of these parameters to be learned by the model. It was unclear how to set the aforementioned parameters in the Tsuda model in order to make a comparison. While Liu et al. also studied the impact of neuromodulation in RNN models, their work focused primarily on the impact of neuromodulation on training and not during task performance. Due to the difference in motivation, we did not compare to their model.\n\nWe agree that training our model on more complex tasks will be an important step forward. Since vanilla RNNs are used to model more complex tasks and combinations of tasks [Driscoll], we believe our model should also be amenable to training in these situations. We appreciate the suggestion for studying reaching/control perturbation tasks and agree that training the NM-RNN in this task setup could help provide valuable insights into the relationship between neuromodulation and adaptation.\n\nTo answer your questions:\n\nHave the authors tried training the neuromodulatory subnetwork alone while treating the task subnetwork as a reservoir? This could potentially provide for a parameter-efficient multi-task learning framework (loosely related, see the recent paper by Williams et al., 2024 [3]). \n\nWe like this idea and think it would be an interesting extension for the multitask section of the paper. While we did not try random reservoir dynamics as in Williams et al., we did analyze the effect of retraining only the neuromodulatory signal while leaving the low-rank dynamical components fixed when learning a new task (paper section 5). These results could be viewed as a proof-of-concept that the NM-RNN can effectively reuse existing dynamics when learning new tasks.\n\nCould the authors clarify what the three different NM-RNN curves are in Fig. 5C? If they are different seeds or configurations, this should be specified. \n\nWe apologize for the confusion—these are different hyperparameter settings for the NM-RNN, parameter-matched to the N=10 LSTM. We will update the caption of this figure in the final version to indicate the exact hyperparameters used in each curve. For reference the settings are (M=5, N=18, R=8), (M=5, N=13, R=12), and (M=10, N=12, R=7). Our overall goal in this figure was to show that LSTMs and NM-RNNs are able to solve the task, while vanilla RNNs cannot train on it.\n\nHave the authors tried using other activation functions in calculating z(t), i.e., not restricting the neuromodulatory signal to values between 0 and 1 (and allowing negative values)? Could this lead to improved performance? \n\nWhile this may lead to improved performance, we believe it would negatively impact the biological plausibility of the modulatory signal. We capped the neuromodulatory signal at 1 to model a saturating effect of neuromodulators on downstream synaptic strength—the assumption being that synaptic strength cannot increase indefinitely. Similarly, we did not allow negative saturation, instead setting the floor value of the neuromodulator to 0 to indicate complete silencing of synaptic connections and preserve the sign of the connection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EU8IjCDhXn", "reviewer": "Reviewer_znSr", "summary": "The authors introduce and implement a novel biologically-inspired variant of standard recurrent neural networks (RNNs), which they evaluate on a number of tasks that require dynamics to generalize across task conditions (Measure-Wait-Go), switch between tasks/task contexts (four-task set from Duncker et al.), and to capture long-term dependencies (Element Finder Task). Their variant, the neuromodulated RNN (NM-RNN), derives from gated coupling between two networks, where the so-called neuromodulation network dynamically scales the recurrent weights of the output-generating network. The authors provide mathematical intuitions about NM-RNNs and how they relate to LSTMs, compare their performance on above-mentioned tasks, and analyse and visualize the networks' learned dynamics.", "questions": "Conceptual:\n\nCan you clearly define what exactly is meant by **structured flexibility**? Is flexibility in standard RNNs unstructured? Specifically, the EFT analysis and visualization convincingly demonstrate structure in the sense that computations are divided between neuromodulatory and output-generating network; in the Memory/Delay/Anti/Pro, this structure is imposed; but for the MWG task, the authors do not seem to analyse structure. \n\nWhat have we learned about the computational implications of synaptic gain scaling?\n\n**EFT**\n\nHow do the authors predict the performance of the model will change with increasing T? How did the inputs at test time differ from the inputs during training? Is the zeroing out of s(t) components a signature of overparameterization relative to task complexity? \n\n\nTechnical:\n\nFig. 2A: The corresponding text talks about different components k, but it is not clear what is shown in the figure (a single component across time, or different components? How do changes in s(t) come about?)\n\nCould this be a plug-in replacement for GRUs/LSTMs? If so, how would the model perform standard sequence modeling tasks, such as sequence classification?\n\nThe authors don't mention the vanishing gradient issue that RNNs severely suffer from. Did the not have this problem?\n\nFig 3. E: please offer more interpretation for what is shown.\n\nMinor:\n\nln. 104: \"Likewise, artificial neural networks trained to solve tasks that mimic those found in neural experi-\nments also often exhibit low-rank structure.\" Please provide a reference.\n\nFig 4 A: This panel is not clear. What is shown? Please add labels to all traces and to the axes, and make clear how the subpanels’ axes relate to each other. \n\nThe Memory/Anti task is not readily understandable from the description and visualization in the paper. Please explain this better to make the paper self-contained.\n\nFig 5C what are the squiggly lines? Fig E not clear where trajectories begin and end, black line not visible. Fig F similarly hard to see/interpret.\n\nEq. 4: Please specify what \\sigma is. \n\nFig 3 references mixed up in ln.208.\n\nln. 267: \"Curiously, the positive/negative relationship flips for θ ∈ (π, 2π), likely relating to the symmetric structure of sine and cosine.\" where can we see this?", "limitations": "The authors adequately address the limitations of the work.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Originality**: To my knowledge, the architecture is novel.\n**Quality**: The paper is very well written and analyses are thorough. The different tasks networks are evaluated on are well-motivated. \n**Clarity**: While the paper is well written, clarity could improve if the motivation was spelled out more clearly and if figures were self-contained (via better legends and labels).\n**Significance**: The work is an important contribution, but the real-world significance will depend on whether the suggested RNN architecture can be scaled, or used to generate insights about biology (depending on what the author's main motivation is)", "weaknesses": "I find the main weakness of the paper to be a lack of clarity in motivation and terminology. I gather the main motivation of the paper is to \"bridge the gap between [...] highly biologically-accurate models and general network models (i.e. RNNs) of neuronal activity by adding a biologically-motivated form of structured flexibility\" (lines 64-65) - in order to better be able to study flexibility and generalization capabilities observed in biological neural networks.The latter is not very clear from the paper. \n\nThe link to neuromodulation features prominently in the Introduction, but should be spelled out more clearly, especially how exactly and strongly it links to the suggested architecture (or it should be clearly stated that the inspiration is only of loose nature). The suggested architecture effectively performs synaptic gain scaling, which is only one of many effects of neuromodulation. Streamlining terminology here might help to improve the clarity of the paper and to guide the reader. As an example, the authors mention dopamine as \"a well-known example [...] implicated in motor deficits resulting from Parkinson's disease\", but that statement needs to be better connected to the presented study and why it motivates the study. \n\nBoth Motivation and Discussion would benefit from an explicit positioning of the approach within the field of NeuroAI: how much does this work contribute to bringing artificial NNs closer to biological NNs, and how much does it help in using ANNs to study BNNs?\n\nIn a similar vein, authors may want to discuss their work in relation to:\n - Auzina, I. A., Yıldız, Ç., Magliacane, S., Bethge, M., & Gavves, E. (2023). Modulated Neural ODEs. Proceedings of the 37th Conference on Neural Information Processing Systems. Retrieved from https://github.com/IlzeAmandaA/MoNODE.\n- Naumann, L. B., Keijser, J., & Sprekeler, H. (2022). Invariant neural subspaces maintained by feedback modulation. ELife, 11, 76096. https://doi.org/10.7554/eLife.76096", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough comments. Indeed, our motivation was to create a model somewhere between RNNs and biologically-accurate biophysical models. Specifically, we identified neuromodulation as a feature of biological networks that is not often modeled in RNNs. The goal of this paper is to study the impact of synaptic scaling, one specific effect of neuromodulation, on the performance and generalization of RNN models. We have discussed our motivation further in the General Rebuttal and will make this positioning more clear in the final version.\n\nThank you for suggesting Auzina et al. and Naumann et al., we will add these papers to our related work section to highlight the use of modulatory signals in non-RNN models.\n\nTo address your questions: \n\nConceptual: \n\nBy structured flexibility, we mean the introduction of additional parameters (i.e. flexibility) to a model, with these parameters having a clear motivation and interpretation (i.e. structure). In the case of our NM-RNN, the structured flexibility comes from adding a flexible neuromodulatory signal which impacts RNN dynamics in a constrained way (by scaling synaptic weights). In the EFT and multitask settings, we show that this allows the network to distribute different parts of the task across the subnetworks. In the MWG task, we show that ablating each component of the neuromodulatory signal leads to performance deficits (paper fig. 3F). In particular, ablating s_3(t) destroys the ability of the network to produce output ramps of different slopes, implying that the neuromodulatory subnetwork is involved in controlling the output interval length. In all three ablations, the general ramping shape of the output is preserved, implying that this behavior is stored in the dynamics of the output-generating network.\n\nOverall, we learned that synaptic gain scaling can improve generalization capabilities of low-rank RNNs and make them more like LSTMs. More generally, we offer this model as an alternative to commonly-used low-rank RNNs, as a tool to model biological task completion with neuromodulation.\n\nEFT:\n\nEach of our models was first trained in a continual learning regime in which each step of gradient descent was performed using a randomly generated batch of input sequences. Then, at test time, test input sequences were again sampled i.i.d. from the distribution of all valid EFT sequences. (It should be noted that for main paper figs. 5C, D, and F, we randomly generated test inputs while holding the query index and/or target element value fixed.)\n\nAs the sequence length T increases while keeping the NM-RNN model size, number of training batches, and training batch size fixed, we predict that model performance will degrade. The set of possible EFT input sequences grows exponentially in T, making it challenging for the model to solve the task within the same number of training iterations; indeed, this is what we observed in our preliminary simulations. However, we expect that model performance would improve if the number of training iterations and/or the model size is suitably increased.\n\nTechnical:\n\nFig. 2A: The corresponding text talks about different components k, but it is not clear what is shown in the figure (a single component across time, or different components? How do changes in s(t) come about?)\n\nThis figure is meant as an illustrative example of how the neuromodulatory signal s(t) can impact dynamical timescales. In this example, s(t) is a 1-dimensional signal so k=1. The changes in s(t) are artificially imposed for illustrative purposes, to show how the decay rate of w(t) is modulated by the value of s(t). We will clarify this in the final version.\n\nCould this be a plug-in replacement for GRUs/LSTMs? If so, how would the model perform standard sequence modeling tasks, such as sequence classification?\n\nWe are not arguing to replace GRUs/LSTMs in standard ML workflows. However, we show that they do share some capabilities, achieving LSTM-like performance on some tasks.\n\nThe authors don't mention the vanishing gradient issue that RNNs severely suffer from. Did the not have this problem?\n\nThis issue did not arise for the tasks that we trained on. To speculate, perhaps the NM-RNN does not encounter this issue as often due to its similarities to the LSTM. This could also be a potential reason why the RNNs are failing at the EFT. \n\nFig 3E: please offer more interpretation for what is shown.\n\nIn fig. 3E we show the 3-channel neuromodulatory signal s(t) during the MWG task. On the left, the signals are aligned to when networks receive the measure cue. On the right, the signals are aligned to when networks receive the wait and go cues. The left figure shows how the s(t) responds to the measure cue and evolves throughout the task. The right panel shows how s(t) readies responses for the different interval lengths. In particular, we can see that between the wait and go cues, s_1(t) separates shorter intervals from longer ones, setting up initial dynamics for the go period when the ramp is generated. The third component s_3(t) seems to be involved with ending the output ramp, since it saturates first for the shorter intervals and then the longer ones.\n\nMinor:\n\nln 104: we will add a citation to Schuessler et al. “The interplay between randomness and structure during learning in RNNs”, NeurIPS 2020\n\nFig. 4A: we have updated this figure in the PDF attached to the general rebuttal. Please let us know if additional adjustments would help to clarify.\n\nFig. 5: The squiggly lines are the actual loss curves, while the bold lines show a moving average to indicate overall trends. We will add improved explanations and better-contrast lines to E and F.\n\nEq. 4: \\sigma is the sigmoid nonlinearity, we will clarify this in the final version.\n\nFig. 3/ln. 208: thank you for catching this, we will amend it in the final version.\n\nln 267: we have added these plots to the general rebuttal PDF and will include them in the final paper fig. 4D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRUsWgvDiw", "reviewer": "Reviewer_rG4S", "summary": "The paper proposes an RNN architecture that include synaptic modulation, motivated by neuromodulatory factors in the brain. In essence, the paper shows it is possible to linearly influence the connectivity matrix of a low-rank RNN by scaling it with the output of smaller RNN, the latter nominally describing neuromodulation. The achieved composition of networks outperforms classical RNNs on tasks with timing and history dependence. The presence of the neuromodulatory signal also allow for multitask learning and reuse of previously learnt dynamics to learn a new task. The paper also compares the performance of the NM-RNN with those of a LSTM, both trained on a digit recall task, where LSTMs are known to outperform RNN. Instead, the result show the added flexibility of NM-RNN.", "questions": "The architecture seems to require an a prior specification of the maximum rank of the connectivity matrix. From an application standpoint, how would this be determined?\n\nIs the proposed method amenable to batch training? Can the authors characterize the trainability/computational implementation considerations of the proposed architecture, relative to say an LSTM? Certainly, there would seem to be more overhead relative to a vanilla RNN.", "limitations": "The discussion contains several limitations and future work directions, which are largely appropriate and appreciated.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a nicely written paper, clear in motivation and generally well-executed. The idea of dynamically scaling the weights of a RNN is a well-reasoned concept, if not entirely novel (as per the prior literature on hypernetworks and synaptic scaling more generally). Adding this form of flexibility could be a useful concept to generalize neural network architectures, and it is certainly an interesting line of study from the perspective of theoretical neuroscience.", "weaknesses": "While I generally enjoyed the paper, there are a few issues that impact my initial score.\n\nThe domain of impact of the paper is mostly in theoretical neuroscience, since it is not clear from the results that the absolute performance of the NM-RNNs actually exceeds popular constructs such as the LSTM (e.g., F5B). Because a lot of the paper is spent discussing the parallels of the NM-RNN gating mechanism and the LSTM, this becomes important from a mechanistic perspective. Can we gain insight into why there is a plateau in performance, despite the argued parallels in architecture? This could be useful from a fundamental ML perspective.\n\nRelated to the above, it would be good to understand actual performance of these networks, in addition to the loss (since there is not always a 1-1 correspondence between the loss and performance).\n\nIt is also important to discuss the computational implications of the architecture from a trainability standpoint (see Question below).\n\nFrom the theoretical neuroscience standpoint, the assumption of neuromodulation acting in this way (weighting low rank components) seems quite abstract and strong. I can understand the argument that networks are low rank, but the idea that modulation effectively selects or weights these components would seemingly require a high degree of precision and some ‘knowledge’ of these components at the level of the modulation generation (I note some discussion of this in the Future Work). Is this realistic, and/or are there ways that the developed models could be substantiated in actual experiments? Are there other ways of neuromodulation entering the model that would be similar effective, or does it really need to be as this sum of low rank components?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful comments. We have addressed some shared concerns regarding performance comparisons to LSTMs and vanilla RNNs in the general rebuttal. To respond to your comments regarding weaknesses of the paper:\n\nIn general, we don’t believe the NM-RNN will outperform the LSTM. As you mentioned, there is a plateau in performance despite similarities in structure. While our NM-RNN captures some aspects of the LSTM via its synaptic weight scaling, other key features of the LSTM are not implemented. In particular, the input and output weights of the NM-RNN are held constant. Perhaps modulating these weights would help recover more LSTM-like performance. However, our main goal was to offer this model as an alternative to commonly-used low-rank RNNs, as a tool to model biological task completion with neuromodulation. In contrast, LSTMs are not commonly used as “brain-like” models of neural dynamics. \n\nWe give example outputs of each model on the Measure-Wait-Go task to highlight specific performance (paper fig. 3D and rebuttal fig. B). For the multitask setting and EFT, example outputs are noisier and are less visibly distinguishable between model types, so we did not visualize them, instead choosing to summarize performance via % correct and loss metrics.\n\nWe chose to have neuromodulation differentially impact low-rank components to mimic the selective way neuromodulators act on particular synapses [1], and to situate our model alongside the widely used low-rank RNN. However, this choice is certainly not the only way to model neuromodulation in an RNN—one alternative might be to implement neuromodulatory weights on a mixture of sparse weight matrices, instead of low-rank components. As another example, in Tsuda et al. the authors choose particular subsets of weights on which to apply neuromodulation. However, these subsets are prespecified instead of learned during training.\n\nTo answer your questions:\n\nThe architecture seems to require a prior specification of the maximum rank of the connectivity matrix. From an application standpoint, how would this be determined?\n\nThe rank of the connectivity matrix is indeed a hyperparameter that must be determined before training. In general, we performed a parameter sweep over various numbers of ranks to determine the lowest rank that achieved satisfactory performance. For the MWG task, we chose rank-1 networks due to their theoretical tractability (and because even at this lowest possible rank, the NM-RNN can solve the task). However, to compare to the low-rank RNN we chose rank-3 NM-RNNs, to match prior work which used rank-3 RNNs to model this task [2].\n\nIs the proposed method amenable to batch training? Can the authors characterize the trainability/computational implementation considerations of the proposed architecture, relative to say an LSTM? Certainly, there would seem to be more overhead relative to a vanilla RNN.\n\nThis method is amenable to batch training, which we used in the multitask and EFT settings. For the MWG task, the space of possible samples was small enough to do full batch training. In terms of computational implementation, all models must be run sequentially, so while there are slight differences in training time the overall scaling rules likely do not change. We will add a discussion of computational implications to the final version.\n\nCitations \n\n[1] Peter Dayan. Twenty-five lessons from computational neuromodulation. Neuron, 76(1):240–256, 2012.\n\n[2] Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mehrdad Jazayeri, and Srdjan Ostojic. Parametric control of flexible timing through low-dimensional neural manifolds. Neuron, 111(5):739–753, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would first like to thank all of the reviewers for their insightful and thorough comments on our paper. We have carefully considered your feedback and appreciate your help in both strengthening our current paper’s claims and providing future directions for us to consider.

In this general rebuttal, we would like to provide answers to some common concerns as well as highlight the rebuttal figures shown in the attached PDF. First, a request shared among multiple reviewers was to compare the performance of our NM-RNN to LSTMs and vanilla RNNs in the Measure-Wait-Go (MWG) task and multitask setting. The comparisons among all four models (parameter-matched low-rank RNNs (R=3, N=106), NM-RNNs (R=3, M=5, N=100), vanilla RNNs (N=31), and LSTMs (N=15)) on the MWG task are shown in fig. A&B. In fig. A we’ve plotted the loss for 10 instances of each model type, both on linear and log scales. On trained intervals, the LSTM achieves the lowest median L2 loss due to its ability to accurately shape the output ramps (fig. B). However, on the extrapolated intervals the NM-RNN achieves the lowest median loss out of all four model types. Looking at the example outputs in fig. B, we see that this is because the NM-RNN better generalizes the slope of the output ramp to extrapolated intervals, compared to the LSTM. Overall, we see that our NM-RNN enjoys enough complexity to both train and generalize well on the task, without suffering from generalization performance losses like the LSTM.

We have also trained all four model types (parameter-matched low-rank RNNs (R=3, N=100), NM-RNNs (R=3, M=20, N=100), vanilla RNNs (N=18), and LSTMs (N=8)) in the multitask setting, on both the three initial and one retrained tasks. For retraining the LSTM and vanilla RNN, we froze all recurrent and output weights and retrained only the weights which directly receive the input. The percent correct metric for 10 instances of each model is shown in fig. D. On the trained tasks, the NM-RNNs, vanilla RNNs, and LSTMs are able to train to high accuracy. On the retrained task, these three models still show similar high performance. While there is an outlier NM-RNN (also shown in the paper), the majority of the NM-RNNs achieve similar retraining performance to both LSTMs and vanilla RNNs.

We would also like to use this general rebuttal to clarify our goals. In particular, our motivation was not to create a model that would outperform the LSTM on most tasks; rather, we aimed to create a version of an RNN which includes a biologically motivated implementation of synaptic gain scaling. One of the many known effects of neuromodulation on neural computation is synaptic scaling. This work aims to highlight how traditional low-rank RNN models of task completion may be overlooking synaptic scaling, a relevant biological factor that we show aids in performance and generalization. In developing this model we discovered similarities between our low-rank modulation and gating in LSTMs. This prompted us to further explore the relationship between the NM-RNN and LSTM. However, LSTMs are not commonly used as “brain-like” models in computational neuroscience, so we did not initially compare their performance on the neuroscience-relevant tasks. We believe this model offers a unique framework by which to study and hypothesize the impact of neuromodulation on task performance and generalization in biological networks. For example, we could use NM-RNNs trained on timing tasks to predict the effect of decreased dopamine.

Rebuttal figs. C and E are both in response to Reviewer znSr. Fig. C is an updated version of paper fig. 4A. We aimed to clarify the distinctions between the four tasks in the multitask setting by adding highlights to the figure indicating separate task labels, inputs, and outputs, as well as verbal description of task goals and boxes around the separate tasks. Fig. E shows the flip in sign of PCs 4&5 of our example multitask NM-RNN. The network distinguishes between Pro/Anti versions of the same task by flipping the sign of PC5 as the measured angle crosses $\pi$.

Meta-review: This paper proposes a low-rank recurrent neural network model with neuromodulation that modulates the strength of the low-rank factors, and it develops analytical connections with LSTMs. Overall, the reviewers agreed this paper presents a novel solution to an important problem in theoretical neuroscience, while establishing theoretically sound connections with LSTMs. Therefore, I recommend acceptance.
Decision: Accept (poster)