Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators
Authors: Yuhan Helena Liu
Stephen Smith
Stefan Mihalas
Eric Todd SheaBrown
Uygar Sümbül
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computational neuroscience
learning and plasticity
biologically plausible learning
neuromodulation
cell types
neural circuit mechanisms
temporal credit assignment
recurrent neural networks
Hebbian learning

Abstract: The spectacular successes of recurrent neural network models where key parameters are adjusted via backpropagation-based gradient descent have inspired much thought as to how biological neuronal networks might solve the corresponding synaptic credit assignment problem [1, 2, 3]. There is so far little agreement, however, as to how biological networks could implement the necessary backpropagation through time, given widely recognized constraints of biological synaptic network signaling architectures. Here, we propose that extra-synaptic diffusion of local neuromodulators such as neuropeptides may afford an effective mode of backpropagation lying within the bounds of biological plausibility. Going beyond existing temporal truncation-based gradient approximations [4, 5, 6], our approximate gradient-based update rule, ModProp, propagates credit information through arbitrary time steps. ModProp suggests that modulatory signals can act on receiving cells by convolving their eligibility traces via causal, time-invariant and synapse-type-specific filter taps. Our mathematical analysis of ModProp learning, together with simulation results on benchmark temporal tasks, demonstrate the advantage of ModProp over existing biologically-plausible temporal credit assignment rules. These results suggest a potential neuronal mechanism for signaling credit information related to recurrent interactions over a longer time horizon. Finally, we derive an in-silico implementation of ModProp that could serve as a low-complexity and causal alternative to backpropagation through time. 

Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators
Yuhan Helena Li3", Stephen S Stefan Mihalas 1.2.3, Eric Shea-Brown!... 1.2.3 and Uygar Sümbül
Department of Applied Mathematics, University of Washington, Seattle, WA, USA Allen Institute for Brain Science, 615 Westlake Ave N, Seattle WA, USA
*Computational Neuroscience Center, University of Washington, Seattle, WA, USA
DDpaarmeet of Molecular and Cellular Physiology. Stanford University, Stanford CA, USA "Correspondence: hyliu24 @ww.edu uygars aagars@alleninsituuee.oo
Abstract
The spectacular successes of recurrent neural network models where key parame- ters adjusted via backpropagation-ben bbaadd gradient descent have inspired much thought as to how biological neuronal networks might solve the corresponding synaptic credit assignment problem [1.3]. There is far little agreement, however, as to how biological networks could implement the necessary backpropagation through time, given widely recognized constraints of biological synaptic network signaling architectures. Here, we propose that extra-synaptic diffusion of local neuromodulators such neuropeptides may afford an effective mode of back- propagation lying within the bounds of biological plausibility. Going beyond existing temporal truncation--an based gradient approximations 446) our approxi- mate gradient-basee update rule, ModProp, propagates credit information through arbitrary time steps. ModProp suggests that modulatory signals can act on receiving cells by convolving their eligibility traces via causal, time-invariant and synapse- sype-specific filter taps. Our mathematical analysis of ModProp learning, together with simulation results on benchmark temporal tasks, demonstrate the advantage of ModProp over existing biologically-plann -lluusble temporal credit assignment rules. These results suggest potential neuronal mechanism for signaling credit informa- tion related recurrent interactions over longer time horizon. Finally, we derive in-silico implementation of ModProp that could serve as low-complexity and causal alternative to backpropagation through time.
Introduction
Recurrent connectivity is hallmark of neuronal circuits. While this feature enables rich and flexible computation, mechanisms enabling efficient task learning in large circuits remain a central problem in neuroscience and artificial intelligence research. Fundamentally, the problem stems from the fact that potentially all history of all neurons, including synaptically far away ones, can affect neuronal activity and contribute task output. Motivated by the success of gradient descent learning, several biological learning models approximate the exact gradient in recurrent neural networks using known biological processes and gain insights into computational principles of how the brain might learn 6
By ignoring dependencies beyond few recurrent steps - thus severely truncating the gradient computational graph these existing models succeed in representing their approximate gradient- based update rule as a combination of terms that resemble known synaptic physiological processes: "eligibility trace", which maintains fading memory of coincidental activation of presynaptic and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

postsynaptic neurons 11 combined with third "modulatory" factor top-down learning or "reward" signals [52 for which dopamine a prominent candidate [[6]. Despite the impressive performance of such approximations on a variety of tasks, truncating relevant credit information results in a significant performance gap compared to algorithms using exact gradient information; ceckpropagttio-througee
(BPTT) and real time recurrent learning (RTRL.) 6 17].
How might neural circuits account for long-term recurrent interactions assign credit (or blame) to neuronal firing that happened arbitrary steps before the presentation of reward? Given the rich repertoire of dynamical and signaling elements in the brain, one avenue could be examine biological processes that have been underexplored in existing models. Dopamine whose cellular actions are exerted by activation of G protein-couples receptors (GPCRs), which can greatly impact STDP is not the only neuromodulator involved in learning [1 More recently, transcriptomic studies have uncovered strong evidence for many other neuromodulatory pathways throughout the brain that also act via the activation of GPCRs, leading to similar downstream actions as dopamine [22 23 This suggests that, similar to dopamine, they could also play a role in shaping synaptic credit assignment. A conspicuous family within these pathways is neuropeptide signaling because peptidergic genes are densely and abundantly expressed in the forebrains of divergent species, including the human, in cell-type-specifie fashion 24], suggesting widespread interaction between synaptic and peptidergic modulatory networks for synaptic credit assignment [ 25]. Moreover, intra-cortical expression allows neuropeptides to potentially carry information local to the cortical network, cell type specificity enables sculpted signals for different recipient cells, and their diffusive nature could enable communication between neurons that are not synaptically connected. Finally, peptidergic signals have timescales much longer than the time scales for axonal propagation of action potentials synaptic delays 26 Taken together, these properties make cell-type-spechl local neuromodulation seem promising for propagating credit signal over multiple recurrent steps. Developing explicit computational principles of how these local modulatory elements could propagate credit signal over arbitrary recurrent steps could advance our understanding of biological learning and may inspire more efficient low-complexity bio-inspired learning rules.
Motivated by the question above as well as shortcomings of gradient approximations based on severe temporal truncations, we investigate how biological credit signals could be propagated through arbitrary recurrent steps via widespread cell-type-specifie neuropeptidergic signaling [23 25). While
Ref. [6] recently introduced framework exploiting properties of neuropeptidergic signaling for
temporal credit assignment, similar to [4) and [5), their approach performs severe temporal truncation of the error gradient and does not consider credit propagation beyond disynaptic connections. Our main contributions are summarized as follows:
We derive theory that provides mechanism and intuition for the effectiveness of synapse- type-specific modulatory backpropagation (through time) weights (Theorem
We develop model predicting how modulatory signaling could be the basis for biolog- ical backpropagation through arbitrary time spans (Figures | and 2). Unlike temporal truncation- based approximations 6 our model enables each neuron to receive filtered
(rather than precise) credit signals regarding its contribution to the outcome via neuromodu- lation.
We demonstrate the effectiveness of modulatory signaling via than synapse-specific, modulatory weights on learning tasks that involve temporal credit assignment (Figure 4 In particular, we demonstrate online learning setting where weights are updated causally and in real-time (Figure
rather
We also derive low-complexity in silico implementation of our algorithm suitable for online learning (Proposition
2 Related works
Neuromodulatory factors in synaptic plasticity: One of the most fundamental learning rules, Hebbian plasticity, attributes lasting changes in synaptic strength and memory formation to correla- tions of spike timing between particular presynaptic and postsynaptic neurons 27 28 However, multiple experimental and theoretical investigations now indicate that the Hebbian rule alone is insufficient. First, there have been numerous suggestions that some persistent "eligibility trace" must exist to bridge the temporal gap between correlated firings millisecond timescales and behavioral

---

timescales lasting seconds 17-10
Moreover, impacts of correlated spike timing must be augmented by one or more additional modulatory factors to steer weight updates toward desired out- 38 This is commonly known as learning with three factors. A prominent candidate for such a modulatory factor dopamine [3]. Dopamine influences receiving neurons via activation of G
29 30
come [ 7 8 15 16
31
receptors (GPCRs), which can regulate membrane excitability and key parameters of synaptic plasticity rules.
Besides dopamine, recent transcriptomic evidence has uncovered genes encoding a plethora of other neuromodulatory pathways throughout the brain, including neuropeptide signaling genes that are abundantly expressed in the forebrains of tetrapods, including the human [23] (4]. Like dopamine, their cellular actions are exerted by the activation of G protein-couples receptors (GPCRs), which can persistently modulate Hebbian synaptic plasticity [[5] 39 40). This suggests that they too, could play role in shaping cortical learning and synaptic credit assignment. Moreover, nearly all neurons express one or more neuropeptide signaling gene (23), which suggests a dense interplay between synaptic and peptidergic modulatory networks to shape synaptic credit assignment [25]
Approximate gradient descent learning: Standard algorithms for gradient descent learning in recur- rent neural networks (RNNs), real time recurrent learning (RTRL) and backpropagation through time (BPTT), are not biologicallypplasiin However, multiple studies have shown that learning algorithms that approximate the gradient, while mitigating some of the problems of exact gradient computation, can lead to satisfactory learning outcomes
[I and have vast computational and storage demands [42]
In feedforward networks, plenty of pologically-pllusibbe learning rules have been proposed and demonstrated impressive performance that rival hackpropagation on many different tasks 14
For efficient online learning in RNNs, approximations to RTRL have been proposed 4 17 50H54),
For instance, recent influential study [5] conceived how the three-factor learning framework could approximate the gradient. Among the biologically-plausii mainly been based on temporal truncations of the gradient computation graph and because of that, their ability to learn dependencies across arbitrary recurrent steps has been limited. Lastly, non- truncation-bar based approximations have been proposed outside of bio-plausible research [50] 51. For further discussions on related algorithms, please refer to Appendix C
proposals [46], approximations have
Results
3.1 Learning rule overview
Biological plausibility is guiding principle in developing our model. Hence, the model choices are based either on established constraints of neurobiology such the locality of synaptic transmission, causality, and Dale's Law, or on emerging evidence from large-scale datasets, such as the cell-type- specificity of certain neuromodulators (23) or the hierarchical organization of cell types [22). We consider a discrete-timee rates, as 2 at time and the corresponding internal states 8t- W denotes the recurrent weight matrix with (pg)th entry Wpq representing the synaptic connection strength between presynaptic neuron q and postsynaptic neuron p. See Supp. Section Aor the neuron model.
based RNN similar to the form in [56] with observable states, i.c. firing
Gradient descent learning in RNNs: RNNs are typically trained by gradient descent learning on the task error (or negative reward) E. However, the two equivalent factorizations of error gradient in RNNs, BPTT and RTRL, both involve nonlocal information that is inaccessible to neural circuits. This is due to recurrent connectivity: synaptic weight W pg can affect loss E through many other neurons in the other network addition to its pre- and postsynaptic neurons. To see this more concretely, the following is the exact gradient at time t using RTRL factorization, on which we base our approximation for online learning:
E
aE døja
Σ
d Wp as d W
dsj.
Asja + Σ
0zja d d111-1
Dsg.t aw pq
Bz. daa Osjæt d8j.-1 + EEtt W B1 d d 05 W
d W pg
OW pq
dst.-1 d WWp pg
depends on all weights Wa (2)
3

---

Top-down Leedback L
DWN
W
Recelving secretions
Filtered
credit signal
q
Neuron
P
Activity-deeeddett secretion of topo-oow feedback received
Local modulatory signaling, type
Synaptic
Connections
Intracellular
Eligibility
Top-down signal reception
Modulatory fitter type
Local modulatory signaling, type
signaling
Figure 1: Biologically-plausibb temporal credit assignment via modulatory and synaptic mes- sage passing. In addition to established biological learning ingredients (cligibility traces and a top-down learning signal 8 (55). synapss-type-specda
local modulatory networks may also be involved in weight updates [25]. Our learning rule, ModProp, conceives the action of participating modulators receiving cells convolution of the eligibility trace with causal, time-invariant cell-type-specifee filters. Each circle represents a neuron and the synaptic weight of interest is Wpg: illustrate the cellular processes of postsynaptic neuron p. Our derivation (Supp. Section [A]and Theorem I predicts that the modulatory signal each neuron receives can represent a filtered credit signal regarding how its past firings (arbitrary steps back) contribute the task outcome.
where denotes direct dependency and d accounts for all (direct and indirect) dependencies, following the notation in (5). (See Appendix A.2 for further details on the notation.) While _E Fsyss* which considers only the direct contribution the internal state of neuron at time to the loss, is easy to compute, the factor d53 memory and O(N)) computations. This makes RTRI, expensive to implement for large networks. Moreover, the last factor aW py poses a serious problem for biological plausibility: the nonlocal terms in Eq. 2 requires knowledge of all other weights in the network update the weight W pg- Existing biologically-plann plaaussbe solutions to this problem apply severe truncations: references 4 and [5] completely ignore the third nonlocal term (Figure 2). whereas reference [6] restores terms within one recurrent step through local modulatory signaling but truncates further terms.
is memory trace of all inter-cellular dependencies and requires O(N3)
a
d's
ModProp approximation overview: Derivation of gradient descent- based weight updates involving neuromodulatory signaling from Eq. [ suggests two approximations to Eq, [S4 and to move beyond severe truncations of the exact gradient while retaining biological plausibility. Approximation replaces the activation derivative with constant:
Ozj4 R (approximation 1),
where μ represents the average activity of neurons in ReLU network. This approximation assumes stationarity in neuron activity and uncorrelatedness of such activity with a small subset synaptic weights, explained in derivations leading up Appendix Eq. S14 Eq. S2T (Methods in Ap- pendix While neuronal activity and synaptic weights are not necessarily uncorrelated. considering that a single neuron may have thousands of synaptic partners, the activity of the neuron or its time derivative is typically weakly correlated to any one synaptic weight. This also indicates that such approximation might work better for large networks with many neurons and synaptic partners for each neuron, as is the case for biological neural networks. We take advantage of this phenomenon in model and ignore these weak correlations. This approximation enables the filter taps (Eq, () to be

---

Spatial view
No feedback
Feedback on direct influence
Feedback on direct and indirect influence
Hebbian
Aii
Three -actoratttrroo rule
ModProp
Alli Backpropagation through time
Di depends on activity of cells q p. ke
Temporal view
Bi Exact gradient, precise and nonlocal communications
Bii
Truncated gradient (three-factor rule)
Biii
Filtered gradient (ModProp)
E 2, Hildenstate
Row loxact) gred compusstion
Flow of computation for cligitiity trace
prapettrroonttiio jrreet) Forward propogation indirecti
Signals graclene
Biological learning ingredients:
lop-dovn-sggn
Local roskaittryy
computation
reception
Figure Local modulatory signaling for gradient estimation. A) A spatial view of learning rules for updating weight Wyg. (i) Hebbian learning, where weight update depends only pre-/post- synaptic activities. (ii) Three-factor learning [4, [5] (55), which updates weights using additional top-down learning signals, severely truncates the exact gradient. (iii) ModProp also accounts for (filtered) distant feedback information delivered through synapse-type-speciee
neuromodulation; (iv) BPTT computes the exact gradient: weight update involves nonlocal information, i.e. activities of indirectly connected units. B) A temporal view. Bi) BPTT propagates the precise intercellular dependencies in acausal manner. Bíi) Three-factor learning rule neglects all the intercellular dependencies in the temporal propagation of the credit signal. Biii) ModProp approximates such spatiotemporal dependencies through local neuromodulatory signals (Eq. 6 ModProp approximates the exact gradient by assuming similar connectivity among cells of the same type, and filtering the indirect effects on loss from neurons that are potentially many synapses away. Figure provides summary of approximations made by ModProp.
time-invariant, property likely required for biological plausibility. We also define S, the arbitrary number of credit propagation steps, and it will become clear later that this corresponds to the number of modulatory filter taps. With this, the estimated gradient becomes:
d d W pg
OE
OE 5
le
Epgit Σ
(W)*) Dáp -0=1 Epgit-s
(4)
Dzp.t
0sj,
where Cpq.t aWp0 (Cpg. is defined precisely in Appendix Eq.
$24) can be interpreted as persistent Hebbian "eligibility trace" 10 that keeps fading memory of past coincidental pre- and postsynaptic activity [5) Here, (W*), jip represents the (jp) entry of W W raised to the power of s, and (W ),p = Wjp- Details of the derivation can be found in Appendix
dsp.t
It is important to note that Approximation is applied only to the nonlocal gradient terms, i.e. exact activation derivative is used during the computation of the eligibility trace that is local the pre- and postsynaptic neurons. Also, we treat as hyperparameter in our simulations and explain how this is tuned in Appendix F One should note that tuning the hyperparameter l properly is important, because the estimated gradient can explode when μ is too large and ModProp approaches the three-factor rule when μ is too small. Future work involves improving ModProp with an adaptive for better numerical stability and accuracy.
5

---

Approximation 2 replaces synapse-specific feedback weights with type-specific weights:
(W') x (W")o, Jo (approximation 2).
(5)
Here, cell j belongs to type a, cell p is of type and C' denotes the set of cell types. (e.g.. Was pe Wiph jol. a E C.) This approximation is due the type-specific nature of modulatory channels [23]. We call these modulatory weights synapse-type-speciff (as opposed to cell-type-specifie) to emphasize the connectivity-hant hased grouping. Details of how these modulatory weights were obtained can be found in Appendix A
filtering credit signals via local neuromodulation: Substituting Approximation 1 and Approximation 2 into Eq, S4 and 2
ModProp:
leads to the ModProp update:
AW pola ModProp x Lp Epq +
Σ aEC
Σ jEa
L activity,) x Fas
Epgs
Fas,s 1 (W")
(6)
where L and denote top-down learning signal and eligibility trace, respectively. We postulate that Fas represents sppe-specific filter taps of GPCRs expressed by cells of type 3 precursors secreted by cells of type * is the convolution operation with S the number of filter taps. Note that the matrix powers (W*)as appearing in the values of different filter taps Fa could be genetically pre-determined as part of the cell identity and optimized over evolutionary time scales. (Appendix Figure STT shows successful learning using fixed modulatory weights.) Details of a biological interpretation of Bq. 6 can be found in Appendix D.I Briefly, the secretion of top-down (TD) learning signals can selectively activate a biochemical process at the post-synaptic neuron, which can then act temporal filter the eligibility trace.
Observing that neurons of the same type demonstrate consistent properties across a range of features (e.g. connectivity, physiology, gene expression) 223 57, we use two cell types with consistent wiring, neuromodulation, and type of synaptic action (excitaorr/inhiiitory))
our relatively simple models.
In summary, we propose a synaptic weight update rule, where the eligibility trace is compounded not only with top-down learning signals as modern moologialll--llusibie learning rules 7 g but also with local modulatory pathways through convolution (Figure I. This modulatory mechanism allows the propagation of credit signals through an arbitrary number recurrent steps.
3.2 Properties of ModProp
Remark. The biologically plausible implementation (Eq. 6 of ModProp complexity scales as O(SN2) per time step t.
Here, N and S denote the number of recurrent units and the number of filter taps. As seen in Eq. the number of filter taps corresponds to the number of recurrent steps for which the credit information is propagated. We also present an alternative implementation with potentially lower cost later in Proposition I Details for cost analysis and biological interpretation can be found in Appendix D.I
We next show through Theorem I that learning with yynaps-typeespeeee reduction at every step on average. The mechanistic intuition behind Theorem [ that, in the presence of statistical connection between synaptic weights (forward path) and modulatory weights (feedback path), the modulatory feedback signal each neuron receives can good estimate of how its activity contributes to the overall task error, which can be used for effective tuning of incoming synaptic weights to reduce the eтror. Figure S5 compares the angle with the exact gradient across different learning algorithms to demonstrate that the direction of the approximate gradient computed by ModProp is similar to (aligned with) the direction of the exact gradient, thereby reducing the loss.
weights leads to loss
In the following, we define the "residual weights" away from the cell-type averages as Gi := WG Was and (ca)iy == (W*);; (W*)ags where i a.j E B, a. B C. We consider these terms as stochastic so that the circuit output and the eligibility traces are also stochastic, functions of ex. We consider the cell type averages and the output connection strengths as deterministic.
(Merely modifying the uncorrelatedness assumption below would extend our results to stochastic output connection strengths.) Below, short for E,, AF denotes the change in the task error at time and AE pg.t denotes the contribution of the synapse (pq) to it.

---

Theorem 1. Consider linear RNNs N. N with weight matrices W, W, respectively and identical architectures. Let w Was. a.. Vi E a, E B. Assume small enough learning rate such that the remainder of the first-order Taylor expansion of loss is negligible. For network N. if Elcip] = 0. (ex)a and Ekj are uncorrelated for + j. and Ea and (y+ - vi ppq are uncorrelated for any 5,,, then BB pp.. S 0 and [[ S O. Moreover, E E| < 0 if gradient descent is possible for network N. (Proof is in Appendix E
Note that the uncorrelatedness assumption above is relatively mild because any single connection strength is typically a very poor predictor of network activity, especially as the network size grows. Note also that gradient update can introduce a drift in residual weights, requiring a similar drift in cell-type averages for the strict inequality to be applicable over multiple update steps.
3.3 Simulation results
A
Pattern Generation
B
Delayed XOR
Sequential MNIST
Classty
Cuc
Daky
2
LILL
W
0:
5
1:
Pattem Generation.
Delsyed
Sequental NNGT
300
07
175
Tho Factee
0.6
Satter
09-
155-
125-
3.1
0.2
B911 Mothee
01-
Thaer Lute:
Fenetinn 101)
Figure 3: Modulatory signaling of credit information on long-term recurrent interactions can improve learning outcomes. ModProp improves the learning performance over existing bio- plausible rules. This experiment examines the performance due to Approximation (Eq. (3) before any cell-type approximation of modulatory weights (Eq. 55) A) Learning produce time-resolved target output pattern. B) A delayed XOR task, where the network determines two cue alternatives the presence absence of input represented by or 0 match mismatch after delay, requiring memory via recurrent activity. c) Piixel-by-pixel MNIST task [58). Note that this task is unlikely to be solved effectively by humans. (See text.) Consistent with the original MDGL paper (6), also find that MDGL confers little advantage over the three-factor rule (e-prop) under dense connectivity. Solid lines/shaded regions: mean/standard deviation of loss curves across five runs.
To test the ModProp formulation, we study its performance in well-known tasks involving temporal processing: pattern generation, delayed XOR, and sequential MNIST. We compare the learning performance of ModProp with the state-of-the-arg
pelloiccally-plasibbe learning rules (MDGL (6] and e-prop [5] (labeled as "three-factor"", as well as BPTT to provide a lower bound on task error. Note, BPTT and RTRL both compute the exact gradient, so they should be identical in terms of performance. Here, we chose BPTT due to computational efficiency. Consistent with [6], we also found MDGL to confer little advantage over e-prop when all neurons connected to the readout (Figure so we focus our analysis on that case.
We first study pattern generation with RNNs, where the aim is to produce a miedimmesional target output, generated from the sum of five sinusoids, given a fixed Gaussian input realization. We change the target output and the random input along with initial weights for different training runs, and show the learning curve in Figure A across seven such runs. By using a densely connected network, we observe that MDGL confers little advantage over the three-factor rule (e-prop), as reported in the original paper ( Moreover, communicating longer indirect effects, despite being filtered, leads to improved learning outcomes, demonstrated by the superior performance of ModProp over MDGL.
7

---

Pottern Generation
Delayed XOR
MDGL
BPTT
1.75
Three-fs fector
Three-fa factor
BFTT
HOGL
Mod WWo
Mochoog yMao
Ho dProp
Mioohroo
1.25
1.00
0.75
0.58
0:-
0.2
0.00
0.0
Training Iaratima (X22)
(422)
Figure 4: Efficiint learning with type-specific modulatory weights. In addition to Approximation I studied in Figure 3 this figure investigates the effect of Approximation 2 (labeled as ModProp_ __ab), which uses type- specific,,,, rather than synapse- synapse-specific feedback weights for signaling credit infor- mation (Eq. 5). Here, ModProp_ Wab uses only two modulatory types mapped to the two main cell classes. The cell type approximation does not result any signifieant performance degradation in A) the pattern generation task and B) the delayed XOR task. This analysis is not done for the sequential MNIST task, where neurons were not divided into and I types. Solid lines/shaded regions: mean/standard deviation of loss curves across five runs.
Next, to study how RNNs learn to process discrete cues that impact delayed outcomes, we consider delayed XOR task: two cue alternatives. or O, are encoded by the presence/absence of input. The network is trained to remember the first cue and learn to compare it with the second cue delivered at a later time to determine the two cues match. Figure 3 illustrates the learning curves for this task. We observe the same general conclusion the previous task. Some learning curves have a standard deviation, which indicates that the network struggled to learn the task with these rules for some seeds; based the standard deviation of the curves, it seems possible for ModProp perform similarly as farter for some seeds, but the focus here is to examine performance across many runs. In Appendix Figure [33] also examined the learning performance of the same set of learning rules for a longer delay period. Interestingly. the performance of ModProp approaches that of BPTT for this task and the previous one, further closing the gap between artificial network training and biological learning mechanisms in performing credit assignment over a long period.
Finally, we study the pixel-by-pixel MNIST (58) task, which is popular machine learning benchmark. Although is not a task that the brain would solve well (i.e., humans would struggle to predict the digit with only one pixel presented at a time), we investigate it to test the limits of spatiotemporally filtered credit signals for tasks that demand temporally precise input integration. Figure 3C illustrates the learning curves for this task. While the performance ordering of learning rules is still the same as in previous tasks, we observe a wider gap between ModProp and BPTT. Since the time-invariant filter approximation (Eq 3) restricts the spatiotemporal resolution of the credit signal, this is in line with our expectation that ModProp will struggle with tasks that demand highly precise spatiotemporal integration, such as the pixel-by-pin -y-ieel MNIST task that even the brain would struggle solve.
As proof-of-concept study, we initially focused our analysis on the approximation performance by imposing time-invariant filter taps (Eq. 3). Next, in Figure 4 we investigate the learning performance using type-specific rather than synapse-specifie feedback modulatory weights (Eq. 5) These type- specific weights were calculated using weight averages in (6). Later, we repeat these simulations using fixed random modulatory weights in Appendix Figure(SI. we also demonstrate the superior performance of ModProp with fixed random modulatory weights for the "copy task" [59] in Appendix Figure [54] Little performance degradation is observed for only two modulatory types mapped onto the two main cell classes, indicating the effectiveness of cell-type discretization. Other than the sequential MNIST task, cells are divided into two main cell classes - with 80% of the cells being excitatory (E) and 20% being inhibitory (I) and obeyed connection sign constraints.
Lastly, we demonstrate the advantage of our weight update rule in an online learning setting of the pattern generation task, where weights are updated in real time (Figure55. For that, we also derive cost and storage efficient silico implementation of ModProp in Proposition
8

---

A
2.00
Weight update every 100 steps
2.00
]e-]
Weight update every step
HDG:
1.73
Modhrop Maac
1.75-
MDGL
Threa-lactor
155-
Modhros Web
1
1.00-
1.40
0.75
0.52
0,50-
0.25
0.21
0.00
20
0.00
20
Traiiing torations 02259
Figure 5: Superior performance of ModProp in an online learning setting. We investigate an online learning version of the pattern generation task, where weights are updated either A) every 100 time steps B) every single step. Here, ModProp uses the efficient online learning implementation derived in Proposition]] T Plotting conventions follow those of previous figures.
Proposition 1. ModProp has an online in-silico (not necessarily biologically ) implementa- tion with O(CN2) storage and O(C² N2) computational complexity, where C is the number of cell types. (Proof is in Appendix D.2
4 Discussion
A central question in the study of biological and artificial intelligence is how the temporal credit assignment problem is solved in neuronal circuits B]. Motivated by recent genomic evidence on the
widespread presence of local modulatory networks (23 (4], we demonstrated how such signaling could complement Hebbian learning and global neuromodulation (e.g., via dopamine) to achieve biologically plausible propagation of the credit signal through arbitrary time steps. Going beyond the scalar feedback provided by global top-down modulation, our study proposes how detailed vector feedback information [ can be delivered in neuronal eircuits. Instead of the severe temporal truncations of the gradient information proposed by the state-of-the-an [ ModProp offers framework where the full temporal feedback signal can be received, albeit via low-pass filtering at the post-synaptic neuron due to specificity at the level of neuronal types, and not individual neurons. Moreover, predictions generated by ModProp on the role of a family of signaling molecules (e.g. neuropeptides) could potentially be tested experimentally: physiology of multiple individual cells can be monitored modern neurobiology experiments 225 60]. Blocking certain peptidergic receptors of the neurons that are involved with learning task and comparing the performance to that without blocking can provide a test for the role of peptidergic communication.
While feedback aligmment [44) addresses the weight transport problem in feedforward networks, it not clear in RNNs which biological pathways would implement temporal feedback. Our model suggests that such pathways could come from synapse-type-speccie addition to improved performance compared to the of-he-of across all of our simulations, our theoretical and experimental results show that ModProp can be implemented efficiently for online learning. Figure [ briefly mentions some of the connections of ModProp to feedback alignment. Together. these findings suggest that synapse-type-specffe local modulatory signaling could be neural mechanism for propagating credit information over more than just a few recurrent steps.
local neuromodulation. In
Among the many future directions, a natural extension could be to investigate the performance of ModProp across broader range of tasks. This could include situations where the assumptions in deriving the rule (e.g., stationarity of activity) are severely violated. This might improve the approxi- mations introduced here. Similarly, we observed a significant gap between ModProp and BPTT for the sike-by--pixl MNIST task that demands precise temporal integration of input (Figure p)) but we discussed earlier that this is a challenging task for the brain. Additionally, this study focuses on dense networks with ReLU activation: future directions include investigating two biologically relevant paradigms: sparse connectivity and a diverse set activation functions (spike-based Although ModProp can be applied in theory to temporal credit assignment over arbitrary duration, the presented learning rule accounts for dependencies every single step (as for BPTT). This means that, similar to BPTT, it is ill-suited for very long-ten credit assignment [61. An interesting line of
particular).

---

research attends to the issue of extracting relevant, rather than full, memories [6] 62]. Investigating how our approximations could potentially be combined with memory sparsification techniques to perform very long-term biologically biologically-plausible plaussble credit assignment can be fruitful 61]
While our paper advances the basic science of learning and we do not foresee immediate societal impacts of our work, its benefits to both neuroscience and deep learning research could have long- term (positive or negative) societal impacts. It is hard overstate the philosophical implications of understanding how the brain works (and, in particular, learns). Moreover, such understanding could guide us toward curing certain diseases of the brain. Nevertheless, in the same vein, such future tools could also enable abuse if left unregulated. On the machine learning side, our method can be considered low-cost alternative to the ubiquitous BPTT algorithm. In this sense, our algorithm or a future method that builds on it can make tasks that are currently approachable by just few wealthy entities available to many more practitioners. On the flip side, as with many other capable machine learning tools, such entities are also capable of utilizing low-cost learning to conquer even harder tasks, which is potentially reason for concern. Finally, data-driven tools will ultimately reflect the various biases in their training data and our method is no exception.
Exact gradient
ModProp
W
B,
B
B
W
(' (M, (* w WF Mw.. ma (10)) w f W W.2. W
Filter Taps: (")(
(1) [8
(109) (89)
B B<sWs
Equivilent backooard computation (direct) Equivalent backward computation (ndirect)
Figure 6: ModProp brings two approximations to the nonlocal gradient terms. First, ModProp uses type-specific feedback weights (modulatory weights), rather than cell- opecifii feedback weights (Eq. B Second, ModProp approximates the activation derivative (Eq. (5.. Similar to feedback alignment for feedforward networks [44), different weights used during the backward pass than the forward pass due to the well-known weight transport problem. This, however, won't 't be adequate for RNN settings. On top of the type-specific feedback weights approximation, time-invariant activation derivative approximation was also applied for time-invariant filtering (as explained in the texts surrounding Eq. and Eq. 6 Any unspecified symbols in the illustration were defined in Figure 2
Acknowledgements
We thank the Allen Institute founder, Paul G Allen, for his vision, encouragement and support. YHL is supported by the NSERC PGS-D program. This work was facilitated through the use of the UW Hyak supercomputer system.
References
[1] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
[2] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. deep learning framework for neuroscience. Nature neuroscience, 22(11):1711-177, 2019.
[3] Timothy P. Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82-89, apr 2019.
10

---

[4] James M. Murray. Local online learning in recurrent networks with random feedback. eLife, 8, may 2019.
[5] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communicatiinn, 11(1))1-15, dec 2020.
[6] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea- Brown, and Uygar Sümbül. Cell- sype-specific neuromodulation guides synaptic credit assignment in spiking neural network. Proceedings of the National Academy of Sciences, 1 18(51)) 2021.
[7] Jeffrey C. Magee and Christine Grienberger. Synaptic Plasticity Forms and Functions. Annual Review of Neuroscience, 43(1):95-117, jul 2020.
[8] Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Coreill, and Johanni Brea. Eligi- bility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules. Frontiers Neural Circuits, 12:53, jul 2018.
[9] Sho Yagishita, Akiko Hayashi-Takaan Graham C.R. Ellis-Davies, Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science,
1616-166,, sep 2014.
[10] Stijn Cassenaer and Gilles Laurent. Conditional modulation of for olfactory learning. Nature, 482(7383):47-51, feb 2012.
plasticity
[11] Magdalena Sanhueza and John Lisman. The CaMKII/NDDAR complex as a molecular memory. Molecular Brain, 100) feb 2013.
[12] Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series, 2001.
[13] Wolfram Schultz. Neuronal reward and decision signals: from theories to data. Physiological reviews, 95()):833951, 2015.
[14] Jonathan E Rubin, Catalina Vich, Matthew Clapp, Kendra Noneman, and Timothy Verstynen. The credit assignment problem cortico-basal ganglia-thalamie networks: A review, a problem and possible solution. European Journal of Neuroscience, 53(7):2234-222) 2021.
[15] Zuzanna Brzosko, Susanna B Mierau, and Ole Paulsen. Neuromodulation of spike-timing dependent plasticity: past, present, and future. Neuron, 103(4):563-588 2019.
[16] Wolfram Schultz. Dopamine reward prediction-error signalling: a two-component response.
Nature Reviews Neuroscience, 17(3):183, 2016
[17] Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of Machine Learning Research, 21 21(135):1-34, 2020.
[18] Jean-Christophe Cassel. The role of serotonin in learning and memory: a rich pallet of experimental studies. In Handbbokk of Behavioral Neuroscience, volume 31, pages 549-570. Elsevier, 2020.
[19] Amjad H Bazzari and H Rheinallt Parri. Neuromodulators and long-term synaptic plasticity in learning and memory: A dxceredgllutamateree perspective. Brain sciences, 911):300, 2019.
[20] Eva Borbély, Báliint Scheich, and Zsuzsanna Helyes. Neuropeptides in learning and memory. Neuropeptides, 47(6):439-450, dec 2013.
[21] Trevor J. Hamilton, Sara Xapelli, Sheldon D. Michaelson, Matthew E. Larkum, and William F. Colmers. Modulation of distal calcium electrogenesis by neuropeptide Y1 receptors inhibits neocortical long-term depression. Journal of Neuroscience, 33(27): 184-11193,, jul 2013.
[22] Bosiljka Tasic, Zizhen Yao, Lucas T. Graybuck, Kimberly Smith, Thuc Nghi Nguyen, Darren Bertagnolli, Jeff Goldy, Emma Garren, Michael N. Economo, Sarada Viswanathan, Osnat Penn, Trygve Bakken, Vilas Menon, Jeremy Miller, Olivia Fong. Karla E. Hirokawa, Kanan Lathia, Christine Rimorin, Michael Tieu, Rachael Larsen, Tamara Casper, Eliza Barkan, Matthew Kroll, Sheana Parry, Nadiya V. Shapovalova, Daniel Hirschstein, Julie Pendergraft, Heather A. Sullivan, Tae Kyung Kim, Aaron Szafer, Nick Dee, Peter Groblewski, Ian Wickersham, Ali Cetin, Julie A. Harris, Boaz P. Levi, Susan M. Sunkin, Linda Madisen, Tanya L. Daigle, Loren Looger, Amy Bernard, John Phillips, Ed Lein, Michael Hawrylycz, Karel Svoboda, Allan R. Jones, Christof Koch, and Hongkui Zeng. Shared and distinct transcriptomic cell types across neocortical areas Nature, 563(7729):72-778, nov 2018

---

[23] Stephen J. Smith, Uygar Sümbül, Lucas T. Graybuck, Forrest Collman, Sharmishtaa Seshamani, Rohan Gala, Olga Gliko, Leila Elabbady, Jeremy A. Miller, Trygve E. Bakken, Jean Rossier, Zizhen Yao, Ed Lein, Hongkui Zeng, Bosiljka Tasic, and Michael Hawrylycz. Single-cell transcriptomic evidence for dense intracortical neuropeptide networks. el.ife, 8, nov 2019.
[24] Stephen J Smith. Transcriptomic evidence for dense peptidergic networks within forebrains of four widely divergent tetrapods. Current opinion in neurobiology, 71:100-109, 2021.
[25] Stephen J. Smith, Michael Hawrylycz, Jean Rossier, and Uygar Sümbül. New light on cortical neuropeptides and synaptic network plasticity. Current Opinion in Neurobiology, 63:176-188, aug 2020.
[26] Anthony N. van den Pol. Neuropeptide Transmission in Brain Circuits. Neuron, 76(1):98-115, oct 2012.
[27] Yang Dan and Mu-Ming Poo. Spike timing-dependent plasticity: from synapse perception. Physiological reviews, 86(3):1033-1048, 2006.
[28] Sen Song. Kenneth D Miller, and Larry Abbott. Competitive hebbian learning through spike-timing-deeendeer synaptic plasticity. Nature neuroscience, 3(9):919 99699999,,--9 2000.
[29] Marco P Lehmann, He A Xu, Vasiliki Liakoni, Michael H Herzog, Wulfram Gerstner, and Kerstin Preuschoff. One-shot learning and behavioral eligibility traces in sequential decision making. Elife, 8:e47463, 2019.
[30] Aparna Suvrathan. Beyond STDP - towards diverse and functionally relevant plasticity rules. Current Opinion in Neurobiology, 54:12-19, feb 2019.
[31] Michael A. Farries and Adrienne L. Fairhall. Reinforcement Learning With Modulated Spike Siming-Dependent Synaptic Plasticity. Journal of Neurophysiology, 98(6):34483366) dec 2007.
[32] Verena Pawlak, Jeffery R Wickens, Alfredo Kirkwood, and Jason ND Kerr. Timing is not everything: neuromodulation opens the stdp gate. Frontiers in synaplic neuroscience, 2:146, 2010.
[33] Pieter Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1963):166-18) feb 2018.
[34] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, pages 1-10, 2021.
[35] Johnatan Aljadeff, James D' d'amour, Rachel E Field. Robert C Froemke, and Claudia Clopath. Cortical credit assignment by hebbian, neuromodulatory and inhibitory plasticity. arXiv preprint arXiv:1911.000 1000007, 2019.
[36] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike- plasticity, and theory of three-factor learning rules. Frontiers Neural Circuits, 9(JAN2016:85, jan 2015.
[37] Räzvan V Florian. Reinforcement learning through modulation of spike-timing-dependeer synaptic plasticity. Neural computation, 1966):1468-1502 2007.
[38] Roman Pogodin and Peter Latham. Kemeeiized information bottleneck leads to biologically plau- sible 3-factor hebbian learning in deep networks. Advances in Neural Information Processing Systems, 33:7296-7307, 2020.
[39] Nicolas X. Tritsch and Bernardo L. Sabatini. Dopaminergic Modulation of Synaptic Transmis- sion in Cortex and Striatum. Neuron, 76(1):33-50) oct 2012.
[40] Eve Marder. Neuromodulation neuronal circuits: back to the future. Neuron, 76(1):1-11, 2012.
[41] Blake A. Richards and Timothy P. Lillicrap. Dendritic solutions to the credit assignment problem. Current Opinion in Neurobiology, 54:28-36, feb 2019.
[42] R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin and D. E. Rumelhart, editors, puck-propagttion: Theory, Architectures and Applications, chapter 13, pages 433-486. Hillsdale, NJ: Erlbaum, 1995.
12

---

[43] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. arXiv preprint arXiv:2005.11366 2020.
[44] Timothy P Pillierap,, Daniel Cownden, Douglas B Tweed, and Colin Akerman, Random synap- tic feedback weights support enror backpropagation for deep learning. Nature communications, 7(1):1-10, 2016.
[45] Isabella Pozzi, Sander Bohté, and Pieter Roelfsema. A biologically plausible learning rule for deep learning in the brain. arXiv preprint arXiv: 2018.
[46] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. arXiv preprint 1rXivv1810.111 13393, 2018.
[47] Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation deep convnets by drastically reducing its gradient estimator bias. Frontiers neuroscience, 15:129, 2021.
[48] Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computational neuroscience, 13:18, 2019.
[49] Beren Millidge, Alexander Tschantz, Anil K Seth, and Christopher I. Buckley. Activation relaxation: local dynamical approximation to backpropagation in the brain. arXiv preprint arXiv:2009.03359 2020.
[50] Asier Mujika, Florian Meier, and Angelika Steger. Approximating Real-Time Recurrent Learn- ing with Random Kronecker Factors. In 32nd Conference on Neural Information Processing Systems, pages 6594 6603, 2018.
[51] Corentin Tallec and Yann Ollivier. Unbiased Online Recurrent Optimization. In ICLR, feb 2018.
[52] Christopher Roth, Ingmar Kanitscheider, and Ila Fiete. Kernel RNN Learning (KERNL). In ICLR, sep 2019.
[53] Jacob Menick, Erich Elsen, Utku Evei, Simon Osindero, Karen Simonyan, and Alex Graves, A practical sparse approximation for real time recurrent learning. arXiv preprinl 07X1v22006.07232, 2020.
[54] Friedemann Zenke and Emre O Neftei. Brain-inspired learning neuromorphic substrates. arXiv preprint arXiv:2010. 1931, 2020.
[55] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
plasticity,
[56] Daniel B Ehrlich, Jasmine T Stone, David Brandfonbrener, Alexander Atanasov, and John D Murray. Psychrnn: An accessible and flexible python package for training recurrent neural network models cognitive tasks. Eneuro, 8(1), 2021.
[57] Nathan W Gouwens, Staci A Sorensen, Jim Berg, Changkyu Lee, Tim Jarsky, Jonathan Ting, Susan M Sunkin, David Feng, Costas A Anastassiou, Eliza Barkan, et al. Classification of clectrophysiological and morphological neuron types the mouse visual cortex. Nature neuroscience, 22(7): 182-11998 2019.
[58] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. comfexdb/mnist/, 1998.
[59] Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. Advances in Neural Information Processing Systems, 31, 2018.
[60] Sarah Melzer, Elena Newmark, Grace Or Mizuno, Minsuk Hyun, Adrienne C Philson, Eleonora Quiroli, Beatrice Righetti, Malika R Gregory, Kee Wui Huang, James Levasseur, et al. Bombesin- like peptide recruits disinhibitory cortical circuits and enhances fear memories. Available at SSRN 3724673, 2020.
[61] Guangyu Robert Yang and Manuel Molano Mazon. Next-generation of recurrent neural network models for cognition. 2021.
[62] Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-89, 2019.
13

---

[63] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown and Uygar Sümbül. A solution to temporal credit assignment using cell-type-specife modulatory signals. bioRxiv, pages 2020-11, 2021.
[64] Yu Hu, Steven I. Brunton, Nicholas Cain, Stefan Mihalas, J Nathan Kutz, and Eric Shea-Brown. Feedback through graph motifs relates structure and function in complex networks. Physical Review E, 98(6):062312, 2018.
[65] Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR. International Conference on Learning Representations, ICLR, dec 2015.
[66] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-leam in networks of spiking neurons. In 32nd Conference on Neural Information Processing Systems, pages 787-797, 2018.
[67] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, et al. (TensorFlow), A system for (Large-Seale) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283, 2016.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit, In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We have supported our key contribution points (see Introduction) with simulations and mathematical proofs.
(b) Did you describe the limitations of your work? [Yes] Please see discussion on future direction in the Discussion section.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see the Discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have carefully read the ethics review guidelines and confirm that this paper conforms them.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We have made sure that all assumptions are precisely stated in the Theorem statement, Details for where these assumptions come into the proof can be found in Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] We have provided complete and detailed proofs in Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL.)? [Yes] Anonymized code link is provided in Appendix F
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We experiments across random seeds and reported uncertainty for all applicable figures. We explained the plotting convention in the
caption.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix F
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? (Yes) Please see Appendix F
(b) Did you mention the license of the assets? [Yes] Please see Appendix F]]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "yR037qddZov", "reviewer": "Reviewer_fMfy", "summary": "In this paper, the authors propose a biologically-plausible temporal credit assignment rule for recurrent neural networks called ModProp. This work is motivated by recent experimental evidence on the presence of local neuromodulatory networks in the brain. Here, they propose that these synapse-type-specific local modulatory signals are received via low-pass filtering of eligibility traces at post-synaptic neurons.  \nUsing the framework of discrete-time rate-based RNNs, they derive the ModProp synaptic weight update rule and describe its properties. They also provide simulation results comparing the performance of ModProp vs. other biologically-plausible learning rules and Backprop through time on three temporal processing tasks. ModProp outperforms other previously proposed learning approaches, indicating that it is a promising candidate for understanding how the biological networks perform temporal credit assignment. \n\n", "questions": "- Have you tried the delayed XOR task with different delays between the cues? How would the performance of ModProp vary as a function of this delay?\n", "limitations": "The authors adequately address the limitations and the potential long-term societal impact of their work in the discussion section. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper addresses the important question of how credit assignment might occur in biological recurrently connected neural networks. Here they explore the potential role of local neuromodulatory signaling mechanisms and propose the ModProp learning rule. The idea of synapse-type-specific local modulatory signaling complementing Hebbian learning and global neuromodulation does seem like a viable theory. Backed by the empirical results presented here, I think ModProp is a promising candidate for biologically plausible learning in recurrent networks and warrants further exploration. \n\nAlthough ModProp can technically perform temporal credit assignment over arbitrarily long durations, the influence of distant events in time is negligible. This is akin to the problem of vanishing gradients in backpropagation through time, which makes ModProp ill-suited for very long-term credit assignment. The authors do mention this drawback in the discussion section.\n\nI think this paper is quite constrained by the page limit. Despite this constraint, I believe the authors have done a reasonably good job of describing the important concepts involved in the proposed learning rule.  \nMinor suggestions: the main paper refers to certain equations (e.g. 14) and figures that are in the supplementary material. It might be helpful for the readers to point out that this content is in the methods/supplementary material. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "vDUy5jmiO-T", "reviewer": "Reviewer_2iDW", "summary": "Evaluating the parameter gradients of recurrent neural networks forward in time ([17]) is both costly and not biologically plausible. This paper introduces novel approximations to the synaptic weight gradients of recurrent neural networks, which can be evaluated forward in time, using procedures that are both less costly and more biologically plausible than full RTRL. The authors discuss possible biological implementations based on plasticity modulation and compare the performance of their method to relevant baselines in a number of experiments.", "questions": "- How would the matrix power which appears in the weight update be calculated in a biological network? Even when averaging over connections of the same type, it's unclear to me how plausible this operation is.\n\n- Can the authors discuss in more detail how to approach the problem of determining cell types? For example, could it make sense to group by time constants, when there is heterogeneity in time constants? Can the authors provide some heuristics or intuitions on how they expect such choices to affect their rule?\n\n- Can the authors discuss KeRNL and SnAp in more detail?\n\n- As discussed in the strengths & weaknesses section, can the authors extend their experimental evaluation to more clearly demonstrate the superiority over e-prop/RFLO, including the random modulatory version?", "limitations": "Apart from the relatively weak experimental evaluation of the proposed approximations, the authors have done a good job in discussing the limitations of the current work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Finding biologically plausible learning rules which improve upon e-prop/RFLO is a very important research topic. The decomposition of the gradient and approximations proposed in this paper are a sensible approach forward, similar in spirit (but different from) recent algorithms such as SnAp and KeRNL.\n\nMy major concern with the current paper is the somewhat weak experimental evaluation. I leave some additional comments for the authors below, on points where I think the paper could be improved:\n\n- The presentation of section 3.1 could be improved. In particular the proposed biological implementation (an essential point of the present work) should be clarified and discussed in greather depth. [Small additional note: there is a problem with equation references; for example, in line 140, Eq. 14 (supplemental material) is referenced instead of Eq. 2.]\n\n- Weakness of experimental part of the paper: the paper would substantially improve if the authors demonstrated the superiority (and the cases where the different approximations fail) of modprop over e-prop in benchmarks that are not as toyish as pattern generation and delayed XOR, but perhaps not as difficult as sequential MNIST. Perhaps the copy task (with varying sequence length) is a good additional synthetic benchmark? In particular, it would be very good to assess how well the update with random modulatory weights performs in more difficult problems. Given that the focus of this work is on introducing a new approximation to the full gradient on RNNs, the somewhat weak experimental study is the major current weakness of the paper.\n\n- Given that there is a section on efficient computer implementations, a small application (or at least a discussion) to LSTM-like models and a comparison to algorithms such as SnAp (which also reduce the bias compared to e-prop/RLFO) would also enrichen the paper and help understand how well the approximations introduced here impact performance, as well as show their potential usefulness in machine learning, beyond their biological plausibility merits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gIkEZrrxzBi", "reviewer": "Reviewer_rAKi", "summary": "The authors propose a method of temporal credit assignment method for training recurrent neural networks, which is an approximation of real time recurrent learning. Their method can be implemented in a biologically plausible neural network with neuromodulators.", "questions": "Numerics:\n\n- Why don't you compare your algorithm with RTRL since it's an approximation of RTRL? Wouldn't that help isolate if the performance gap with BPTT is due to the approximations or because RTRL isn't as good as BPTT?\n\nRelation to biology:\n\n- I think it's worth further discussing the relationship of this work to [6]. Both works use cell-type specific neuromodulation, but the similarities and differences of the biological interpretations are not clearly stated.\n\nDerivation: I think the clarify and precision of the technical arguments can be improved, especially for a novice like me who is not familiar with RTRL.\n\n- Should Eq. (1) and Eq. (10) be the same?\n- Line 129: What are the mathematical definitions of $\\partial$ and $\\text{d}$? One involves the chain rule and the other doesn't?\n- Line 134: There are a number of forward references to equations in the appendix, which makes the paper difficult to read on its own.\n- Eq. (3): Should \"$\\text{d}$\" be \"$\\partial$\"? \n- Is $z=\\text{ReLU}(s)$? It seems like that needs to be true for Eq. 3 to hold. Why not say so in the main text?\n- Line 143: Is it reasonable to assume that neural activities and weights are uncorrelated? Do you have experimental evidence or a reference? My naive impression would be that this is *not* true.\n- Line 149: Based on Eq. (1), should $e_{pq,t}$ be $\\frac{\\text{d}s_{p,t}}{\\text{d}W_{pq}}$? And wouldn't that be $\\frac{\\partial s_{p,t}}{\\partial W_{pq}}$ in this case?\n- Eq. (11): Should $\\frac{\\text{d} z_{j,t}}{\\text{d} s_{j,t}}$ be $\\frac{\\partial z_{j,t}}{\\partial s_{j,t}}$? Is $h_{j,t}$ simply the derivative of the ReLU function evaluated at $s_{j,t}$? \n- Line 568: A more detailed justification for approximation (a) would be greatly appreciated. I see why the approximation is exact in the linear setting. What does it mean for the $W$ and $h$ chains to be uncorrelated? Is that a reasonable assumption in the ReLU setting?", "limitations": "The authors make a number of approximations when deriving their algorithm and I am interested to better understand to what extend these approximations affect the performance of their algorithm. In the discussion section, the authors suggest investigating their algorithm in situations where the assumptions are violated.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors tackle and challenging and important problem in theoretical neuroscience: biologically plausible training of RNNs (though an area I'm not very familiar with). I find their general approach quite interesting and it appears to suggest (potentially testable) predictions about the role of neuromodulators. Overall, I think it is a useful contribution to the field.\n\nWeakness:\n\nThe derivation of their algorithm is highly technical and requires a lot of notation. I found it challenging to follow their arguments. First, the main text includes a number of forward references to the appendices, and then the equations in the main text and appendices are not aligned. Sometimes this appears due to some approximation that is made, but other times it's not clear to me. I think overall their technical arguments could be presented with more clarity. I also think that the authors could devote more space to explaining the approximations that they make, and when these approximations are valid or not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z8EbupXOFME", "reviewer": "Reviewer_GvaQ", "summary": "### UPDATE 08/08/2022: Contribution score increased.\n\n\n### UPDATE 08/05/2022: Score increased slightly after reading the response and updated version of the submission. \n\n\nThis paper proposes a biologically plausible implementation of training recurrent neural networks as a biologically plausible alternative to back-propagation through time and already existing less plausible algorithms. This works lays out the approximation necessary for making the model believable as to what type of cell could be implementing it. The paper also proposes some experiments to valide the model. ", "questions": "What predictions can be made based on your model of the modulating candidates? \n\nHow do you propose to present the main justification of the important assumptions that make the model plausible? \n\n", "limitations": "They have addressed some of the limitations. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, and results and theorems are nicely explained using informal explanations and leaving the full proof to the appendix. The paper is somewhat novel and addresses an important question, which is often less addressed than that of backprop in non-recurrent networks. \n\nUnfortunately, the \"framework\" proposed seems to be more of a patch of approximations, each solving various problems rather than a mathematically grounded approach. The paper offers a long introduction of what are the possible candidates for their algorithms to justify this work, but it is not validated, nor any predictions are made that could make us believe that it is at all plausible. \n\nThe arguments for each approximation are mostly left in the appendix, which is the core of the paper. I would have wanted to be shown in the paper why they are possible and not have to trust the authors and move on. I observe mainly the discrepancy between the claims of the paper and what is actually shown in the paper. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept