Distinguishing Learning Rules with Brain Machine Interfaces
Authors: Jacob Portes
Christian Schmid
James M Murray
Venue: NeurIPS 2022 Accept (2022)
Keywords: brain-machine interface
recurrent neural network
reinforcement learning
biological learning rules

Abstract: Despite extensive theoretical work on biologically plausible learning rules, clear evidence about whether and how such rules are implemented in the brain has been difficult to obtain. We consider biologically plausible supervised- and reinforcement-learning rules and ask whether changes in network activity during learning can be used to determine which learning rule is being used. Supervised learning requires a credit-assignment model estimating the mapping from neural activity to behavior, and, in a biological organism, this model will inevitably be an imperfect approximation of the ideal mapping, leading to a bias in the direction of the weight updates relative to the true gradient. Reinforcement learning, on the other hand, requires no credit-assignment model and tends to make weight updates following the true gradient direction. We derive a metric to distinguish between learning rules by observing changes in the network activity during learning, given that the mapping from brain to behavior is known by the experimenter. Because brain-machine interface (BMI) experiments allow for precise knowledge of this mapping, we model a cursor-control BMI task using recurrent neural networks, showing that  learning rules can be distinguished in simulated experiments using only observations that a  neuroscience experimenter would plausibly have access to. 

Distinguishing Learning Rules with Brain Machine Interfaces
Jacob P. Portes
Christian Schmid Institute of Neuroscience University of Oregon
Center for Theoretical Neuroscience
Columbia University
j p.rotes@columbi..... edu
cschmid9@uureeonn. edu
James M. Murray"
Institute of Neuroscience University of Oregon jmurraySeuoregon edu
Abstract
Despite extensive theoretical work on biologically plausible learning rules, clear ev- idence about whether and how such rules are implemented in the brain has been dif- ficult obtain. We consider biologically plausible supervised- and reinforcement- learning rules and ask whether changes in network activity during learning can be used to determine which learning rule being used. Supervised learning requires a credit-assinnment model estimating the mapping from neural activity behavior, and, a biological organism, this model will inevitably be an imperfect approxi- mation of the ideal mapping. leading to bias in the direction of the weight updates relative to the true gradient. Reinforcement learning, on the other hand, requires credit-assignment model and tends to make weight updates following the true gradient direction. We derive a metric to distinguish between learning rules by observing changes in the network activity during learning, given that the mapping from brain to behavior is known by the experimenter. Because brain- trainnmachine inter- face (BMI) experiments allow for precise knowledge of this mapping, we model a cursor-control BMI task using recurrent neural networks, showing that learning rules can be distinguished in simulated experiments using only observations that neuroscience experimenter would plausibly have access to.
Introduction
In order to update synaptic weights effectively during learning, biological artificial neural network must solve the problem of credit assignment. That is, a neuron must infer whether it should increase or decrease its activity in order to decrease the error in the behavior that it is driving. Supervised learning (SL. provides one approach for solving the credit assignment problem by endowing the synaptic learning rule with credit assignment mapping, i.e. an estimate of how each neuron's 's activity affects the behavioral readout. Indeed, gradient- based methods such as backpropagation for feedforward networks and backpropagation through time for recurrent networks do exactly this by multiplying a (possibly multi-dimensional) readout error by the downstream readout weights in order to update hidden-layer synaptic weights. However, because neurons in biological neural network likely lack complete information about their downstream projections, ideal credit assignment mappings such those used in gradient-based methods are biologically implausible-an issuc known as the weight transport problem [ 3]. Recent work has shown that effective supervised learning
corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in both feedforward networks and RNNs can still take place as long as there is a positive overlap between the true readout weights and the credit assignment matrix used for learning, providing one possible solution to this problem
6 7
Reinforcement learning (RL) is another approach for solving the credit assignment problem. Node perturbation algorithms correlate noise injections with scalar reward signals in order to update network weights 8 g [0). Such algorithms avoid the weight transport problem, as the update rule does not explicitly require a credit assignment matrix mapping the vector error back to the recurrent weights. While the weight updates for algorithms in this family tend be noisy, the volicy-gradient theorem guarantees that they tend to follow the true gradient of the objective function (Appendix B.2
g I1).
machine interfaces (BMI) provide an ideal paradigm for investigating questions about learning and credit assignment in real brains, as they enable the experimenter to define the mapping (the "decoder") from neural activity onto behavior. For example, monkeys can be trained to control a cursor using a BMI in the motor cortex, and the behavior can be relearned following different
manipulations of the BMI decoder in the course of a single day [12 13 [4] or over the course of
multiple days [15 16 [7]. In the case of BMIs, the weight transport problem is particularly acute, since the readout weights mapping the neural activity onto the cursor position may be changed abruptly by the experimenter, and there is no plausible way for the agent to generate an instantaneous estimate of the new readout weights in order to then apply a learning rule. If an agent is to use a credit- assignment model at all in such a task, it must consist of weight matrix that imperfectly approximates the readout mapping, i.c. biased internal model of credit assignment. We refer to this as the decoder alignment problem.
The key insight in work is that, under an SL rule, the agent will invariably perform credit assignment using a biased credit-assignmet model, whereas no such model is required under a policy gradient-like RL rule. Using this insight, we develop a framework for modeling BMI experiments with recurrent neural networks (RNNs) using biologically plausible versions of SL and RL. We show that an imperfect credit-assignment model introduces systematic bias in the weight updates, as well as the direction in which neural activity evolves during learning. Under the assumption that SL is biased but RL is not, this therefore provides a means by which to distinguish these different learning strategies. We derive a statistical metric for comparing observed changes in neural activity with the changes predicted assuming knowledge of the BMI decoder, and we show how this metric can be used to detect learning bias in our simulated experiments. While we focus on distinguishing biased SL from unbiased RL, the approach described here could be used more generally to distinguish different biased vs. unbiased learning rules- or even biased learning rules with two different types of bias-frrm one another.
Related work. While there has recently been a renaissance of research on biologically plausible learning rules 44
IO 6 18 3), there has been relatively little work on how such proposals might be experimentally verified in neural experiments. Lim et al. [19] inferred hyperparameters of a Hebbian learning rule based on neural activity from a visual cortical area in monkeys. Their work. however, assumed a single class of learning rules rather than trying to distinguish between multiple classes. Nayebi et al. 20 showed that different optimizers and hyperparameters used to train convolutional neural networks can in principle be distinguished by a classifier based on both weight changes and neural activations. However, the classifier was trained using ground- truth data about the identity of the learning rule, which is the very thing that we aim to infer in our model. Kepple et al. [21] proposed curriculum- baaed method for discerning the objective being optimized with SL and showed via RNN modeling that different objectives lead to distinct task-acquisition timescales. While their approach analyzed evidence accumulation and decision tasks and focused on distinguishing different objectives, we focus primarily on continuous control tasks and on distinguishing learning rules. While Feulner and Clopath [22 simulated BMI learning with RNNs to compare learning that required changes within vs. outside of the intrinsic manifold of neural activity, and Feulner et al. (23) used similar approach to analyze how learning at upstream or recurrent synapses affected the covariance of neural activity, neither of these studies attempted to distinguish learning rules. To our knowledge, our study is the first to propose a framework for distinguishing between learning rules using BMIs.
2

---

A
Whemi
B
C
D
predict
W
Mc(e)
bmist
C(
SL rule
observe P(A
vecctor
Whoa
observe
error Whes
P(A
Wreee
10
W huai
true gradient biased updale high variance update
W
R(O)
RL. rule
compare
W wattr
scalar
protrain
change
train
h1
AP(AG
reward
decoder
predict
Figure 1: Approach. (A) We model brain-machine interface experiment in which monkey learns to move a cursor on a screen using either supervised learning (SL, top) or reinforcement learning (RL, bottom). (B) We pretrain an RNN to perform center-out reach task with fixed decoder W/mi0 switch the decoder to Whmil, then train the recurrent weights Wre again with either SL or RL. (C) SL with biased credit assignment mapping will lead to biased recurrent weight updates, whereas RL leads to noisy but unbiased weight updates. (D) By inferring the neural activity flow fields from observed data and comparing the observed change during learning with the predicted changes assuming SL. or RL, we infer which learning rule was used to train the RNN.
2 Theoretical approach
The overall approach for our work is shown schematically in Fig. We train RNNs to perform BMI-inspired cursor-control task using different learning rules, with the goal of analyzing the observed activity to infer the learning rule that was used to train the network. Inspired by numerous
BMI experiments [12 24 16] we pre-train the RNN using a particular choice of decoder weights, then require the RNN to relearn the task using new set of decoder weights. Our main result is to predict the different ways in which the RNN activity is expected evolve through learning, then to compare these predictions with the observed changes in the activity in order to infer the learning rule. In the sections that follow, we first introduce the SL and RL update rules that are used for training our networks. We then show that they lead to distinct predictions about how the RNN activity is expected to evolve with training. The remainder of the paper presents empirical results showing that the learning rules can be inferred by observing the network activity, first for a relatively simple case and then under more generalized assumptions.
While our main results can be obtained using either a feedforward or recurrent circuit model, we mostly focus the latter case because BMI experiments typically involve control tasks driven by time-varying activity in motor cortex, for which an RNN model is more appropriate. The feedforward version of our results provided in Appendix A
2.1 Modeling BMI learning with RNNs
We model motor cortex as a vanilla RNN characterized by input weights Win, recurrent weights WFee feedback weights ftf and BMI decoder weights Whmi, Since the experimenter has complete control over the decoder in a BMI experiment, we treat Whmi as fixed and assume that learning occurs in recurrent weights Wree alone. The activity of the recurrent units h' is given by
h = (1 - (1-
Σ
Wrech 1-1 + Ewit
Σ W 1
c (1)
where T is the RNN time constant, Φ(.) = tanh(-) ) a nonlinear activation function, x" is an input signal, and 5 ~ ~N(0,, o is isotropic noise added to the recurrent units. This activity is read out by Whmi to obtain the cursor position ví. Σ, Whbii, The error at each time step is defined c y ka
V where y* is the target output for each time step. The loss to be minimized is then L = 77 Σi= 1 ΣN" (ct)2 where T is the trial duration and N₂, = 2 is the number readout dimensions.
Supervised learning. Standard supervised algorithms for RNNs such as Backpropagation Through Time [25] and Real-Time Recurrent Learning [26] are widely acknowledged to be biologically

---

implausible. The reasons for this are two-fold. First, both algorithms assume knowledge of the readout weights for purposes of credit assignment, which we referred above as the weight transport problem. Second, updating a particular recurrent weight with one of these algorithms requires knowledge of all of the other weights in the network, which is not information that biological neuron would likely have access to.
recently proposed alternative, Random Feedback Local Online (RFLO) learning [5], is an approx- imate gradient- based algorithm with A Wre R -0L/ that addresses these problems by (i) assuming that error information projected back into the network with weights that imperfectly approximate the ideal credit assignment mapping, and (ii) dropping nonlocal terms from the update rule. The recurrent weight update with this learning rule
A ij
= Σ
[Me)] 1.Pig
pij - -
-1- Pig +
- du()))
(2)
with of Ex Wrech 1 Ee ix # + Ex W vi. and p/, se /OW//O Further details are provided in Appendix B.I The learning rule that we implement differs slightly from RFLO in that the readout weights, as a BMI experiment, are fixed during training rather than learned. In required in order to align the readout weights with the
RFLO, learning of the readout weights credit assignment matrix. Here we will instead assume that the credit assignment matrix M is fixed and partially aligned with (Wbmi)T T, that the cosine similarity of the flattened matrices satisfies sim(M, (W ) >>> O.
In (2). the credit assignment mapping Mj assigns responsibility for the readout error ef. neuron While the ideal mapping would be M (Wibi)), learning can still be effective as long as there is positive alignment between M and (Wbmi)7 [5] (as shown previously in the case of feedforward networks 44). In the case where they are not equal, as we presume they cannot be in a BMI experiment, the matrix M constitutes biased rredit-assigmeett mapping. Below, we show that this leads to a predictable bias in the change of the RNN activity as the task is learned.
Example output trajectories from an RNN trained on the cursor-control task with the SL rule described above are shown in Fig. [A. The results in Fig. 2 show that learning the task is possible whenever there is partial alignment between M and (W bymry and that faster learning occurs this alignment increases.
Reinforcement learning, To train RNNs with RL, we use a node perturbation learning rule closely related to REINFORCE [8) We first define a scalar reward R |c' Node perturbation seeks to increase this reward by trial and error using the noise appearing in the recurrent population activity in (I] If the noise to a given neuron causes the reward to increase, then the weights will be updated so that that neuron's input will be greater in subsequent trials. Including an eligibility trace in the learning rule further makes it possible to assign credit for changes reward to noise events at previous time steps. Finally, we can subtract off baseline from the reward so that learning is guided by a reward prediction error, which somewhat decreases the variance of the weight updates without introducing bias [II]. The resulting learning rule, which we derive using the policy gradient theorem 27 in Appendix B.2 is
T AW = E(n' 1e99.
d -
1- -
1 7
Very similar RL rules for RNNS have been previously proposed and studied in Refs. 28 Importantly for our purposes, the fact that this learning rule is derived using a policy gradient approach guarantees that, upon averaging over isotropic noise, the weight updates follow the true gradient of the loss function (in our case given by L= E, RR),, making this an unbiased learning rule. Like 2 33) does not have nonlocal terms. Example output trajectories and the learning curve from an RNN trained on the cursor-control task using (3) are shown in Fig, D.E.
2.2 Characterizing changes in neural activity with vector flow fields
Because observing changes in neural activity is much more straightforward than observing changes in synaptic weights in neuroscience experiments, we aim to identify experimentally observable signatures in the neural activity that might distinguish biased SL from unbiased RL. Because the

---

RNN activity consists of time- dime-dependent trajectories, and changes in the weights will generally affect entire trajectories cumulatively from one time step to the next, calculating the change Åh' due to an update A W is not entirely straightforward (though cf. the feedforward case described in Appendix where this can be done). A more useful approach is to quantify the learning-indun induced change the vector flow field that shapes the RNN activity, where the flow field can be estimated empirically by observing neural activity trajectories. To do this, we linearize (T), assuming o(u) = u, to obtain an expression for vector flow field function F(h) that maps activity h' + h and is defined for all points h in neural activity space:
F(h) = [(W 1)h + w x + w
where, in our simulations, At = 1. Thus, from ( we see that F(h') h h4. If the recurrent weights change according to Wree + W + + W then the flow field exhibits corresponding change F - F + AF, where 4F(h) Wre to changes in neural activity (note that this assumes that plasticity is occurring in the recurrent weights Wree alone, and that Win and wi do not change). As illustrated in Fig. D our approach will be to (i) predict the expected change in flow field A (h) under either SL or RL, (ii) estimate AFobs(h) empirically using activity observed early and late in learning, and (iii) determine which learning rule was used to train the network by comparing the empirically observed vs. predicted flow field changes.
& Awe.. This equation enables us to relate changes in
In order predict the flow field change in the SL case for point h in neural activity space, we predict the direction of weight change using an approximation of (2):
A A (h) A W h
AW pred
ij SL
Σ
ΣΣ
Ma hig
(5)
nemid
where n refers trials, and mid denotes that we sample from trials during the middle part of training, when most of the learning is presumed to take place. In the RL case, we similarly predict the direction of weight change using an approximation of
AFpred RL (h) = A A RL
A
AW.red
pred sj
RL
ΣΣΣN
War not hz k
(6)
nEmid
In this equation, Σ is the recurrent noise covariance, which, under conditions of isotropic noise, reduces to a prefactor J.
Importantly for our purposes, the predicted changes in flow field from (5) and (6) depend entirely on quantities that are known or can be estimated from experimental data, including Whmi as well as samples of h. and en.,, As shown in Section 3.4lwllww M can also be estimated from data. Because we are asking only about changes in the flow field rather than the flow field itself, these quantities do not depend explicitly on the recurrent connectivity Wree itself, which is presumably unobtainable experimentally. The key distinction between (5) and (6) is that the change in flow field in the SL case depends on M. while the change in flow field in the RL case depends (Wbmi)T T. We motivate (5) and (6) by computing (A Wrre) for each learning rule (averaging over noise) in Appendix B.I and Appendix B.2 respectively. We provide empirical evidence that updates approximately follow consistent direction for each rule in Appendix C.5
Next, we wish to estimate the empirical change in the flow field given neural activity observed at different points in learning. The approach is somewhat simplified if we assume that the trials during which learning takes place are preceded and followed by "early" and "late" blocks of trials, in which the performance is observed but the weights are not updated. The assumption that the amount of learning during these carly and late blocks is negligible is justified in the limit where the learning rate is small and the number of training trials is large. (Evidence that our main results do not depend strongly on this assumption is given in Appendix A where learning was allowed to take place during the early and late blocks.) Then, from
4. we define the observed flow field from time step t in trial n as either Fearly (h) = (A early 1)h Flate (h) (A late I)h, where A early is the least squaresuares solution to the autoregression equation h,.++1 = A n. for all time steps i in all early trials and A late is the corresponding solution for late trials. The empirical change the flow field due to learning for any point h is then given by AFobs(h) Flate (h) Feacly (h).
5

---

A
B
C
Train with SL
Train with
Corr(A Fobs, FFrred)
BPTT
Loss (MSE)
0.10
0.75
sim1
sim0.6
0.5
0.05
sim0.4
0.25
RL
sim0
0
SL
1500
3000
0.4
0.5
0.6
0.8 0.9
Trials
sim(M, (W bmit) T)
D
F
Train with RL
Train with RL
-0.5
Corr(A AFored)
0.75
RL
reward
-1.0
0.50
SL(M)
-1.5
0.25
-2.0
0
25000 trials
50000
0.2
0.4
0.6
0.8
sim (M, (W Tm
Figure 2: Distinguishing learning rules in a trained network. (A) Example output traces from RNN trained with SL to move a cursor to one four targets. (B) When trained with SL, the similarity of the credit assignment mapping with the readout weights determines the speed of training (red curves). Learning in an RNN trained with backpropagation through time shown for comparison (black curve). (C) For RNN trained with SL, the similarity of the observed flow field change with the predicted change assuming either SL (red) or RL (blue). (D) Example output traces from an RNN trained with RL to move cursor to one of four targets. (E) Learning curve for an RNN trained with RI. (F) For an RNN trained with RI., the similarity of the observed flow field change with the predicted change assuming either SL with a randomly sampled M (brown) or RL (blue).
Having defined the observed and predicted changes in the flow fields for arbitrary h, we can define the correlation between these quantities as
Nicialo
T
A (h m..)
A F n..t)
Corr(A Corr(AFoos A Appred)
Nerials T
Fobs (h
Fored hn.))
We refer to this the flow field change correlation (FFCC). One important point to note is that, because this quantity is computed only at observed values of h it is not the case that the flow field needs be estimated everywhere in the high-dimensional space of neural activity. If the activity tends to occupy a low-dimensional manifold within this space, then the flow field needs to be determined only within this manifold. Another point note is that this metric is independent of the magnitudes of Fobs and A so that it does not depend on overall prefactors such as the learning rates and network time constant. In the simulations that follow, the correlation metric (7) will be computed separately under our two hypothesized learning rules, i.c. with A = AF SL and with AFrred = A While the absolute magnitudes of this quantity are not informative on their own, we show below that the relative values can be compared to infer which of the two learning rules is more likely to have generated the data.
3 Simulation results
In this section, we use the theory outlined above to show that the SL and RL rules we consider can be distinguished simulations, first in basic case with simplifying assumptions, and then in more realistic cases where these assumptions are relaxed.

---

A
B
Train with SL
Corr(AFebr AFpredy
0.8
sim(M, ( (mmm T)
0.6
SL
W baii
0.4
Woe
0.2
Mc(t)
Learn M
0
0
500
1000
Trials
C
D
E
W M
Train with SL
Train RL
Corr AFpred
(AoraA AFered)
0.6
RL
RL
0.8
SL
0.4
SL
0.6
0.4
0.2
W bani
0.2
W
Mc(t)
0.5
4 Y (strength of (/))
3
5
0.5
2
3
4
5
Y (strength of (D)
Figure 3: Learning the credit assignment mapping and driving with visual feedback. (A) In an RNN trained with SI., the credit assignment mapping M is learned online during training. (B) The similarity of M to (Wbmi)T increases with training due to learning of M (green). The flow field metric correctly identifies SL as the learning rule early in training when M and W bii are dissimilar, but not later training when they are very similar. (C) An RNN (now with fixed M) is additionally driven by a feedback signal via weights wh o M. (D) Comparison between predicted and observed changes in the activity flow field as a function of feedback gain for an RNN trained with SL. (E) Comparison between predicted and observed changes in the activity flow field a function of feedback gain for an RNN trained with RI.
3.1 Distinct neural signatures of biased SL and unbiased RL
In order emulate monkey BMI experiments (e.g.. 12 16 15
14 we first pretrain the recurrent weights Wre of an RNN with fixed, random readout weights W bmio to perform a center-out cursor- control task, in which cursor must be moved to one of four target locations specified by the input to the RNN. We then change the BMI decoder to Whmil, make identical copies of the network, and train one with SL using credit-assignment mapping M and the other with RL. For these simulations, shown in Fig. 2 we make the following simplifying assumptions (to be generalized in later sections): (i) the credit assignment mapping M fixed and known; (ii) the feedback weights Th are zero; and (iii) the noise g ~ NO0, I is isotropic.
13
For the networks trained with biased SL. and sim (Wbm00 Wbmil = 0.5, we find that a comparison of the observed and predicted flow field changes correctly detects the bias (Fig. 2C). As we vary the alignment between M and (Wbmil)? T, we find that the FFCC correctly detects the bias present during learning whenever these two matrices are sufficiently distinct. For the networks trained with RL, we find that the change activity is aligned more closely to the prediction using true decoder Whmil than when using a random matrix M that is partially aligned to Wbmil As we increase the alignment sim (M, (Whmil) T), this distinction is less discernible (Fig. 2F). We show in Appendix C.I that these results are robust to changes in hyperparameters.
In the empirical results that follow, we will generalize the assumptions made thus far and show that there is significant range of parameters where the results still hold. In particular, we will consider the roles played by (i) estimation and/or learning of the resit-asssgnment model M, (ii) driving feedback via the weights fh and (iii) non-isotropic noise.
7

---

A
Train with SL
Train with RL
Alignment
0.8
Forr AFPred)
0.6
(L(M)
SL(M)
RL
Corr(AF AFarod
SL((M) RL
0.6
0.4
0.4
0.4
0.2
Sim M)
0.2
0.2
sim LimMM.(Wmmg)) (
2
3 4 5
8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
PC PCAcomponents
sim(womio wbmit)
sim(w w wbmit)
SM (Whminy) y))
Figure 4: Estimating the credit assignment mapping. (A) The similarities between the estimated M and the true M (magenta) and between M and (Wbmio)T (blue) as a function of the number of principal components used in the regression in RNN trained with SL. (B) Comparison between predicted and observed changes in the activity flow field as a function of old vs. new decoder alignment for an RNN trained with SL. Model predictions assume SL using the true credit assignment mapping M (red), SL using the M estimated from pretraining (brown), or RL (blue). (C) Same as (B), for an RNN trained with RL (and where M is estimated from pretraining).
3.2 Learning the credit assignment mapping
In order to optimize learning in supervised setting, a learner's 's credit assignment mapping could improve over time to align more closely with the decoder (Fig. 33). To investigate the effect of such learning in our model, applied our analysis to RNN in which recurrent weights W were trained with SL, with a biased credit assignment mapping M that slowly improved via weight mirroring, a perturbation-base algorithm for learning a credit assignment matrix 66 18) In simulations (Fig. 33, the alignment between M and (Wbmll) is initially set at 0.5, then slowly increases as M is learned simultancously along with the recurrent weights WFe Instead of predicting the change in flow field before and after learning, we now apply our analysis over discrete windows during learning. As expected, activity changes are biased-and hence the SL and RL learning rules are bistinguishabbe-eann in learning. However, these differences disappear as M approaches (Whmil)T T later in learning.
3.3 Incorporating driving feedback
So far, we have not included driving feedback, which can strongly influence the dynamics of population activity. We next included fixed driving feedback weights f 7M and varied the strength of these weights by a scalar factor 7 (Fig. BC. M was randomly generated such that sim(M, (W/miilTT = 0.5. Note that the networks trained with SL therefore used the same mapping M for both the learning rule and for the driving feedback. We found that bias in an RNN trained with SL is correctly identified for wide range of 7 (Fig.BP) For RNNs trained with RL, we found that the correct learning rule is identified for weak feedback, but that the FFCC metric fails for strong driving feedback (Fig. B). This is likely because neural activity in this limit dominated by the contribution from W which is fully aligned with M and less aligned to (Wbmil)
3.4 Estimating the credit assignment mapping from observed activity
Our approach thus far estimates the extent to which the activity updates follow either the direction of the gradient defined by the decoder weights or the (possibly biased) gradient defined by the credit assignment mapping. This estimation assumes knowledge of the credit assignment mapping M. In a neuroscience experiment, however, this quantity would not be known a priori and would need to be estimated. To obtain an estimate M of M, we follow an approach previously used in experiments, in which dimensionality reduction is first applied the neural data, followed by linear regression to relate the network activity to the cursor movements [12 16 22). (A possible alternative approach
could be to use the Internal Model Estimation framework proposed by Golub et al. (29] .) More specifically, we (i) pretrain the RNN with decoder bbmio and biased credit assignment mapping M (where sim (M, (Wbmi0) T) = 0.6), (ii) observe the activity and cursor trajectories produced by the trained network, (iii) perform principal component analysis on the network activity and relate
8

---

the principal components to the cursor positions with linear regression, and (iv) use the regression coefficients to define the estimated credit assignment mapping M = DC, where C is the matrix of compoonntt eigenvectors and D is the matrix of regression coefficients. Fig. I shows that the M obtained in this way is similar to the true M used to train the RNN. It is important to note for our aim of distinguishing learning rules that M is more similar to M than to W/mlo and correctly identifies the alignment between M and (Wbmil)T. T. We also see from Fig. 4A that this result depends only weakly on the number of principal components used to describe the network activity.
After estimating M. we changed the decoder to Whmil and trained the network to proficiency with either SL. (using M in the learning update) or RI. (which doesn't use M). For an RNN trained with SL. (Fig. 3) ) the correct learning rule is identified using either the true M (red curve) or the estimated M (brown curve) for low and intermediate degrees of similarity between Whbi0 and Whmil For an RNN trained with RL (Fig. BC), correct identification of the learning rule does not depend on the similarity between W and W Together, the above results show that, for relatively wide range of parameters, the credit assignment mapping can estimated from observed data, and that this estimate is sufficient to distinguish between learning rules used to train different networks.
3.5 RL with non-isotropic noise
In the theoretical and empirical results above, we have assumed that the noise covariance is isotropic. If we instead allow for non-isotropic recurrent noise covariance Σ, we find that this covariance appears in the expected RL weight update (Appendix [.2). As in feedforward networks (30), this introduces a bias in the weight updates, so that the noise-averaged weight updates no longer follow the true gradient. Given that RL no longer unbiased in this case, we thus asked under what conditions we might still be able to distinguish biased SL from RL.
We first established that the RNNs can learn cursor-control task with non-isotropic noise via either SL (trivially) or RL. (as long as some of the noise is in the subspace of the decoder). To do this, we pretrained networks with Wbmio changed the decoder to Whmil, and then trained using either SL or RL. In the SL case, we chose a biased credit mapping M that has partial overlap with the new decoder, with (im (Whmill) TT) = 0.6. The noise covariance was chosen to be d-dimensional, with rank(2) = d, and isotropic within those dimensions. The first two of these dimensions were selected to lie in the subspace spanned by M for the Sl.-trained RNNs or by Wbmil for the RI- trained RNNs Other components of E were added in random dimensions orthogonal to this subspace and to one another.
The results in Fig. show that, when training with SL, cannot reliably distinguish the learning rule used to train the network when the noise is very low-dimensional (Fig. BB. When training with RL, however, the learning rule can be distinguished even with low-dimensional noise (Fig. (C). As the noise dimensionality increases, the learning rules become more easily distinguishable in both cases. This remains true in the RL case even if we do not presume to know the true noise covariance matrix, but instead use a naive assumption of isotropic noise in order to generate the predictions (dark blue lines in Fig. C,
4 Discussion
In this work, we have proposed method for distinguishing biased SL from RL under the assumption that the mapping from neural activity to behavior is known, as in BMI experiment. More generally, the approach could be used to distinguish between different candidate learning rules with different degrees or types of bias. We have illustrated the approach by modeling BMI cursor-control experiment in motor cortex, where there is ample experimental evidence for plasticity during motor learning 31I. Both SL and RL are plausible candidates for how motor learning might be implemented in the brain, either within motor cortex itself or within extended motor circuits involving motor cortex together with the cerebellum B2 33
or basal ganglia [35 36].
One possible limitation of our approach is that the number of learning rules that might conceivably be implemented in recurrent circuits in the brain is much greater than the two that we have focused on here. Nevertheless, we view the SI. and RL rules considered above as representative examples of two fundamental classes of learning rules: one in which learning is based on model for credit

---

A
C
Train with SL
Train with RL
&~ {~,(
Corr(AFor AFpredy
SL RL(o"1)
RL(2)
CArr(A AFFredy
0.6
SL(M) RL(o"1) (Z)(Z)
0.75
0.4
0.5
0.2
0.25
W Wiomi
0
W
25
10
25
noise dimension
noise dimension
Figure 5: Distinguishing learning rules with non-isotropic noise. (A) Correlated noise with covariance matrix Σ is injected into the recurrent units of the network during training and testing. (B) For RNNs trained with SI., the similarities between the observed flow field change and the predicted flow field change assuming SL (red), RL using the true noise covariance (light blue), or RL using naive isotropic estimate of the noise covariance (dark blue). (C) Same as (B), but for RNNs trained with RL. Brown line is correlation of observed flow field change and predicted flow field change assuming 0.6.
SL with random M, where sim(M, [M, (Wbmil)
assignment, where this model is necessarily imperfect, and another which uses no such model. While we cannot test every learning algorithm from these classes, we conjecture that the basic fact that allows us distinguish our SL and RL rules-uaaaee rules-namely, that the expected flow field change depends on credit-assignment matrix for model-based rules but not for non non-model-basee rules is a fairly generic feature of these two classes. To support this, we show in Appendix C.2 that our main results hold for different version of the biased learning rule that we have studied.
While have shown that different learning rules in principle distinguishable in simulation, there will be challenges in extending this analysis to real data. Reliable estimation of the FFCC metric might require large populations of neurons recorded for long periods of time. Other potential challenges that will need to be addressed include accounting for the possibility that multiple types of learning might be occurring simultaneously different locations, as well as considering cogmitive mechanisms such as attention and contextual inference, which have not been included in this model. Finally, there are possible ambiguities consider in mapping our model onto the brain's motor circuitry. While the most straightforward interpretation is that the RNN in our model corresponds to motor cortex, more elaborate interpretation could be made by noting that both the cerebellum and the basal ganglia have well-established roles in motor learning (37 and are likely to play roles in BMI learning as well (36 38
Acknowledgments and Disclosure of Funding
The authors would like to acknowledge Larry Abbott, Andrew Zimnik, Laureline Logiaco, Kaushik Lakshminarasimhan, and Vivek Athalye for fruitful discussions. Thanks to Amin Nejatbakhsh, David Clark, and Danil Tyulmankov for feedback on the manuscript. Support for this work was provided by NIH-NINDS (ROONS1 14194).
10

---

References
[1] Stephen Grossberg, Competitive learning: From interactive activation to adaptive resonance.
Cognitive Science, 11(1):23-63, 1987.
[2] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al.
A deep learning framework for neuroscience. Nature Neuroscience, 22(11):1761-1770) 2019.
[3] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-344 2020.
[4] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support enror backpropagation for deep learning. Nature Communications, 7(1):1-10, 2016.
[5] James M Murray. Local online learning in recurrent networks with random feedback. eLife, e43299, 2019.
[6] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in Neural Information Processing Systems, 32, 2019.
[7] Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in Computational Neuroscience, 13:18, 2019.
[8] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-25 1992.
[9] Justin Werfel, Xiaohui Xie, and H Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Advances in Neural Information Processing Systems, 16, 2003.
[10] Thomas Miconi.
Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. elife, 6:e20899, 2017.
[11] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[12] Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven Chase, Stephen I Ryu, Elizabeth C Kyler-Kabara, M Yu Byron, and Aaron P Batista. Neural constraints on learning. Nature, 512(7515):423-426, 2014.
[13] Xiao Zhou, Rex N Tien, Sadhana Ravikumar, and Steven M Chase. Distinet types of neural reorganization during long-term learning. Journal of Neurophysiology, 121(4):1329-134) 2019.
[14] Darby M Losey, Jay A Hennig, Emily R Oby, Matthew D Golub, Patrick T Sadlter, Kristin M Quick, Stephen Ryu, Elizabeth Hyler-Kabara Aaron P Batista, M Yu Byron, et al. Learning alters neural activity to simultaneously support memory and action. bioRxiv, 2022.
[15] Vivek R Athalye, Karunesh Ganguly, Rui M Costa, and Jose M Carmena. Emergence of coordinated neural dynamics underlies neuroprosthetic learning and skillful control. Neuron, 93 (4):955 970, 2017.
[16] Emily R Oby, Matthew D Golub, Jay A Hennig, Alan D Degenhart, Elizabeth C Kabara, M Yu Byron, Steven M Chase, and Aaron P Batista. New neural activity patterns emerge with long-term learning. Proceedings of the National Academy of Sciences, 116(30): 15210-111211 2019.
[17] Ethan Sorrell, Michael Rule, and Timothy O'Leary. Brain-machine interfaces: Closed-loop control in an adaptive system. Annual Review of Control, Robotics, and Autonomous Systems, 4: 167-189, 2021.
[18] Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording. Learning to solve the credit assignment problem. International Conference Learning Representations, 2019.
11

---

[19] Sukbin Lim, Jillian L McKee, Luke Woloszyn, Yali Amit, David J Freedman, David L Sheinberg. and Nicolas Brunel. Inferring learning rules from distributions of firing rates in cortical neurons. Nature Neuroscience, 18012)1800-1110, 2015.
[20] Aran Nayebi, Sanjana Srivastava, Surya Ganguli, and Daniel L Yamins. Identifying learning rules from neural network observables. Advances in Neural Information Processing Systems, 33, 2020.
[21] Daniel R Kepple, Rainer Engelken, and Kanaka Rajan, Curriculum learning as a tool to uncover learning principles in the brain. International Conference on Learning Representations, 2021.
[22] Barbara Feulner and Claudia Clopath. Neural manifold under plasticity in a goal driven learning behaviour. PLoS Computational Biology, 17(2)ee1086221, 2021.
[23] Barbara Feulner, Matthew G Perich, Raeed H Chowdbury, Lee E Miller, Juan A Gallego, and Claudia Clopath. Small, correlated changes in synaptic connectivity may facilitate rapid motor learning. Nature Communications, 13(1): 1-14,, 2022.
[24] Matthew D Golub, Patrick T Sadtler, Emily R Oby, Kristin M Quick, Stephen Ryu, Eliza- beth C Tyler-Kabara, Aaron P Batista, Steven M Chase, and M Yu Byron. Learning by neural reassociation. Nature Neuroscience, 21 (44)6667766 2018.
[25] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[26] Ronald J Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-28)) 1989.
[27] Peter Marbach John N Tsitsiklis. Simulation- optimization of markov reward processes. IEFE Transactions on Automatic Control, 46(2):191-209 2001.
[28] Ila R Fiete and Sebastian Seung. Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physicul Review Letters, 97(4)0048104 2006.
[29] Matthew D Golub, M Yu Byron, and Steven M Chase. Internal models for interpreting neural population activity during sensorimotor control. eLife, 4:e10015, 2015.
[30] Daniel N. Scott and Michael J. Frank. Beyond gradients: Noise correlations control hebbian plasticity to shape credit assignment. bioRxiv, 2021.
[31] Andrew Peters, Haixin Liu, and Takaki Komiyama. Learning in the rodent motor cortex. Annual Review of Neuroscience, 40:77, 2017.
[32] James S Albus. A theory of cerebellar function. Mathematical Biosciences, 10(1-2):25 61, 1971.
[33] John Porrill, Paul Dean, and Sean R Anderson. Adaptive filters and internal models: multilevel description of cerebellar function. Neural Networks, 47:134-149, 2013.
[34] Jennifer I. Raymond and Javier Medina. Computational principles of supervised learning in the cerebellum. Annual Review of Neuroscience, 41:233-253, 2018.
[35] Peter Redgrave, Tony Prescott, and Kevin Gurney The basal ganglia: a vertebrate solution to the selection problem? Neuroscience, 89(4): 104991000-1023, 1999.
[36] Vivek R Athalye, Jose M Carmena, and Rui M Costa. Neural reinforcement: re-entering and refining neural dynamics leading desirable outcomes. Current Opinion in Neurobiology, 60: 154, 2020.
[37] John W Krakauer, Alkis M Hadjiosif, Jing Xu, Aaron L. Wong, and Adrian M Haith. Motor learning. Comprehensive Physiology, 0000661-663, 2019.
[38] Jay A Hennig, Emily Oby, Darby M Losey, Aaron Batista, M Yu Byron, and Steven M Chase. How learning unfolds in the brain: toward an optimization view. Neuron, 2021.
12

---

[39] Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea. Eli- gibility traces and plasticity on behavioral time scales; experimental support of neohebbian three-factor learning rules. Fronfiers in Neural Circuits, 12:53, 2018.
[40] Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of Machine Learning Research, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See, the Discussion
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We emphasize important assumptions in the main text, and derive more extensive proofs in the Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are included in the Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]] Code included as supplementary material
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] All simulations were run on CPU
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "k59hCcLXDy", "reviewer": "Reviewer_Jh97", "summary": "This paper introduces a novel metric for distinguishing biological learning rules from changes in neural activity.  The proposed metric is the correlation between the observed change in network activity and the predicted change in network activity for each learning rule under consideration.  Predictions are made by assuming that the brain can be modeled as a vanilla RNN, and by then deriving the expected change in neural activity for each learning rule of interest.  The authors show that their metric allows them to distinguish a biologically-plausible variant of backpropagation (\"RFLO\", Random Feedback Local Online) from the REINFORCE learning rule of Williams (1992) in simulation for a wide variety of parameter settings.", "questions": "My questions are mostly clarification questions:\n- Linearity is assumed to get to equation 4.  Can you justify why this is a valid thing to do both in the simulated settings you consider, as well as how this assumption may or may not be valid when you move to real data.\n- On line 170, you say that $M$ can be readily obtained from experimental data.  It was not obvious to me how you would do this.  My understanding is the $M$ is the matrix that the subject (e.g. monkey) $\\it{thinks}$ maps neural activity to the cursor location, as opposed to the true readout matrix.\n\nWhile answering these questions would help me better understand the paper, the biggest thing that would make this paper a clear accept for me would be a demonstration of the metric to real world data.", "limitations": "The authors were upfront about the limitations of their work, namely that:\n- They only demonstrated the ability of their metric to distinguish two different learning rules (RFLO and REINFORCE)\n- They only demonstrated the application of their metric in simulation", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clarity: this paper, with the few exceptions noted below, is well written. The figures were clear and easy to understand.\n- Quality: the paper is rigorous and examines the usefulness of the proposed metric in a wide range of settings and in cases where the initial assumptions are invalid.  (As an aside: you might want to come up with a name for your metric!  The ML field loves an acronym and right now, I am not sure how best to reference it, other than \"proposed metric\".)\n- Significance: the paper tackles an important problem in the computational neuroscience community, namely that of identifying biological learning rules from neural activity.  As noted by the authors (line 64), there is a lot of interest in proposing biologically plausible learning rules.  Having new ways to distinguish competing hypotheses is useful.\n- Originality: to the best of my knowledge, this paper tackles an open-problem in the computational neuroscience community with a novel approach.\n\nWeaknesses:\n- Significance: the major limitation of this paper is the lack of application to real data.  The authors demonstrate the usefulness of their metric only when applying it to data simulated from an RNN.  Thus, it is hard to judge how useful their metric will be in real-world scenarios,  where some of the assumptions made by the authors may not hold.  For example, in lines 170-172, the authors say: \"By averaging over noise, we effectively assume that the cumulative updates during learning that make up differ only in having different noise realizations, but are otherwise in a consistent direction.\"  Real world data is always messy, and I would be surprised if this assumption holds there.\n\nMinor:\n- Clarity: there's a typo in Equation 1, where the superscript on y is $t$, but I think it should actually be $t-1$ (as in Equation 15 in the supplement).  For equation 1, neither $\\tau$ or $\\phi$ are explicitly defined.  For reproducibility purposes, it would be useful for these quantities to be defined and for their values to be explicitly provided. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jtHQyhnNI9X", "reviewer": "Reviewer_VxtR", "summary": "In this paper, the authors propose a method for distinguishing two biologically-relevant learning rules (unsupervised- and reinforcement-learning) based on neuronal activities in a recurrent neural network throughout an experiment similar to conventional studies with brain-machine interfaces. The authors verify their theoretical results in a series of simulated experiments where they confirm the model’s ability to distinguish the considered learning rules under various assumptions of different strengths.", "questions": "After reading the manuscript, I have a few minor questions remaining.\n\n-the correlations of the observed and predicted changes in the flows (delta F observed and delta F predicted; Equation 5) are evaluated using the network’s states h visited in the experiment. Why? Is it because such states h are considered representative for the distribution of the possible states of the network, or is there another logic involved? Please clarify.\n\n-Unless I’m missing something, in the model, all the network’s states h are considered observable and the only input x to the model is considered to be the task-relevant visual input. In the actual biological experiments, however, it’s hard to imagine recording from an entire functional circuit. Moreover, due to the unknown wiring of neurons on the individual-cell level, it may be hard to know what part of the functional circuit is being recorded. Therefore, it seems reasonable to assume that i) there’s an additional input to the network and ii) parts of the network’s activations are not observed. What modifications, if any, should be introduced to the proposed algorithm to account for such omission of the data?\n\n-Although it’s outside the scope of this work, it would be great to see some estimate of how much data is needed to perform the proposed distinction on the biological data. How this volume of the data compares to the existing datasets and which ones can be used to run the proposed algorithm?\n\n-Figure 3(E) caption: looks like it should say RL, not SL.\n\n-The paper’s title is slightly confusing as there is no actual BMI data for now.\n", "limitations": "The limitations appear to be described/addressed fully. Overall, I think that this work is a thorough study of a problem highly relevant to the NeurIPS community.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The goal of this paper is extremely worthy: whereas there’s plenty of theoretical work proposing various biologically plausible learning rules, little consideration is given to verifying these theories in practice using brain data. To address this issue, the authors here focus on the two prominent learning rules (i.e. the representative algorithms for supervised and reinforcement learning), for which they derive a theory and perform synthetic tests allowing distinguishing such rules based on typically-available data.\n\nThe paper has multiple strengths. The text is clearly written and well structured; the math is fully described in the appendix (requires no further reading!) and appears correct. The two considered learning approaches are representative of the learning algorithms, provided that the learning algorithms are often split into supervised, unsupervised, and reinforcement learning. The hypotheses in this work are introduced gradually (e.g. in the initial theory and simulations the credit assignment matrix M is considered known, while later on, this assumption is relaxed to “correlated with the real one”, and then to “learned from the data”). This approach streamlines the reading of the paper and also helps estimate the proposed method’s sensitivity to various parameters in the data. To this end, the authors propose the criteria of the method’s applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BeeVVG38BK", "reviewer": "Reviewer_GGdc", "summary": "The authors provide a framework to distinguish biased vs unbiased learning rules by observing only the unit activity and demonstrate the usability of the framework for recurrent neural networks. The authors demonstrate that observing the changes in flow field is indicative of the learning rule that leads to synaptic weight changes and can thus be used to distringuish if the learning rule was biased or unbiased wrt the true gradient signal. Subsequently, the results show that the framework can distinguish the two classes of learning rules when there is sufficient bias in one of them, i.e. the similarity between the credit assignment mapping and the true readout weights is low. Furthermore, they add more factors, namely the feedback weights, anisotropic noise as well as a changing credit assignment mapping matrix and show that the proposed framework can identify bias in the learning rule. Given the results, it would be interesting to see how this framework extends to realistic setting with neural data from BMI experiments.", "questions": "1. See weakeness point 1\n2. I am unsure why the RL line in Fig 2F fluctuates with changing $\\hat{M}$. In my understanding $\\hat{M}$ is only used by the SL predictions and therefore I can understand the different values of the SL line.\n3. While computing $\\Delta F$, don't you need the same $h$ for both $F_late$ and $F_early$? Is this achieved in the current setting by setting the t to be the same for early and late stages? How can this be achieved in the realistic scenario?\n4. In the RHS of Eq 4, I am assuming by $h$, you mean $h^{t}$. Is that correct?\n5. The relation between $\\Delta F$ and $\\Delta W^{rec}$ only holds when it is assumed that $W^{in}$ or $W^{fb}$ is unchanged. Is that a reasonable assumption for realistic BMI settings? If yes, kindly add citations. If not, I'd suggest adding that to the limitations section of your work. \n6. Furthermore, the relation between $\\Delta F$ and $\\Delta W^{rec}$ assumes same values of $x,y$. Assuming that the early and late stages of your estimation deals with similar inputs (hence same $x$), the output of the system should surely change given that there is some learning. So isn't it unreasonable to assume that $y$ would be same in both early vs late stages?\n7. It is a bit hard to grasp the correlation values in the plots. Although the relative values are higher for the correct learning rule, the absolute values are not very informative. Is it possible to add some sort of \"noise ceiling\" to indicate what could be the best achievable correlation value from theory such that it is possible to compare the efficacy of the framework in correctly identifying the ground truth learning rule?\n8. The framework seems to be able to distinguish between two learning rules when one has significant bias and the other does not. Can it also provide an indication as to what amount of bias is the \"correct\" amount of bias in the weight updates wrt the gradient? Given the current results, it seems unlikely but I am curious to know if you have further insights into this. ", "limitations": "Overall, I think the authors do a commendable job in presenting their framework. If the issues highlighted above can be addressed, I feel this framework could have significant impact in evaluating the plausibility of different learning rules in the brain. Currently, this work has certain limitations:\n1. A major limitation is the presentation of the flow field change as a way to understand the change in synaptic weights and how the correlation metric is a good summary metric that is indicative of this change. Specifically, I feel it is not immediately clear how the absolute values can be used to identify the correct learning rule or indicate if a learning rule is not plausible. \n2. If the authors choose to present their framework as being a good way to compare the biological realism of learning rules, I feel they would need to add more learning rules and show how the metric can either rank the \"closeness\" to the true learning rule or select the correct learning rule from several candidate learning rules. Although this could involve significant experimentation, I feel this would be helpful in establishing the credibility of the framework as a way to order the different proposals currently acknowledged in the field.\n3. The framework lacks analysis of high variance learning rules, which would be crucial in establishing the tradeoff/impact of bias and variance in bio-plausible learning rules and thereby guide the field into proposing learning rules with specific constraints in mind. \n4. It would be great if the authors could add a potential simulation of how the framework would perform in realistic BMI settings where the recording SNR could be a major bottleneck. However, I understand that this point is more a future work and might not fit in the current scope, if the authors choose to present it as such.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using flow field changes to infer changes in synaptic weight is neat and seems promising. The authors explain the key learning rules that they explore as well as the flow field well.\n2. Overall, the paper is well-motivated and well-presented. It is easy to understand the main goal and the general approach of the paper.\n3. The authors systematically add various aspects and complexities of the system, which provides an insight into how the different components interact with the proposed framework.\n4. The results are encouraging enough for the community to extend this work and either use it to analyze other learning rules or use it for BMI experiments.\n\nWeakness:\n1. I felt the later half of the results section a bit hard to read or follow. Specifically, it was not very clear how the solution to the autoregression problem in h translates to being the mapping from h to F(h). If $h^{t+1} = A h^{t}$, then shouldn't $F(h^{t}) = h^{t+1}-h^{t} = (A-I)h^{t}$, i.e. shouldn't the mapping from $h$ to $F(h)$ be $(A-I)$ instead of $A$ ?\n2. The correlation metric values are not setting invariant, i.e. the absolute values are not necessarily indicative of the correct learning rule. For instance, the correlation values for the RL-predicted flow changes are similar at low values of $sim(M,(W^{bmi1})^T)$ and $sim(\\hat{M},(W^{bmi1})^T)$ in Fig 2C and 2F respectively, although the ground truth learning rule is different. Therefore, it is not very clear how to use the value in realistic experimental settings to identify the correct learning rule. In my understanding, the metric can be used to compare between candidate learning rules but given that correlation values of the ground truth can also be as low as 0.25, it is unclear how reliable it would be for other learning settings. \n3. This work provides a comparison of two learning rules - biased Supervised Learning and unbiased Reiforcement Learning. The proposed framework is able to distinguish (barring some specific cases) between the two rules. However, it is unclear how noise in the learning rule would impact the framework. Specifically, the authors used a noise-averaged version of the RL weight update as the learning rule. But in realistic scenario, weight updates could be characterized by noise around the true gradient direction (high variance update as noted in Fig 1). It is unclear if the framework is relevant for distinguishing such learning rules.\n4. The plots are missing statsitical significance test to infer if the metric is statistically significant in the cases that the authors claim te usability of the framework and if it is statistically insignificant in the cases where the authors claim the framework is not usable. Having the statistical tests will also allow to characterize the limits of the framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0pslcBXGOHQ", "reviewer": "Reviewer_SqGf", "summary": "This paper presents a method for distinguishing learning rules in the brain using data observed via brain-machine interface (BMI) experiments. The attempt to distinguish learning rules is motivated by the well-known fact that backpropagation is biologically implausible due to (potentially among other reasons) the weight-transport problem.\n\nThe weight transport problem is especially acute when fitting to data from BMI experiments because the weights mapping neural activity read by the BMI to behavior exhibited by the subject may change abruptly and cannot be immediately estimated for global error propagation. This approach therefore requires setting up a machine learning experiment that avoids the weight transport problem. \n\nThe authors model a BMI task using a vanilla recurrent neural network (RNN) with input weights W_in, recurrent weights W_rec, feedback weights W_fb, and BMI decoder weights W_bmi. W_bmi is fixed and learning only occurs in W_rec. In one set of experiments, the RNN is trained with supervised learning, specifically the Random Feedback Local Online (RFLO) rule that avoids the weight-transport problem by assuming that the weights carry an imperfect approximation of the ideal credit assignment mapping and dropping local terms from the weight update. This is expected to be a biased estimator. In the other experiments, the RNN is trained with reinforcement learning with node perturbation and subtracting off a baseline, yielding a somewhat noisy but unbiased estimator. Running these training processes yields a synthetic version of the data that would come from a BMI.\n\nThe two different learning paradigms are distinguished via a metric based on vector field flows over training. The authors hypothesize that supervised learning will have bias emerge over the course of training because of RFLO while reinforcement learning will not, making them distinguishable. They present the results of the training performance and distinguishability via flow field-based metrics. \n", "questions": "-\tWhy does the weight matrix mapping neural activity onto cursor position change abruptly?\n-\tHow much does the systematic bias claim extend to anything beyond RFLO vs. node-perturbed RL?\n-\tHow does the experimenter know the decoder? In general, more background on BMIs would be useful – not because I doubt the claims, but because it would help to have a better mental model.\n-\tHow is the assumption about partial alignment with W_bmi^T justified? For those not deeply familiar with RFLO, could you expand a little bit on this detail?\n", "limitations": "The experimental and ethical limitations are stated sufficiently. I consider there to be more limitations, but these are likely the products of a difference of opinion between myself and the authors. The fact that they aren’t here is not a knock on the quality of this section. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Edit: author comment + context from other reviews is more convincing of the value of this experiment for the field. Raising score.\n\n**Strengths**\n*Originality*: This paper takes the well-known problem of understanding and modeling biological learning rules and the existing tool, brain-machine interfaces, and creates a novel application. This application requires creative training formulations and metrics to test and evaluate. To my knowledge, this is novel – the problem and tool exist, but in different fields. The application is non-trivial and unique. The authors show a representative sample of related prior work regarding investigating biological learning rules from experimental data, including the most similar studies I know of (Lim et al. and Nayebi et al.) and distinguish from them. Furthermore, they present some related work on modeling BMI experiments with RNNs; I am not aware of any BMI work that is more related than this. \n\n*Quality*: This paper justifies all its small claims (i.e. details) correctly. The experiment setup with supervised learning and reinforcement learning seems correct to my understanding, the hypothesis about bias specifically from RFLO and noise without bias from node-perturbation-based RL are reasonable, results from execution make sense, and the metrics are presented and used to validate the approach convincingly. \n\nIn my opinion, the paper does not deliver on larger claims in a cascading way. First, the central hypothesis explaining the distinguishability seems to be that supervised learning (not just RFLO) will show bias emerge in training trajectories and reinforcement learning won’t. However, only one local supervised learning rule is shown and the explanation for bias emerging is specific to it. The related works cited here present other local supervised learning rules; they should at least be discussed to make a claim about supervised learning that avoids the weight-transport problem. This then cascades into a larger problem – being able to distinguish between some local supervised learning and some reinforcement learning doesn’t support the larger claim that this method can distinguish between learning rules, at least not in my opinion of the spirit of this claim. Finally, even if we assume these claims are supported, this cascades into the fact that we don’t know enough about biological learning rules to know that ability to distinguish between local supervised learning and reinforcement learning will be highly productive; there is no biological experiment to support this. The authors cite some related work about biological plausibility of each of these, but basing claims from purely simulated experiments on these fit experiments seems insufficient. \n\nEven if others disagree with me on the spirit of the claim the paper makes in its title, intro, and discussion, ultimately we have seen that simulated BMI experiments with flow field-based metrics can distinguish between (an example of) biased and unbiased learning trajectories. I consider this far from the scope of the main claims. \n\n*Clarity*: The writing in this paper is clear and useful for understanindg. I didn’t notice any concerning typos, and the ideas get across. The clarity suffers somewhat on the structure and figures. For structure, the paper basically has two qualitative sections (intro and conclusion, total ~2 pages) and one very long section (termed “results”) with everything else in it; the results only arrive in the second half, after methods have all been communicated. At the moment, the fact that “Results” are signposted before the reader knows experiment setup, the fact that the results don’t come until a long time after “Results” has begun, and the general lack of structure make this paper confusing and a bit overwhelming to read. It would be as simple as breaking this section into “Methods” and a couple results sections (e.g. one for performance, one for distinguishability) to fix that.\n\nThe figures contain the right information, but also aren’t clearly presented. Most of the figures contain many different subfigures, which is fine, but they aren’t well-annotated – it is helpful to show significant points on the charts directly. Furthermore, schematic parts of the figures (like all of figure 1) could benefit from less visual detail in the icons – lots of squiggly lines may look nice and even be more faithful to reality, but if they don’t improve our understanding of the point of the figure, they aren’t necessary and can be visually cluttered. \n\n*Significance*: As stated in the quality section, the claims of this paper are big and worthwhile. The setup and idea are also completely worthwhile: the idea of doing ML-analogous real experiments with BMIs is promising and this paper gives a good example of setting such a direction up. The fact that this setup led to conclusive experiments about learning rules may be significant enough to warrant publication (I’m unsure). But not only do the results not live up to the broad claims of the paper, they don’t make a very significant *result contribution* for machine learning for the reasons discussed in the quality section. We know simulated BMI experiments are a promising tool because of this paper, but not that they will provide publication-level value. Combined with the fact that this promise hasn’t been validated in real BMI experiments, I feel that we end up without a real “get” in this paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept