Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for distinguishing learning rules in the brain using data observed via brain-machine interface (BMI) experiments. The attempt to distinguish learning rules is motivated by the well-known fact that backpropagation is biologically implausible due to (potentially among other reasons) the weight-transport problem.

The weight transport problem is especially acute when fitting to data from BMI experiments because the weights mapping neural activity read by the BMI to behavior exhibited by the subject may change abruptly and cannot be immediately estimated for global error propagation. This approach therefore requires setting up a machine learning experiment that avoids the weight transport problem. 

The authors model a BMI task using a vanilla recurrent neural network (RNN) with input weights W_in, recurrent weights W_rec, feedback weights W_fb, and BMI decoder weights W_bmi. W_bmi is fixed and learning only occurs in W_rec. In one set of experiments, the RNN is trained with supervised learning, specifically the Random Feedback Local Online (RFLO) rule that avoids the weight-transport problem by assuming that the weights carry an imperfect approximation of the ideal credit assignment mapping and dropping local terms from the weight update. This is expected to be a biased estimator. In the other experiments, the RNN is trained with reinforcement learning with node perturbation and subtracting off a baseline, yielding a somewhat noisy but unbiased estimator. Running these training processes yields a synthetic version of the data that would come from a BMI.

The two different learning paradigms are distinguished via a metric based on vector field flows over training. The authors hypothesize that supervised learning will have bias emerge over the course of training because of RFLO while reinforcement learning will not, making them distinguishable. They present the results of the training performance and distinguishability via flow field-based metrics. 

strengths_and_weaknesses: Edit: author comment + context from other reviews is more convincing of the value of this experiment for the field. Raising score.

**Strengths**
*Originality*: This paper takes the well-known problem of understanding and modeling biological learning rules and the existing tool, brain-machine interfaces, and creates a novel application. This application requires creative training formulations and metrics to test and evaluate. To my knowledge, this is novel – the problem and tool exist, but in different fields. The application is non-trivial and unique. The authors show a representative sample of related prior work regarding investigating biological learning rules from experimental data, including the most similar studies I know of (Lim et al. and Nayebi et al.) and distinguish from them. Furthermore, they present some related work on modeling BMI experiments with RNNs; I am not aware of any BMI work that is more related than this. 

*Quality*: This paper justifies all its small claims (i.e. details) correctly. The experiment setup with supervised learning and reinforcement learning seems correct to my understanding, the hypothesis about bias specifically from RFLO and noise without bias from node-perturbation-based RL are reasonable, results from execution make sense, and the metrics are presented and used to validate the approach convincingly. 

In my opinion, the paper does not deliver on larger claims in a cascading way. First, the central hypothesis explaining the distinguishability seems to be that supervised learning (not just RFLO) will show bias emerge in training trajectories and reinforcement learning won’t. However, only one local supervised learning rule is shown and the explanation for bias emerging is specific to it. The related works cited here present other local supervised learning rules; they should at least be discussed to make a claim about supervised learning that avoids the weight-transport problem. This then cascades into a larger problem – being able to distinguish between some local supervised learning and some reinforcement learning doesn’t support the larger claim that this method can distinguish between learning rules, at least not in my opinion of the spirit of this claim. Finally, even if we assume these claims are supported, this cascades into the fact that we don’t know enough about biological learning rules to know that ability to distinguish between local supervised learning and reinforcement learning will be highly productive; there is no biological experiment to support this. The authors cite some related work about biological plausibility of each of these, but basing claims from purely simulated experiments on these fit experiments seems insufficient. 

Even if others disagree with me on the spirit of the claim the paper makes in its title, intro, and discussion, ultimately we have seen that simulated BMI experiments with flow field-based metrics can distinguish between (an example of) biased and unbiased learning trajectories. I consider this far from the scope of the main claims. 

*Clarity*: The writing in this paper is clear and useful for understanindg. I didn’t notice any concerning typos, and the ideas get across. The clarity suffers somewhat on the structure and figures. For structure, the paper basically has two qualitative sections (intro and conclusion, total ~2 pages) and one very long section (termed “results”) with everything else in it; the results only arrive in the second half, after methods have all been communicated. At the moment, the fact that “Results” are signposted before the reader knows experiment setup, the fact that the results don’t come until a long time after “Results” has begun, and the general lack of structure make this paper confusing and a bit overwhelming to read. It would be as simple as breaking this section into “Methods” and a couple results sections (e.g. one for performance, one for distinguishability) to fix that.

The figures contain the right information, but also aren’t clearly presented. Most of the figures contain many different subfigures, which is fine, but they aren’t well-annotated – it is helpful to show significant points on the charts directly. Furthermore, schematic parts of the figures (like all of figure 1) could benefit from less visual detail in the icons – lots of squiggly lines may look nice and even be more faithful to reality, but if they don’t improve our understanding of the point of the figure, they aren’t necessary and can be visually cluttered. 

*Significance*: As stated in the quality section, the claims of this paper are big and worthwhile. The setup and idea are also completely worthwhile: the idea of doing ML-analogous real experiments with BMIs is promising and this paper gives a good example of setting such a direction up. The fact that this setup led to conclusive experiments about learning rules may be significant enough to warrant publication (I’m unsure). But not only do the results not live up to the broad claims of the paper, they don’t make a very significant *result contribution* for machine learning for the reasons discussed in the quality section. We know simulated BMI experiments are a promising tool because of this paper, but not that they will provide publication-level value. Combined with the fact that this promise hasn’t been validated in real BMI experiments, I feel that we end up without a real “get” in this paper. 

questions: -	Why does the weight matrix mapping neural activity onto cursor position change abruptly?
-	How much does the systematic bias claim extend to anything beyond RFLO vs. node-perturbed RL?
-	How does the experimenter know the decoder? In general, more background on BMIs would be useful – not because I doubt the claims, but because it would help to have a better mental model.
-	How is the assumption about partial alignment with W_bmi^T justified? For those not deeply familiar with RFLO, could you expand a little bit on this detail?

limitations: The experimental and ethical limitations are stated sufficiently. I consider there to be more limitations, but these are likely the products of a difference of opinion between myself and the authors. The fact that they aren’t here is not a knock on the quality of this section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a method for distinguishing two biologically-relevant learning rules (unsupervised- and reinforcement-learning) based on neuronal activities in a recurrent neural network throughout an experiment similar to conventional studies with brain-machine interfaces. The authors verify their theoretical results in a series of simulated experiments where they confirm the model’s ability to distinguish the considered learning rules under various assumptions of different strengths.
strengths_and_weaknesses: The goal of this paper is extremely worthy: whereas there’s plenty of theoretical work proposing various biologically plausible learning rules, little consideration is given to verifying these theories in practice using brain data. To address this issue, the authors here focus on the two prominent learning rules (i.e. the representative algorithms for supervised and reinforcement learning), for which they derive a theory and perform synthetic tests allowing distinguishing such rules based on typically-available data.

The paper has multiple strengths. The text is clearly written and well structured; the math is fully described in the appendix (requires no further reading!) and appears correct. The two considered learning approaches are representative of the learning algorithms, provided that the learning algorithms are often split into supervised, unsupervised, and reinforcement learning. The hypotheses in this work are introduced gradually (e.g. in the initial theory and simulations the credit assignment matrix M is considered known, while later on, this assumption is relaxed to “correlated with the real one”, and then to “learned from the data”). This approach streamlines the reading of the paper and also helps estimate the proposed method’s sensitivity to various parameters in the data. To this end, the authors propose the criteria of the method’s applicability.
questions: After reading the manuscript, I have a few minor questions remaining.

-the correlations of the observed and predicted changes in the flows (delta F observed and delta F predicted; Equation 5) are evaluated using the network’s states h visited in the experiment. Why? Is it because such states h are considered representative for the distribution of the possible states of the network, or is there another logic involved? Please clarify.

-Unless I’m missing something, in the model, all the network’s states h are considered observable and the only input x to the model is considered to be the task-relevant visual input. In the actual biological experiments, however, it’s hard to imagine recording from an entire functional circuit. Moreover, due to the unknown wiring of neurons on the individual-cell level, it may be hard to know what part of the functional circuit is being recorded. Therefore, it seems reasonable to assume that i) there’s an additional input to the network and ii) parts of the network’s activations are not observed. What modifications, if any, should be introduced to the proposed algorithm to account for such omission of the data?

-Although it’s outside the scope of this work, it would be great to see some estimate of how much data is needed to perform the proposed distinction on the biological data. How this volume of the data compares to the existing datasets and which ones can be used to run the proposed algorithm?

-Figure 3(E) caption: looks like it should say RL, not SL.

-The paper’s title is slightly confusing as there is no actual BMI data for now.

limitations: The limitations appear to be described/addressed fully. Overall, I think that this work is a thorough study of a problem highly relevant to the NeurIPS community.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a novel metric for distinguishing biological learning rules from changes in neural activity.  The proposed metric is the correlation between the observed change in network activity and the predicted change in network activity for each learning rule under consideration.  Predictions are made by assuming that the brain can be modeled as a vanilla RNN, and by then deriving the expected change in neural activity for each learning rule of interest.  The authors show that their metric allows them to distinguish a biologically-plausible variant of backpropagation ("RFLO", Random Feedback Local Online) from the REINFORCE learning rule of Williams (1992) in simulation for a wide variety of parameter settings.
strengths_and_weaknesses: Strengths:
- Clarity: this paper, with the few exceptions noted below, is well written. The figures were clear and easy to understand.
- Quality: the paper is rigorous and examines the usefulness of the proposed metric in a wide range of settings and in cases where the initial assumptions are invalid.  (As an aside: you might want to come up with a name for your metric!  The ML field loves an acronym and right now, I am not sure how best to reference it, other than "proposed metric".)
- Significance: the paper tackles an important problem in the computational neuroscience community, namely that of identifying biological learning rules from neural activity.  As noted by the authors (line 64), there is a lot of interest in proposing biologically plausible learning rules.  Having new ways to distinguish competing hypotheses is useful.
- Originality: to the best of my knowledge, this paper tackles an open-problem in the computational neuroscience community with a novel approach.

Weaknesses:
- Significance: the major limitation of this paper is the lack of application to real data.  The authors demonstrate the usefulness of their metric only when applying it to data simulated from an RNN.  Thus, it is hard to judge how useful their metric will be in real-world scenarios,  where some of the assumptions made by the authors may not hold.  For example, in lines 170-172, the authors say: "By averaging over noise, we effectively assume that the cumulative updates during learning that make up differ only in having different noise realizations, but are otherwise in a consistent direction."  Real world data is always messy, and I would be surprised if this assumption holds there.

Minor:
- Clarity: there's a typo in Equation 1, where the superscript on y is $t$, but I think it should actually be $t-1$ (as in Equation 15 in the supplement).  For equation 1, neither $\tau$ or $\phi$ are explicitly defined.  For reproducibility purposes, it would be useful for these quantities to be defined and for their values to be explicitly provided. 

questions: My questions are mostly clarification questions:
- Linearity is assumed to get to equation 4.  Can you justify why this is a valid thing to do both in the simulated settings you consider, as well as how this assumption may or may not be valid when you move to real data.
- On line 170, you say that $M$ can be readily obtained from experimental data.  It was not obvious to me how you would do this.  My understanding is the $M$ is the matrix that the subject (e.g. monkey) $\it{thinks}$ maps neural activity to the cursor location, as opposed to the true readout matrix.

While answering these questions would help me better understand the paper, the biggest thing that would make this paper a clear accept for me would be a demonstration of the metric to real world data.
limitations: The authors were upfront about the limitations of their work, namely that:
- They only demonstrated the ability of their metric to distinguish two different learning rules (RFLO and REINFORCE)
- They only demonstrated the application of their metric in simulation
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors provide a framework to distinguish biased vs unbiased learning rules by observing only the unit activity and demonstrate the usability of the framework for recurrent neural networks. The authors demonstrate that observing the changes in flow field is indicative of the learning rule that leads to synaptic weight changes and can thus be used to distringuish if the learning rule was biased or unbiased wrt the true gradient signal. Subsequently, the results show that the framework can distinguish the two classes of learning rules when there is sufficient bias in one of them, i.e. the similarity between the credit assignment mapping and the true readout weights is low. Furthermore, they add more factors, namely the feedback weights, anisotropic noise as well as a changing credit assignment mapping matrix and show that the proposed framework can identify bias in the learning rule. Given the results, it would be interesting to see how this framework extends to realistic setting with neural data from BMI experiments.
strengths_and_weaknesses: Strengths:
1. The idea of using flow field changes to infer changes in synaptic weight is neat and seems promising. The authors explain the key learning rules that they explore as well as the flow field well.
2. Overall, the paper is well-motivated and well-presented. It is easy to understand the main goal and the general approach of the paper.
3. The authors systematically add various aspects and complexities of the system, which provides an insight into how the different components interact with the proposed framework.
4. The results are encouraging enough for the community to extend this work and either use it to analyze other learning rules or use it for BMI experiments.

Weakness:
1. I felt the later half of the results section a bit hard to read or follow. Specifically, it was not very clear how the solution to the autoregression problem in h translates to being the mapping from h to F(h). If $h^{t+1} = A h^{t}$, then shouldn't $F(h^{t}) = h^{t+1}-h^{t} = (A-I)h^{t}$, i.e. shouldn't the mapping from $h$ to $F(h)$ be $(A-I)$ instead of $A$ ?
2. The correlation metric values are not setting invariant, i.e. the absolute values are not necessarily indicative of the correct learning rule. For instance, the correlation values for the RL-predicted flow changes are similar at low values of $sim(M,(W^{bmi1})^T)$ and $sim(\hat{M},(W^{bmi1})^T)$ in Fig 2C and 2F respectively, although the ground truth learning rule is different. Therefore, it is not very clear how to use the value in realistic experimental settings to identify the correct learning rule. In my understanding, the metric can be used to compare between candidate learning rules but given that correlation values of the ground truth can also be as low as 0.25, it is unclear how reliable it would be for other learning settings. 
3. This work provides a comparison of two learning rules - biased Supervised Learning and unbiased Reiforcement Learning. The proposed framework is able to distinguish (barring some specific cases) between the two rules. However, it is unclear how noise in the learning rule would impact the framework. Specifically, the authors used a noise-averaged version of the RL weight update as the learning rule. But in realistic scenario, weight updates could be characterized by noise around the true gradient direction (high variance update as noted in Fig 1). It is unclear if the framework is relevant for distinguishing such learning rules.
4. The plots are missing statsitical significance test to infer if the metric is statistically significant in the cases that the authors claim te usability of the framework and if it is statistically insignificant in the cases where the authors claim the framework is not usable. Having the statistical tests will also allow to characterize the limits of the framework.
questions: 1. See weakeness point 1
2. I am unsure why the RL line in Fig 2F fluctuates with changing $\hat{M}$. In my understanding $\hat{M}$ is only used by the SL predictions and therefore I can understand the different values of the SL line.
3. While computing $\Delta F$, don't you need the same $h$ for both $F_late$ and $F_early$? Is this achieved in the current setting by setting the t to be the same for early and late stages? How can this be achieved in the realistic scenario?
4. In the RHS of Eq 4, I am assuming by $h$, you mean $h^{t}$. Is that correct?
5. The relation between $\Delta F$ and $\Delta W^{rec}$ only holds when it is assumed that $W^{in}$ or $W^{fb}$ is unchanged. Is that a reasonable assumption for realistic BMI settings? If yes, kindly add citations. If not, I'd suggest adding that to the limitations section of your work. 
6. Furthermore, the relation between $\Delta F$ and $\Delta W^{rec}$ assumes same values of $x,y$. Assuming that the early and late stages of your estimation deals with similar inputs (hence same $x$), the output of the system should surely change given that there is some learning. So isn't it unreasonable to assume that $y$ would be same in both early vs late stages?
7. It is a bit hard to grasp the correlation values in the plots. Although the relative values are higher for the correct learning rule, the absolute values are not very informative. Is it possible to add some sort of "noise ceiling" to indicate what could be the best achievable correlation value from theory such that it is possible to compare the efficacy of the framework in correctly identifying the ground truth learning rule?
8. The framework seems to be able to distinguish between two learning rules when one has significant bias and the other does not. Can it also provide an indication as to what amount of bias is the "correct" amount of bias in the weight updates wrt the gradient? Given the current results, it seems unlikely but I am curious to know if you have further insights into this. 
limitations: Overall, I think the authors do a commendable job in presenting their framework. If the issues highlighted above can be addressed, I feel this framework could have significant impact in evaluating the plausibility of different learning rules in the brain. Currently, this work has certain limitations:
1. A major limitation is the presentation of the flow field change as a way to understand the change in synaptic weights and how the correlation metric is a good summary metric that is indicative of this change. Specifically, I feel it is not immediately clear how the absolute values can be used to identify the correct learning rule or indicate if a learning rule is not plausible. 
2. If the authors choose to present their framework as being a good way to compare the biological realism of learning rules, I feel they would need to add more learning rules and show how the metric can either rank the "closeness" to the true learning rule or select the correct learning rule from several candidate learning rules. Although this could involve significant experimentation, I feel this would be helpful in establishing the credibility of the framework as a way to order the different proposals currently acknowledged in the field.
3. The framework lacks analysis of high variance learning rules, which would be crucial in establishing the tradeoff/impact of bias and variance in bio-plausible learning rules and thereby guide the field into proposing learning rules with specific constraints in mind. 
4. It would be great if the authors could add a potential simulation of how the framework would perform in realistic BMI settings where the recording SNR could be a major bottleneck. However, I understand that this point is more a future work and might not fit in the current scope, if the authors choose to present it as such.
review_title: null

Meta-review: metareview: This paper explores the question of experimentally distinguishing between different hypothesized classes of learning rules in the brain (specifically biased supervised learning and unbiased reinforcement learning). It derives a metric to distinguish between such learning rules based on changes in neural activity seen during learning with a brain-computer interface. The authors show that this metric can be used to identify which learning rules are the best account of the observed activity changes.

The reviewers agreed that this paper makes an original and important contribution to the field, and the decision to accept was unanimous.


confidence: Certain
Decision: Accept