Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors in the paper propose a method to train multi-layer neural networks.
The standard back-propagation algorithm aims at computing the influence of the output
neuron in a step wise chain rule fashion. The alternative proposed in this paper
is to compute the influence of a mid-layer neuron on the output of the network
using REINFORCE algorithm.  The proposal is to treat each neuron as a separate
learning agent. The authors main contribution is to implement MAP propagation to
control variance that comes with REINFORCE algorithm. The initial set of results show
that this technique can be comparable to back-propagation in RL settings.
main_review: I don't see any obvious technical faults in the paper.

My question to the authors is around the larger goal of this paper.
I don't see why being close to biological learning,  is so important if the
only goal is to achieve lower training error.
limitations_and_societal_impact: The limitations have been addressed. I don't see any direct societal impacts in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work introduces a local learning rule (with a global reward signal). The basic idea is just a neural network with stochastic units and REINFORCE update each layer (that is, the REINFORCE gradient estimate is not backpropogated through the network, each layer computes an update weighted by the reward. This, by itself, is (as recognized in the paper) an old idea.

They key addition here is an approach to reducing the (prohibitive) variance of this approach by conditioning the update of a particular layer on the state and action (eq 5). This lower-variance update is then approximated by eq 6. One interpretation of this is that the activations of the layer being updated are shifted to be more compatible (lower free energy) with the nearby layers.

They try this approach on some standard simple RL benchmark tasks and find that it performs comparably to backprop but much better then naive REINFORCE (per layer) or a previous method for reducing variance in this situation.
main_review: Originality:
As the author's recognize, the idea of apply REINFORCE to every unit in a network is old. As far as I know the variance reduction approach used here is novel and the main contribution.

Quality:

The approach appears technically sound (I did not check the appendix carefully) and the experimental results reasonable, although only on relatively simple tasks.

The approach introduce here is interesting, because any new learning rule is interesting. But its main claimed benefits are that is more biologically plausible (than backprop) due to the use of only local information and a global reward. For this reason, it seems that other somewhat biological plausible approaches should have also been included as baselines, such as feedback alignment [11-13].

This submission would also be strengthened by discussing how ``biologically plausible'' this learning method is. For example, (approximately) taking the argmax (eq 8) is done layer-wise rather than unit-wise. Is there a plausible neural mechanisms for this? (I don't know the answer).

Clarity:

Overall the paper is well written.

One minor thing is that in some cases I think it would be helpful to be explicit about what the expectation is taken over.

Significance:

Learning rules that are competitive (in learning speed) with backprop while using only local information are of interest to the community. Based on the experiment here, this approach appears of interest. The interest to the neuroscience community would depend on how biologically plausible this learning rule is when examined in more detail.

Minor Issues:

$H$ is referred to the as the ``value'' of a layer. It might be clearer to call this the activation of a layer, as ``value'' could mean weight or activation.

I realize that MAP is a common abbreviation, but I still think that it would be ideal to write out the term in full once.
limitations_and_societal_impact: I don't think this paper raises particular concerns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a local learning rule as a biologically plausible alternative to backprop. The core idea is to train each neuron using REINFORCE and mitigate the high variance issue of REINFORCE using their novel MAP propagation algorithm. They also show how a similar method could be used to train a value function for an actor-critic algorithm. And they are able to demonstrate the effectiveness of their method on few tasks.
main_review: Introducing a biologically plausible alternative to backprop is a very active research area that would be beneficial in bridging the gap between neuroscience and artificial neural networks. And often the proposed alternatives are sub-optimal and not able to match the performance of backprop. The results presented in this work show that their proposed method is able to reliably work. And the algorithm is relatively easy to implement and does not impose many constraints on the network architecture, hence could be a work that the community can build upon it.

In the first paragraph of related work, is the main novelty of this work compared to the cited works is the application of MAP propagation to RL as opposed to (un)supervised learning? Would be great to elaborate more.  

While the text is well-written and I was able to follow the text, the main text is quite dense. Perhaps some restructuring and rewriting could make the paper more easily readable.

Please have a brief description of the multiplexer and scalar regression tasks to make the text self-sufficient. 

Due to inner loop optimization for each layer, there is added computational complexity. Could you please also have a comparison of the computational complexity of this method vs backprop?

The appending L:82 mentions that a different learning rate was used for each layer. This will result in an increase in the number of hyper-parameters with layer size. How difficult was it to tune each of these separately? 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new synapse-local method for reinforcement learning, that is similar to the REINFORCE algorithm, but reduces the high variance caused by REINFORCE's need to assess many stochastic samples, at the cost of added computation. 

IIUC, it does this by 1- making REINFORCE layer-local (Eq 2), 2- Replacing the gradient of probabilities of observed responses  by its expectation given the state and action (which is uncomputable) (Eq 4-5), and 3- Approximating this uncomputable expectation by replacing actual activations with their maximum a posteriori estimate, given the chosen action and observed state. The latter turns out to be computable, with some iterative process.

The algorithm is also shown to be equivalent, update-wise, to plain backpropagation with the reparametrization trick (this is how I understood Theorem 2 and Eq. 11, but I may be wrong, see below). 

Various experiments show that this method learns faster than REINFORCE and about as fast as backprop with the reparametrization trick, in terms of number of steps (though each step is computationally more intensive).
main_review: I found the paper interesting, though somewhat confusing to me, possibly due to my lack of expertise with the details of the algorithms.

To my knowledge, the method is novel, and addresses an important limitation of REINFORCE-like algorithms (the high variance and the difficulties this causes for learning). The *relative* biological plausibility (the rule is local to each neuron, though the actual mechanism seems somewhat involved for neurons to perform) is a nice bonus.

The experiments are well chosen and the results are informative.

The paper is well-written throughout, though could gain some clarity.

First and foremost, it would be *extremely helpful* to provide a very short, intuitive summary of the overall proposal and derivation (similar to the two first paragraphs in the Summary above, but of course more accurate) before going on with the full description, so that the reader would get a general idea of where this is going and why we're doing it before trudging through the whole derivation in section 3.1. 

The introduction mentions what MAP does (lines 41-42 of p. 2) , but not *why*. IIUC, it is so that we need fewer actual samples to estimate the gradients. In fact, the crucial justification for the whole approach seems to that the RHS in Eq. 4 should have lower variance than Eq. 5. It's not quite obvious why, and lines 116-117 in p.3 don't clarify much. The reason for the reduction of variance should be explained a bit better, possibly in an appendix section.

Also, when the introduction and abstract mention that the proposed approach is equivalent to backpropagation. IIUC, it should be "backpropagation with the reparametrization trick". If I am correct, this should be explicitly stated! (And if not, it should be clarified what exactly "backpropagation" means here).

A few other things that confused me:

* Eq 11 and Theorem 2 seem to assert that the method is equivalent to backprop with the reparametrization trick. But the experiments seem to show that the method has quite distinct properties, e.g. being better when more exploration is needed. Evidently I am missing something. Could this be clarified?

* For the regression part (p. 4, l. 146), I do not understand what information is "lost". The appendix seems to suggest a simple centering and normalization method to address this "loss". Again, this could be clarified.


limitations_and_societal_impact: Limitations of the algorithm, especially its higher computational cost, seem adequately stated (e.g. in last paragraph of section 3.2).
review_title: null

Meta-review: This paper considers the problem of biologically plausible learning rules. For this, it looks at a local learning rule that uses REINFORCE to learn local updates for individual neurons. The main contribution is a novel variance reduction scheme for REINFORCE. While there were some fundamental question raised on biologically plausible learning rules (which is an active area of research), the reviewers found this a significant contribution and the paper interesting. As such, I would recommend acceptance of this paper.
Decision: Accept (Poster)