Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes an backpropagation-free algorithm for training in deep neural networks.
The algorithm called DeepAGREL does not use the chain rule for computing derivatives and instead is based on direct reinforcement. 
Authors conduct experiments on MNIST and CIFAR where they find their method to reach performance similar to BP.

Novelty: the method is built upon existing ideas, however, the exact algorithm seems to be novel.

Clarity: although it is possible to understand the algorithm based on the provided textual description, it would not hurt to provide a more formal presentation of the main update equation (1). 

It is not clear why authors provide two different expressions for updates in each layer (in terms of fb^{l} and fb^{l+1}).

I would also appreciate a clear discussion on the theoretical properties of the algorithm, for example, is it guaranteed to converge to a critical point of some loss function? Can we derive the update rule from some known optimisation procedure, e.g. REINFORCE?

Quality: Unfortunately I have a number of major issues with this respect.

1. Authors do not clearly pose what are the properties of back propagation that they find biologically non-plausible and how exactly their algorithm is making progress on them. There seems to be some sort of consensus in the literature about these properties and if authors accept it, then it looks like DeepAGREL is not very plausible too, because it does not address the weight-transport issue at all.
2. Many recent results are not even mentioned in the paper, for example, (Bartunov et al, 2019). From not so recent — the whole branch of research around target propagation algorithm (Lee et al, 2014).
3. The experiments are not convincing to me, as the considered network architectures are quite shallow and thus the ability to perform credit assignment can not be demonstrated. Besides that, the locally-connected architecture has only the first layer locally-connected and it does not make much sense to me, as it does not factor out the impact of weight sharing. 

Overall, I think the paper is not ready for publication at ICLR.

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary:

This paper generalizes the AGREL biologically plausible learning algorithm to deeper networks. It presents experiments to show that the method can scale to medium-sized tasks with only a modest slow down in training speed and essentially no change in classification performance.

Major comments:

This is an interesting paper which shows that a particular biologically plausible learning method can attain comparable performance to gradient descent on small and mid-size visual classification tasks. The proposed mapping to the biology is of a different flavor from many other recently proposed approaches. 

It is striking that the CIFAR10 network trains about as fast as the EBP network, while the CIFAR100 network trains much more slowly. This is presumably because randomly guessing the right answer out of 100 possibilities is the bottleneck. The paper could be strengthened by studying the speed of learning as a function of the number of output classes. For this approach to scale to ImageNet, with a 1000-way classification (or to our human visual recognition abilities with far more classes), this is an important scaling dimension to consider. 

It should be noted that other biologically plausible schemes like feedback alignment were able to solve CIFAR and other smaller image classification tasks, but struggled when applied to the larger scale ImageNet problem. The paper could be improved by pointing to this limitation of the present work, the possibility that performance could change on larger tasks, and the need to conduct larger experiments in future work.

Personally I think the statement at the beginning of the introduction that only RL occurs in animals and humans is overstated. Unsupervised learning occurs in some form in critical period plasticity, and in various unrewarded statistical learning paradigms, at a minimum.

I would also caution against categorical statements of biological plausibility, for instance, in saying that shared weights in convolutions are biologically implausible (bottom of pg 6). The same criticism has been leveled at gradient descent learning, and as the present submission shows, these intuitive judgements can be misleading.

The distinction between RL and supervised learning is a bit blurred in the present submission because it is considering a classification setting in which exactly one out of a number of possible outputs is rewarded on each trial. This looks very similar to the supervised learning scenario, and relies on stochastic softmax-like competition to select a single output. This approach is very reasonable for mutually exclusive classification tasks, but this is a small subset of the tasks that full EBP could be applied to. 

The paper is fairly clear but the explanation of the algorithm could be further streamlined and condensed. Is DeepAGREL equivalent to stochastically selecting a single unit in a softmax layer and then back propagating only its error? It seems so from what I understand, and this may be a straightforward way to explain the algorithm. 

Typos:

The text on page 7 describes MNIST performance as 99.17% but the table has 99.16%.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents DeepAGREL, a framework for biologically plausible deep learning that is modified to use reinforcement learning as a training mechanism. This framework is shown to perform similarly to error-backpropagation on a set of architectures. The idea relies on feedback mechanism that can resemble local connections between real neurons.

This paper is an interesting approach to provide a reinforcement learning paradigm for training deep networks, it is well written and the experiments are convincing, although more explanation about why these specific architectures were tested would be more convincing. I also think the assumptions about feedback connections in real neurons should be visited and more neuroscientific evidence from the literature should be included in the paper. Do we expect feedback to happen at each level of a neuron-neuron interaction and between each pair of connected neurons? Is there a possibility that feedback is more general to sets of neurons, or skips entire layers of neurons? I think more neuroscience background would help this paper (and others on the topic). Nonetheless, I think the paper does offer an interesting proposal of a more biologically plausible form of deep learning.

review_title: Official Blind Review #2

Meta-review: The paper proposes an RL-based algorithm for training neural networks that is able to match the performance of backprop on CIFAR and MNIST datasets.

The reviewers generally found the algorithm and motivations interesting, but some had issue with the imprecision of the notion of "biologically plausible" used by the authors. One reviewer had issues with missing discussion of related work and also doubts about the meaningfulness of the experiments, since the networks were quite shallow.

For this type of paper, clarity and precision of exposition is crucial in my opinion, and so I recommend rejection at this time, but encourage the authors to take the feedback into account and resubmit to a future venue.
Decision: Reject