Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper authors study one possible incarnation of more biologically plausible learning scheme, akin to (direct) feedback alignment methods, where the error signal is linearly mapped onto the update direction, without reusing forward weight (so to break the weight sharing issue, that currently is believed to be impossible for brains). 
Authors approach can be summarised as a mixture of synthetic gradients with a noise-enriched estimation, rather than direct empirical risk minimisation.
Contributions are two fold, first, authors provide some theoretical analysis of the convergence of gradient estimator in a simplified setup, second, the noise-based estimator is empirically evaluated on 2 simple classification tasks.

Paper is well written, and easy to read, with relatively easy to follow notation (which is quite a tricky task for non-gradient based learning methods that require abandoning typical concept of loss minimisation and talking about dynamical systems instead).

I have a few critical comments, that I hope authors can address in the revised manuscript:
- first, high level thing, that seems to be missing from the manuscript is use of baselines that are actually co-adapted, rather than random (e.g. DFA and FA). To be more specific, in papers like "Sobolev Training for Neural Networks" (NeurIPS  2017, https://papers.nips.cc/paper/7015-sobolev-training-for-neural-networks.pdf) one can find 3 basic methods (all requiring one implementation, and differ only in terms of which loss is applied): at each layer h, the gradient predictor g(h, e, B) is composed of (d/dh) CE[softmax(Bh + c), y], which has an analytical form, is biologically plausible (as boils down to a simple affine transformation if h and y. This model can now be supervised in 3 ways: 
a) one can put supervision only on the gradient (and bootstrap from higher layers, if needed, as in FA or "fully decoupled" synthetic gradient model in Jaderberg et al.) [this is essentially "pure" SG from Jaderberg et al. but constrained to the conservative vector fields, so to guarantee convergence]
b) one can put supervision on loss itself matching the "topmost loss" (reminescent of DFA, where error propagation skips entire network) [called Critic Training, Critic Network etc. depending on the source]
c) twe two above can be used jointly (which leads to the full Sobolev training)
Neither of these methods have been compared, and in reviewer's opinion it is critical, as approaches are very similar, and while Czarnecki et al. was not analysing biological plausibility, the models proposed do satisfy the same constraints requested by authors of this paper. Note, that with critic learning on MNIST, one can easily get results matching backpropagation (which, in the current manuscript is claimed as an important property of the introduced method). It might also be important to show that well tuned linear model on MNIST can reach 95% test accuracy too.
- similarly, authors are not disentangling noise-induction, from the overall setup. Which of these two is the actual source of good results? Many previous works would rely on empirical risk minimisation, if one uses noised versions and train "regular" fully decoupled affine synthetic gradients, will the results be analogous? Will this improve the Sobolev training (if implemented and verified)? Given, that both methods are known object in the literature, providing understanding of which components are actually important (maybe it is only the combination that works?) would strengthen the claims.
- Analogously, to further decouple mixed effects, when authors claim that noise-based estimator performs better than adam in the autoencoding case, having an adam with artifically added same amount of noise to gradient estimates would be helpful for the reader to see if the difference lies in the "second order estimates" of Adam, or in simply regularising effects of adding noise.  In particular, note, that this is a well established result, that noising inputs to the neural network is equivalent (up to first order terms) to Tikhonov regularisation.
- the theoretical claim of Thm 1 is quite trivial, and while I really admire authors effort to provide theoretical grounding, it feels a bit overstated in the current form. What authors are showing, is that in the simple setup, where network is not learning, and all the errors are in the sense, independent, the linear predictor is consistent. This property is well known, and both assumptions used - never met in practise, consequently I would strongly suggest downplaying the claim, and stating these results in sentence or two, with proof moved to the appendix, as in the current version of the manuscript section 3 is presented as major contribution, rather than an interesting side note.


Minor comments:
- can authors provide some error intervals for results in Table 1? Results are so close, that claiming that 48 is even "marginally better" than 47 seems odd, unless many repeated runs were conducted and some confidence intervals can be provided?
- it is odd to say "using backpropagation OR Adam". Adam is just an optimiser, it still uses backpropagation (aka chain rule). Could the notation be unified so that when talking about comparing optimisers, authors state Adam and SGD (assuming that this is what is currently called backpropagation?) and when talking about chain rule, the name of backpropagation is there?


Overall I believe it is an interesting study, however, currently missing important baselines and ablations to be a good ICLR contribution. If authors are willing to add these, I will be happy to revise the score assigned.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: It's unclear how multi-layer biological neural networks could implement gradient-based learning, as they don't have the symmetric connections needed for backpropagation. This paper proposes a perturbation-based synthetic gradient estimator that does not rely on symmetric backward connections. Hidden unit perturbation is used to estimate the loss gradient, and backward connections are trained via gradient descent to predict the approximate gradients from the perturbation-based estimator.

The topic is important and the paper is well-written. I don't follow this area closely, but from what I can tell it's a novel idea. The results are strong. The method beats various alternatives and closely matches backpropagation in terms of performance on the MNIST tasks (less so on CIFAR). It's especially curious that the method performs better than backpropagation on the autoencoder task.

I have an unresolved question: What do the learnable backward connections add beyond the perturbation estimator for the gradients? If the perturbation-based estimator could be used to train the forward model, does the trained backward model have advantages in terms of efficiency or performance? I would appreciate more discussion of this key choice, or a direct comparison with only the perturbation-based estimator (only) to understand the differences.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method that addresses the "weight transport" problem [1] (not cited by the authors) emphasizing the biological infeasibility of artificial neural networks (ANNs) which are trained by gradients computed by the backpropagation algorithm [2a, 2b]. It is arguably the most eminent criticism (among many many others) when learning in the brain is modelled by such ANNs.

The authors cite only Rumelhart et al. (1986) for backpropagation (which is the same as the reverse mode of automatic differentiation), although Rumelhart cited neither Linnainmaa, the inventor of the method [2a], nor Werbos, who first applied it to ANNs [2b]. 

The authors propose to estimate the weights for the backward pass using a noise-based estimator and provide theoretical and empirical arguments. The proposed method is compared with backpropagation (the ground truth) and direct feedback alignment (DFA; which resorts to random matrices for the backward pass) on small fully-connected, convolutional, and fully-connected auto-encoder networks for MNIST, CIFAR10, and CIFAR100 and offers analysis based on insights from recent work.

Generally, this work is well-placed and the method straightforward and novel. That said, we found the paper to be difficult to parse with some questionable claims which we don't believe are sufficiently backed up by experiments. Due to the following issues, we think that the paper is not yet quite ready for acceptance.

1.) As the authors point out, the recent work by Akrout et al. [3] is very closely related but the paper is lacking a significant discussion let alone an empirical comparison. Extrapolating from the experiments in this paper we think that the presented method is likely going to underperform since Akrout et al. show on-par performance with backprop on imagenet.

2.) The paper appears to blend synthetic gradients [4a, 4b] and feedback alignment to a somewhat questionable degree. A connection that is usually not done in other recent papers about biologically feasible ANNs. And that for good reasons: synthetic gradients do not address the weight transport problem. They backprop the synthetic gradients in order to train the gradient estimators of the previous layer.

Also, the paper cites only the recent reference [4b] (2016) on synthetic gradients but not the original work [4a] (1990).

3.) The authors claim "Thus our method illustrates a biologically realistic way by which the brain could perform gradient descent learning". Even though their method is a more plausible way by which the brain could perform gradient descent learning, it is still far from realistic. The proposed method still implicitly "transports" a lot of information from the forward pass to the backward pass (such as a. topology or b. the derivative of the activation function or c. the activations of the forward pass (h^i) that are necessary to compute the gradient estimate in either case).

4.) The experimental results of the proposed method but also the baseline are simply too far from the state of the art. This is partially dismissed in section 5 due to regularization and data augmentation "tricks". We'd like to point out that feedback alignment has been reported to achieve <2% error on the MNIST dataset [6] which is significantly better than the results reported in this work without any tricks. Similar seems to be the case for CIFAR10 and CIFAR100. The reported results on CIFAR 10 and 100 are over 20% below the state of the art. Furthermore, many previous papers provide code which arguably makes such experiments particularly easy (see [7]). We do not expect the authors to provide state of the art results but they should be at least in the same ball-park.

5.) The authors claim "(...) this hybrid approach can solve large-scale problems." We don't think this is backed up by experiments as most models are rather small (all < 50k parameters except the CNN architecture) and evaluated on toy datasets (for vision standards; while underperforming as already mentioned).

6.) It is not obvious how the noise-based estimation will scale with width and depth. The authors make the observation that "The number of neurons has an effect". This can be seen in the appendix in Fig 4E where the relative error increases exponentially with the number of neurons in the first layer (note that the second layer doesn't increase since it remains unchanged). This seems to indicate that the method would not scale well.

7.) Why do the authors decide to train the MNIST networks with such a small learning rate (4e-4) for 2 million steps? Better results can be achieved with just a few thousand steps. Does the method work in the "general" setting?

8.) With regards to Figure 2C for layer 2: Given a random initialization of the backward pass for feedback alignment but also the presented method, we would expect that the sign congruence of the first few iterations to be the roughly the same on average. That doesn't seem to be the case according to the figure. Is there are an explanation for this observation?

9.) Why the sudden switch from feedback alignment to direct feedback alignment for the CNN experiment in section 4.3? Does the approach of section 4.2 not work with CNNs?

10) We hope the authors could clear up some confusion with regards to section 3 and the consistency proofs in the appendix.

10.a) Apparently, the notation slightly changes? In section 2, layers are indicated by 1 <= i <= N as the superscript. In section 3, i prevails but a new superscript 1 <= n <= N is used for the layers. Theorem 1 then states that the least-squares estimator for B^(i+1)\tilde{e}^{i+1} is (\hat{B}^{N+1})'. There seems to be a notation issue since it is our understanding that every layer should have its own estimated weights.

10.b) We believe A03 should state that E[\tilde{e}^i (\tilde{e}^i)'] is of full rank and not simply \tilde{e}^i (\tilde{e}^i)' assuming that \tilde{e} stands for the equivalent X of regular least-squares. From the main text (specifically the eq in section 2.1) one might assume that \tilde{e}^i are vectors but the proof seems to treat them like the design matrices X in the regular least-squares estimator.

10.c) It is awkward that in the proof of Theorem 1 and 2 the population size is suddenly T, the same symbol for the transpose.

10.d) The regular OLS estimator of the coefficients is b = (X'X)(X'y) where X is a full-rank design matrix and y are the targets. It is our understanding that \lambda, as defined by eq 12, is equivalent to the y of the regular OLS estimator. But in the following equation (the bottom of page 14), the W^{N+1} of \lambda is factored out, apparently assuming e^{N+1}e^{N+1}' = I. Maybe we made a mistake, we'd appreciate if the authors could clarify this step in the proof.

11.) Finally, we add some additional minor comments or sources of confusion:
- we'd appreciate if all equations were numerated
- In section 2, the definition of the loss is verbose and partially unnecessary. L(y,\hat{y}(x)) never appears in the text. Furthermore, \hat{y} is not introduced as a function so writing \hat{y}(x) is awkward.
- Similar for the equation in section 2.2.
- 4.1 first line: the tilde is misplaced
- Figure 3A, the black line has no label (assumed to be DAE which is missing)
- 4.2 first inline equation: the tilde is misplaced
- Table 1: no decimal values, not std. Not clear if the difference between DFA and note perturbation is significant.
- Figure 4A-E is missing the crucial comparison with feedback alignment


References:

[1] Grossberg, Stephen. "Competitive learning: From interactive activation to adaptive resonance." Cognitive science 11.1 (1987): 23-63.

[2a] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis, Univ. Helsinki, 1970. FORTRAN code on pages 58-60. See also BIT 16, 146-160, 1976.

[2b] Werbos, Paul J. "Applications of advances in nonlinear sensitivity analysis." System modeling and optimization. Springer, Berlin, Heidelberg, 1982. 762-770.

[3] Akrout, Mohamed MA, et al. "Deep learning without weight transport." Advances in Neural Information Processing Systems. 2019.

[4a] J.  Schmidhuber. Networks adjusting networks. In J. Kindermann and A. Linden, editors, Proceedings of `Distributed Adaptive Neural Information Processing', St. Augustin, 24.-25.5. 1989, pages 197-208. Oldenbourg, 1990. Extended version: TR FKI-125-90 (revised), Institut für Informatik, TUM. http://people.idsia.ch/~juergen/FKI-125-90ocr.pdf See section "An Approach to Local Supervised Learning in Recurrent Networks."

[4b] Jaderberg, Max, et al. "Decoupled neural interfaces using synthetic gradients." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.

[5] Czarnecki, Wojciech Marian, et al. "Understanding synthetic gradients and decoupled neural interfaces." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.

[6] Nøkland, Arild. "Direct feedback alignment provides learning in deep neural networks." Advances in neural information processing systems. 2016.

[7] https://paperswithcode.com/sota/image-classification-on-cifar-10?p=maxout-networks

For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.

Edit: After the rebuttal, we increased our score.
review_title: Official Blind Review #2

Meta-review: Initial reviews of this paper cited some concerns about a lack of comparison to SOTA and baselines, and also some debate over claims of what is (or is not) "biologically plausible."  However, after extensive back-and-forth between the authors and reviewers these issues have been addressed and the paper has been improved.  There is now consensus among authors that this paper should be accepted.  I would like to thank the reviewers and authors for taking the time to thoroughly discuss this paper.
Decision: Accept (Poster)