Learning representations for binary-classification without backpropagation
Authors: Mathias Lechner
Venue:  (2020)
Keywords: feedback alignment
alternatives to backpropagation
biologically motivated learning algorithms

Abstract: The family of feedback alignment (FA) algorithms aims to provide a more biologically motivated alternative to backpropagation (BP), by substituting the computations that are unrealistic to be implemented in physical brains.
While FA algorithms have been shown to work well in practice, there is a lack of rigorous theory proofing their learning capabilities.		
Here we introduce the first feedback alignment algorithm with provable learning guarantees. In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classification tasks.
We show that our FA algorithm can deliver its theoretical promises in practice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classification tasks.
Finally, we demonstrate the limits of our FA variant when the number of output neurons grows beyond a certain quantity.

Published as a conference paper at ICLR 2020
LEARNING
FOR BINARY- CLASSIFICATION WITHOUT BACKPROPAGATION
REPRESENTATIONS
Mathias Lechner
IST Austria
Am Campus 1, Klostemeuburg. Austria
m mlechnere tist...c act.ac.at
ABSTRACT
The family of feedback alignment (FA) algorithms aims provide a more bio- logically motivated alternative to backpropagation (BP), by substituting the com- putations that are unrealistic be implemented in physical brains. While FA algorithms have been shown to work well in practice, there is lack of rigorous theory proofing their learning capabilities. Here we introduce the first feedback alignment algorithm with provable learning guarantees. In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classification tasks. show that our FA algorithm can deliver its theoretical promises in prac- tice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classification tasks. Finally, we demonstrate the lim- its of our FA variant when the number of output neurons grows beyond certain quantity.
INTRODUCTION
key factor enabling the successes of Deep Learning is the backpropagation of error (BP) algo- rithm (Rumelhart et al. (986). Since it has been introduced, BP has sparked several discussions on whether physical brains are realizing BP-like learning or not (Grossberg, 1987 Crick 1989). Today, most researchers consent that two distinct characteristics of BP render the idea of BP based learn- ing brains as implausible: 1) The usage of symmetric forward and backward connections and 2) the strict separation of activity and error propagation (Bartunov al. 2018). These two objections have lead researchers to search for more biologically motivated alternatives to BP.
The three most influential families of BP alternatives distilled so far are Contrastive Hebbian Learing
(CHL) (Movelann 1991). target-propagation (TP) (LeCun 1986; Hinton 2007 Bengio 2014) and
feedback Alignment (FA) (Lillicrap al. 2016).
The idea of CHL is to propagate the target activities, instead of the errors, backward through the network. For this reason, temporal dimension is added the neuron activities. Each neuron then adapts its parameters based on the temporal differences of its "forward" and "backward" activity. The two significant critic points of CHL are the requirement for symmetric ffowwrrd-bckwaan connections and the use of alternating "forward" and "backward" phases (Baldi & Pineda 1991
Bartunov al. 2018).
TP shares the idea with Contrastive Hebbian Learning of propagating target activities instead of er- rors. However, rather than keeping symmetric forward and backward paths, the reciprocal propaga- tion of the activities are realized through learned connections. Consequently, each layer has assigned two objectives: Learning the inverse of the layer's forward function and minimizing the difference to the back-projected target activity. Variants of TP differ in how exactly the target activity is projected backward (LeCun 1986 Bengio 2014; Bartunov et al. 2018). Theoretical guarantees of TP rely on the assumption that each reciprocal connection implements the perfect inverse of the corresponding forward function. This issue of an imperfect inverse was also found to be the "bottleneck" of TP in practice (Bartunov al. 2018). When the output of layer has significant lower dimension than its input, reconstructing the input from the output becomes challenging, resulting in poor learning performance.

---

Published as a conference paper at ICLR 2020
Feedback alignment algorithms eliminate the weight sharing implausibility of BP by replacing the symmetric weights in the error backpropagation path by random matrices. The second objection, ie., separate activity and enror chamnels. is attenuated by Direct Feedback Alignment (Nokland) 2016) which drastically reduces the number of channels carrying an error signal. While feedback alignment algorithms work well small and medium-sized benchmarks, recent study identified that they are unable to provide learning on more challenging datasets like ImageNet (Bartunov al., 2018). Another criticism FA algorithms the lack of rigorous mathematical justification and convergence guarantees of the performed computations.
In this work, we investigate feed-forward networks where the weights of all, expect the first, layers are constrained to positive values. We prove that this constraint does not invalidate the universal approximation capabilities of neural networks. Next, we show that, in combination with monotonic activation functions, all layers from the second layer realize monotonically increasing functions. The backpropagation of a scalar error signal through these layers only affects the magnitude of the error signal but does not change its sign. Consequently, we prove that algorithms that bypass the error backpropagation steps, such as Direct Feedback Alignment, can compute the sign of the true gradient with respect to the weights of our constraint networks without the need for backpropaga- tion. Finally, we show that our algorithm, which we call monotone Direct Feedback Alignment, deliver its theoretical promises in practice, by surpassing the learning performance of existing feedback alignment algorithms in binary classification task, i.e., when the error signal is scalar, and provide decent performance even when the error signal is not scalar.
We make the following key contributions:
First FA algorithm that has provable learning capabilities for non- non-linear networks of arbi- trary depth
Experimental evaluation showing that our FA algorithm outperforms the learning perfor- mance of existing FA algorithms and match backpropagation in binary classification tasks
We make an efficient TensorFlow implementation of all tested algorithms publicly avail- a
BACKPROPAGATION AND FEEDBACK ALIGNMENT
We consider the feed-forward neural network
[fWWhi-1++ bi)
ifl < L ifl=L
hy 1h) Wihh bt
ho
(I)
Wi -
by E R
where ff the non-linear activation function, x the input and h2 the output of the network. For classification tasks, hy is usually transformed into a probability distribution with discrete support by a sigmoid or softmax function.
During training, the parameters Wi, 1 I. are adjusted to minimize loss function Cly. hr)) on samples of giving training distribution p(y, x). This is usually done by performing gradient descent
dC
01 + 0, - a d
a R
(2)
with respect to the parameters 0, [ b)), 11. n of the network.
httass//ggtabbbccc////e22222///1
2

---

Published as a conference paper at ICLR 2020
a) BP
b) FA
DFA
d) mDFA
y
y
y
y
ha
3
W
3
B3
W
3
B2
B2
h2
22
W
22
B2
22
W
B
B
h1
W1
W1
W1
Wi
ho
Figure 1: Graphical overview of various error transport methods, adapted from Nøkland 6016) Black arrows represent the forward path, whereas green arrows represent the eгror feedback. Weights that are learned are denoted as W; and fixed weights are denoted by B, Weights that are restricted only positive values are annotated by a + a) Backpropagation. b) Feedback alignment, c) Direct Feedback Alignment, d) monotone Direct Feedback Alignment.
2.1 BACKPROPAGATION
Backpropagation (Rumelhart et al., 1986) is the primary method to compute the gradients needed by the updates in equation ( by iteratively applying the chain-rule
dc dhu T d.
(3)
d8,
db, dh
d.C
= [dbe+1 dhi
(dhh )
dC
(4)
dhu
dhi+1
dhi+1 Wi+idiag(f (f + b(+1)).
(5)
dhi
A graphical representation of how information first flow forward and then backward in BP through each layer shown Figure IT]
Two major concerns argue against the idea that biological neural networks are implementing BP- based learning. The weight matrix Wi of the forward path reused in the backward path in the form of WT (weight sharing), and II) the strict separation of activity carrying forward and error carrying backward connections (reciprocal error transport).
2.2 FEEDBACK ALIGNMENT ALGORITHMS
Feedback alignment addresses the implausibility of reusing WT in the backward path by replacing W by fixed random matrix B, Lillicrap et al. (016) showed that this somewhat counterintuitive approach works remarkably well in practice. The term "feedback alignment" originates from the observations made by Lillierap et al. (2016) that the angle between the FA update vector and the true gradient starts to decrease, i.c.. align, after a few epochs of the training algorithm. Theoretical groundwork on this alignment principle of FA relies on strong assumptions such lincarized network with one hidden layer (Lillicrap et al. 2016)).
FA avoids any weight sharing but does not address the reciprocal error transport implausibility, due to its strict separation of forward and backward pathways, as shown Figure (T) b. Direct Feedback Alignment (DFA) (Nolland 2016) relaxes this issue by replacing all backward paths with direct feed from the output layer error- groor-gradient dCC Consequently, there is only a single error signal that is dhz distributed across the entire network, which is arguably more biologically plausible than reciprocal error connections. The resulting parameter updates of DFA are of the form
dC
if
80,
dhz.
dC
B
hy
i
dhi
3

---

Published as a conference paper at ICLR 2020
where Bi I is random matrix. A graphical schematic of DFA is shown in Figure (I Similar to FA, DFA shows a decent learning performance in mid sized classification tasks (Nokland) 2016), but fails
complex datasets such ImageNet (Bartunov et al. 2018). Theory on adapting the alignment principle to DFA shows that under the strong assumptions of constant DFA update directions and layer-wise criterion minimization, the DFA update vector will align with the
true gradient (Nokland 2016 Gilmer et al. 2017).
Recently, Frenkel et al. (2019) proposed to combine ideas from feedback alignment and target- propagation in their Direct Random Target Projection (DRTP) algorithm. While DRTP shows decent empirical performance, theoretical guarantees about DRTP rely on linearized networks.
2.3 SIGN-SYMMETH ALGORIHHMM
Liao al. 2016) introduced the sign-symmetry algorithm, hybrid of BP and FA. Sign-symmetry locks the signs of the feedback weight Bt to have the same signs as W?", but random absolute value. The authors showed that this approach drastically improves learning performance compared to standard FA. Furthermore, Moskovitz et al. (2018) and Xiao et al. (2019) demonstrated that the sign-symmetry algorithm is even able to match backpropagation for training deep network architec- tures and large datasets such as ImageNet.
While these empirical observations suggest that the polarity of the error feedback is more important than its magnitude, the mathematical justification of sign-symmetry remains absent. Similar to FA, sign-symmetry relaxes the strict weight sharing implausibility, but still relies unrealistic reciprocal error transport.
3 MONOTONE DIRECT FEEDBACK ALIGNMENT
In this section, we first introduce a new class of forward networks, where all, except the first, layers are constrained to realize monotone functions. We call such networks mono-nets and show that they are as expressive as unconstrained feed-forward networks. Next, we prove that for our mono-nets with single output tasks, e.g., binary-llasification. feedback alignment algorithms pro- vide the sign of the gradient. The sign of the gradient is interesting for learning, as it tells us if the value of a parameter should be increased or decreased in order to reduce the loss. At the end of this section, we will highlight similarities to algorithms from literature, which can provide resilient learning by only relying on the sign of the gradient.
Neural networks with monotonic constraints have been already studied in literature You et al. 2017], however not the context of learning algorithms.
Definition 1 (mono-net). A mono-net is feed-forward neural network with L layers h hL each layer I composted of nu units and the semantics
h1(h1-1) = 1(Wihi1++b bt)
i
(7)
Wh4
=
ho = x
(8)
Wi E RRII
(9)
Wi R - for 1>
(10)
b E R
(11)
where R,, are the positive reals, i.e. R,, = Ex R/ 2 0}. f is non-linear monotonic increasing activation function, x the input and hr. the outpul of the network.
The major difference between mono-net and general feed-forwart neural networks is the restriction to only positive weight values in layers from the second layer on. Combined with the monotonic increasing activation function, this means that each layer hu(h)-)), 2 2 realizes monotone in- creasing function. Because functional composition preserves monotonicity, the complete network up to the first layer
h o h1-1 o ... h2(h1)
(12)
implements a monotone increasing function.
4

---

Published as a conference paper at ICLR 2020
mono-nets are Universal Approximators At first glance, this restriction seems counterproductive, as it might interfere with the expressiveness of the networks. However, we proof in Theorem | that universal approximators, meaning that we can
mono-nets with tangent hyperbolic activation approximate any continuous function arbitrarily close. A potential drawback of the monotonicity constraint is that we might need a larger number of units in the hidden layers to achieve the same expressiveness as general feed-forward network, as illustrated in our proof of Theorem !|
Theorem (mono-nets are Universal Approximators), Let In be the n-dimensional unit hypercube [0, 10" and C(I,) denote the set of continuous functions f In + R. We define II as the supremum norm of E (T,)) over its domain I,- For any given f E ((T)) and > 0, there exist function m In + R of the form
m(x) Σ B tanh( Tx + + (-x) + +,) +
(13)
with UE R, M 02 CCER," th R, "...R C R and M such that
[|m[x) (z) <.
In essence, the set of functions m(x) of the form given in 19 dense in C(In).
Proof. See supplementary materials
mDFA provides the sign of the gradient Here, we prove that for 1-dimensional outputs DFA applied a mono-net, which we will call simply mDFA, provides the sign of the true gradient.
Note that we focus our methods on DFA instead of "vanilla" FA, due to the superiority of DFA in terms of biological plausibility and empirical performance (Nøkland 2016 Bartunov et al.) 2018).
Theorem (For 1-dimensional outputs mDFA computes the sign of the gradient). Let Cly, hr) be
loss function, m(x)= == hr. e h .... h2 hı o ha(x)) be mono-net according Desinition 7 with parameters θ == [Wi, bi|l 1.... L.. We denote 60 the update value computed by mDFA ac for any 0 (Wi,b) bi} with SIS If 1, it follows that and V as the gradient 30
(5
(4
20,
(14)
ijj
i5
for each coordinate (i.5) of
Proof. See supplementary materials.
A graphical illustration of how activities and errors propagate in mDFA is shown in Figure (T) d.
Literature on learning by relying only on the sign of the gradient Two learning concepts related mDFA are RPROP (Riedmiller & Braun 1993 Riedmiller 1994) and signSGD (Bernstein al.) 2018). RPROP aims to build a more resilient alternative to gradient descent by decoupling the amplitude of the gradient from the step size of parameter updates. In essence, for each coordinate RPROP adapts the step size based on the sign of the most recent gradients computed. |Riedmiller &
Braun 1993) showed that their approach could stabilize the training of a neural network compared to standard gradient descent.
Performing gradient descent with taking the sign of each gradient coordinate is on an algorithmic level equivalent to the well-known steepest descent method with L. norm (Boyd & Vandenberghe) 2004 Bernstein et al. (2018). signSGD (Bernstein et al. 2018) studies convergence properties of the stochastic approximation of this algorithm.
What about networks with more than one output neuron? Theorem 2 applles only to networks with scalar output. As natural consequence, may ask whether such thcoretical guarantees can be extended to more dimensional output variables. The simple answer is, unfortunately not. In the supplementary materials section A.3 we provide counterexample showing that Theorem 2 naively extended to two output neurons does not hold anymore.
We want to note that the requirement of neural network to have only single output neuron is biologically unjustified. It is known that sub-circuits of biological neuronal networks can feed to multiple motor neuron groups Cook et al. (2019)
Cooo
5

---

Published as a conference paper at ICLR 2020
How does mDFA relate to the non-negative matrix factorization? A seemingly related concept to mDFA is the non-negative matrix factorization (NMF) algorithm. NMF decomposes an observation matrix V into weight matrix W and latent variable matrix H such that V x W H. In contrast to other decomposition--asee unsupervised learning methods, all three matrices V. W and H are restricted non-negative entries. While NMF model data that is inherently non-negative, such has semantic features of images and text, effectively Yuan & Oja (2005); Shahnaz et al. (2006), the method is unable learn subtractive and non-linear structures that are present in the data Lee & Seung 1999).
comiinon--egativy matrix factorization Ding et al. (2008) relaxes the original restriction to non- negative observations of NMF, by only constraining the weight matrix W to be non-negative. Deep semi-NMF Trigeorgis et al. (2014) further enhances the expressiveness of NMF by adding multiple layers and non-linearities between them to the decomposition.
Concerning this work, the semantics of mono-nets from the second layer on equivalent to that of deep semi-NMF models. However, the unconstrained first layer of mono-nets provides universal approximation capabilities, enabling mono-nets to learn subtractive and non-monotonic input de- pendencies. Moreover, while deep NMF models are mostly trained via layer-wise learning in an unsupervised context Trigeorgis et al. (2014); Yu et al. (2018), the sole purpose of mono-nets is to investigate alternatives to backpropagation for training multi-layer classifiers.
EXPERIMENTS
So far, we have only proven the learning capabilities of mDFA. What remains unclear is whether mDFA can deliver its theoretical promises in practice. In this section, we experimentally evaluate the learning performance of mDFA set of empirical benchmarks. We aim to answer the following two questions:
How well does mDFA perform compared to DFA, FA, and backpropagation in "natural conditions," i.c., in binary classification tasks, and
how much does the performance of mDFA degrade in multi-class classification tasks?
Our performance measure is the achieved classification accuracy of a network trained by a particular method. First, we report the highest achieved accuracy on the training set, which tells us how well the algorithm could fit the model to the training data. Secondly, for each method, we tuned the hyperparameters on a separate validation set and selected the best performing configuration to be evaluated on the test data. The obtained test accuracy tells us how well the model generalizes to data outside the training set.
We evaluate fully-connected networks (FC) and Convolutional networks (CNNs) in form of modified all-convolutional--ee (Springenberg et al.) 2013) with tangent hyperbolic, ReLU (Nair Hinton 2010), and the hard-tanh non-lincarity. The hard-tanh function is defined as
hard-tanh(r)) == min(max(x, - 11,,
(15)
Hyperparameters For all training methods, we fixed the batch size to 64, applied no regularization, no normalization, and no data augmentation. Optimizer, i.e., ("Vanilla" Gradient Descent, Adam Kingma & Ba, 2014), Rmsprop (Tieleman & Hinton 2012) ). learning rate, training epochs, and weight initialization method were tuned on the validation set. We tested three different weight initialization schemes; all zeros, a scaled uniform, and normal distribution. Note that all zeros was only tested on the forward weights. Our uniform initialization followed the methodology of Nokland (2016), i.e., scaling the bounds of the distribution inversely by the square-root of the number of incoming connections of a neuron. In order to comply with the weight constraints mono-nets, for mDFA the lower bound of the uniform distribution was set to = 10-3 Moreover, for mDFA we post-processed the initial weights of the normal distribution by taking their absolute values. Input variables are scaled to [0,1]. Detailed network architectures and brief discussion about the best performing hyperparameter configurations are listed the supplementary materials in section B. and section B.3

---

Published as a conference paper at ICLR 2020
4.1 BINARY CLASSIFICATION
We created a series of binary classification benchmarks by randomly sampling two classes from the well-studied CIFAR-100 and ImageNet datasets. We then train and test 1-vs-1 classifier the samples of the two classes. For each dataset, we create five such "binarized" sub-datasets and report the mean and standard deviation over these experiments. CIFAR-100 (Krizhevsky al. 2014) poses a challenging classification dataset, consisting of 32-by-32 RGB images in T00 real-world object categories. ImageNet (Russakovsky et al. 2015) large-scale object classification benchmark and de-facto standard to asses new deep learning methods. Each sample is high-resolution image representing one out of 1000 possible object classes. We pre-processed all samples by cropping and resizing each image to 224-by-228 pixels. Because ImageNet lacks a public test set, we will report the validation accuracy, i.e., as was reported by Bartunov et al. (2018).
The results on the binarized benchmarks are shown in Table D for CIFAR-100 and Table 2 for ImageNet. mDFA could bring the training error to zero for fully-connected networks, and match the test/validation accuracy of backpropagation for convolutional networks.
Poor learning performance for ReLU networks One surprising characteristic in our results is that mDFA fails to provide the same level of learning performance as backpropagation for ReLU net- works. Recall that Theorem D proves the universal approximation capabilities only for mono-net with tanh activation function. A mono-net with ReLU non-linen non-linearity restricts both, the activation and the weight values, to positive values. These constraints arguably limit the approximation ca- pabilities of mono-net in combination with ReLU and thus explain the poor performance of mDFA for ReLU networks. We could validate our hypothesis by testing a rectifier non-linearity that also contains negative values in its image. Coincidentally, the hard-tanh function matches exactly this criterion. Therefore, the decent learning performance of mDFA for hard-tame networks confirms our hypothesis. Note that also FA and DFA expressed an improvement in performance when switching from ReLU to hard-tanh activation. This observation suggests that feedback alignment algorithms in general benefit from symmetric activation functions.
Discrepancy between fully-connected and convolutional networks Though mDFA achieved the same training accuracy as backpropagation for fully-connected networks, the generalization ability, i.e, test and validation accuracy, slightly lacks behind BP in our Sinarized-CIre CIFAR-10 benchmark. This observation aligns with the studies by Nokland (2016); Bartunov et al. (0188). For convolu- tional neural networks, this effect is reversed, i.e., decent test/validation accuracy but higher training error than BP. We speculate that the two tested initialization schemes for the feedback weights caused this discrepancy. Both backpropagation and feedback alignment have been shown to be sensitive to the employed initialization method (Zhang et al.) 2019 Bartunov et al.) 2018). The restriction positive values of the weights used in mDFA, requires re-thinking of the initialization methods examined in the literature. However, we will leave this study open for future work.
Activation
Model
Training accuracy (FC)
Test accuracy (FC)
Training accuracy (CNN)
Test accuracy (CNN)
BP
100.0 + 0.0%
81.3 +: 11.4%
100.0 + 0.0%
8 10.4%
DFA
84.1 8.2%
76.8 11.5%
84.4 + 7.9%
80.0 10.3%
hard-tanh
FA
85.3 - 8.0%
76.9 + 12.0%
82.5 # 10.2%
77.5 12.2%
mDFA
100.0 + 0.0%
77.3 + 12.2%
87.4 + 7.0%
80.9 9..2%
BP
100.0 0.0%
80.0 10.8%
100.0 0.0%
82.2 9.5%
DFA
7.66%
9.29%
73.0 + 15.5%
ReLU
FA
77.9110.29
71.0 12.0%
68.6 + 15.9%
69.8 15.7%
mDFA
62.1 + 12.7%
78.0 + 11.7%
75.5 11.7%
BP
100.0#0.089 0.0%
81.4 = 10.5%
100.0 = 0.0%
81.1 11.9%
DFA
84.2 + 8.2%
77.5 + 11.2%
81.6 + 9.3%
79.7 + 10.5%
tanh
FA
84.9 8.0%
77.6 + 12.2%
94.2 + 3.5%
83.4 8.7%
mDFA
100.0 + 0.0%
77.2 + 12.2%
89.0 + 6.0%
81.1 9.8%
Table 1: Accuracies binarized-CIFAR-I Mean and standard deviation

---

Published as a conference paper at ICLR 2020
Activation
Model
Training acc. (FC)
Validation acc.
Training acc.
Validation acc.
(FC)
(CNN)
(CNN)
BP
98.4 1 3.6%
77.3 6.8%
99.8 1 0.5%
81.7 11.6%
DFA
93.8 = 13.0%
74.0 = 8.5%
100.0 = 0.0%
83.0 + 8.6%
hard-tanh
FA
97.3 = 0.8%
74.7 = 7.7%
51.9 0.5%
54.7 4.6%
mDFA
95.3 + 10.4%
79.5 + 9.8%
79.5 + 9.8%
83.8 + 7.2%
BP
100.0 + 0.0%
78.3 + 7.7%
90.0 = 18.1%
72.5 + 13.7%
DFA
75.1 + 8.1%
73.2 8.0%
50.4 0.9%
52.7 2.4%
ReLU
FA
68.3 + 7.9%
71.0 + 6.7%
52.3 0.9%
54.8 + 2.1%
mDFA
56.9 + 0.9%
59.7 2.4%
52.6 1.6%
58.4 + 4.5%
BP
99.9 = 0.2%
75.8 = 7.7%
0.1%
76.2 = 9.8%
DFA
97.9 + 2.2%
76.8 7.2%
99.3 = 1.5%
83.2 = 8.8%
tanh
FA
98.1 + 2.3%
76.5 7.2%
62.4 + 17.3%
60.8 + 13.8%
mDFA
99.9 + 0.1%
78.0
6.8%
78.2 + 9.4%
83.0 6.8%
Table 2: Accuracies on binarized BraiizedddImaeNeet. Mcan and standard deviation
Set
Classes
BP
DFA
FA 79.0 6.1%
mDFA
100.0 = 0.0%
50.5 9.5%
53.3 + 8.9%
100.0 + 0.0%
33.8 $ 9.7%
68.4 6.0%
34.1 9.8%
Training
10
100.0 = 0.0%
2.8%
52.4
25.9 1.2%
25
99.9 + 0.1%
1..4+
31.3 1.6%
13.1 1.4%
50
99.7 + 0.1%
0.1%
20.0 1.1%
+
77.4 8.7%
49.1
7
50.9 7.2%%
68.5 8.1%
3 11.6%
60.7
$3.9
Test
57.3 +: 4.9%
25.0 + 3.2%
46.9 + 3.8%
26.3 + 2.3%%
25
41.2 + 2.6%
13.4 + 1.9%
28.2 + 1.4%
13.1 + 1.3%
50
31.8 + 1.8%
8.8 + 0.4%
18.1 + 0.9%
8.0 0.4%
Table 3: Mulit-class classification accuracies fully-connected network with tanh activation on n-class CIFAR-100. Mean and standard deviation.
4.2 MULTI-CLASS CLASSIFICATION
Here we modify the classification benchmark creation procedure used above, by randomly sampling n classes from the datasets instead of just two. Due to their compelling results in our binary classi- fication benchmark,
restrict our evaluation to networks with tanh activation. Table and and Table 4
show the results on our n-class CIFAR- 100 benchmark for fully-connected and CNNs respectively. The results on n-class ImageNet can be found in the supplementary materials in Table
mDFA can provide learning for networks with more than one output neuron Though we do not have complete theory mDFA for multi-dimensional outputs, our experiments indicate that mDFA can provide learning for networks with more than one output neuron. particular, the achieved accuracies of mDFA outperforms other feedback alignment methods for convolutional networks with ten or fewer output neurons. However, mDFA falls behind standard DFA for networks with more than output neurons. This observation suggests that our restriction positive weights can be beneficial even for multi-class tasks but eventually hurts learning performance when the number classes grows larger.
Feedback alignment algorithms, in general, tend to struggle with increasing output dimension Our results express the trend that with an increase the number of classes, feedback alignment algorithms struggle to fit the training data. While BP can reduce the training error to almost zero independently of the output dimension, the training errors achieved by FA algorithms are signifi- cantly higher and correlate with the number of output neurons. Our observations suggest that the dimension of the еттог signal affects the training convergence for FA algorithms, in contrast to BP, which appears to be less affected by the dimension of the error signal. This relation potentially provides a step towards understanding why FA algorithms fail on challenging datasets as described by Bartunov al. (2018).
8

---

Published as a conference paper at ICLR 2020
Set
Classes
BP
DFA
FA
mDFA
3
100.0 = 0.0%
86.3 + 5.0%
84.1 = 10.2%
100.0 1 0.0%
100.0 + 0.0%
$4.6 4.6%
74.3 + 8.6%
100.0 0.0%
Training
10
100.0 = 0.0%
33.0%
57.9 3.4%
94.4 1.7%
25
= = 0.0%
81.0 1.5%
36.3 = 1.8%
64.8 1.4%
50
98.9 + 0.0%
80.5 + 1.0%
22.7 + 1.4%
4.5+-099%
3
76.2 + 10.1%
79,8 6.2%
79.6 9.7%
82.0 92.077.6%%
65.4 + 9.8%
7.6%
70.1 9.0%
73.3 + 8.8%
Test
10
57.5 + 5.2%
65.0 + 4.8%
55.1 + 4.1%
63.9 + 4.2%
25
41.8 + 2.2%
48.3 + 2.2%
35.0 = 1.4%
44.0 + 2.5%
50
33.3 + 1.5%
37.1 + 1.5%
22.3 + 1.1%
34.5 + 1.5%
Table 4: Mulit-class classification accuracies of Convolutional Neural Network with tanh activation on n-class CIFAR-100. Mean and standard deviation.
CONCLUSION
Feedback alignment algorithms are promising biologically motivated alternatives to backpropaga- tion. While existing literature provides empirical evidence that FA algorithms can work well in practice, there is still a lack of rigorous theory formalizing their learning capabilities. Here we con- tributed the field of biologically motivated learning algorithms, by introducing the first feedback alignment algorithm that provable provides learning for non-linear networks of arbitrary depth and single output neuron. We showed that our FA algorithm outperforms existing FA algorithms in binary classification tasks, and provide decent learning multi-class problems.
Limitations We demonstrated on empirical benchmarks as well as theoretical examples that our method is limited to networks with scalar output. Indeed, uncovering the mathematical principles behind the decent learning performance of FA and DFA on multi-class tasks remains an open chal- lenge.
this really useful? In terms of scientific significance. this work provided theoretical contributions toward understanding the capabilities and limits of feedback alignment algorithms. From practical point of view, our mDFA algorithm has an advantage over backpropagation concerning training latency, all weight updates be computed in parallel, i.e., Figure Furthermore, have shown that mDFA is superior to other feedback alignment algorithms in binary classification tasks. Consequently, mDFA provides effective solution for binary classification problems with training latency constraints. Dynamic branch prediction in microprocessor pipelines poses such problem instance, where program specific binary branch outcomes, i.e., branch taken/not taken, need to be learned in real-time. Because of this real-time constraint, existing branch predictors often employ only shallow perceptron modules (Jiménez & Lin 2002 Egan et al., 2003). mDFA could enable deeper branch predictor networks to be learned in real-time.
ACKNOWEEDGMENTS
This research was supported in part by the Austrian Science Fund (FWF) under grant 7211-N23 (Wittgenstein Award).
REFERENCES
Pierre Baldi and Fernando Pineda. Contrastive learning and neural oscillations. Neurul Computa- lion, 3(4):526-545, 1991.
Sergey Bartunov, Adam Santoro. Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivate deep learning algorithms and archi- tectures. In Conference on Neural Information Processing Systems (NeurIPS). 2018.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target 2014.
propagation. arXiv preprint

---

Published as a conference paper at ICLR 2020
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning (ICML), 2018.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Steven J Cook, Travis A Jarrll,, Christopher A Brittin, Yi Wang, Adam E Bloniarz, Maksim A Yakovlev, Ken CQ Nguyen, Leo T-H Tang, Emily A Bayer, Janet S Duerr, et al. Whole-animal connectomes of both caenorhabditis elegans sexes. Nature, 571 571 2019, 2019.
Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129-13333 1989.
Chris HQ Ding, Tao Li, and Michael Jordan. Convex and semi-nonnegative matrix factorizations. IEEE transactions on pattern analysis and machine intelligence, 32(1):45-55, 2008.
Colin Egan, Gordon Steven, Patrick Quick, Rubén Anguera, Fleur Steven, and Lucian Vintan. Two- level branch prediction using neural networks. Journal of Systems Architecture, 49(12-15):557- 570, 2003.
Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Direct random target projection as a Beedback-alignmen algorithm with layerwise feedforward training. arXiv preprint urXiv:1909.013) 2019.
Justin Gilmer, Colin Raffel, Samuel S Schoenholz, Maithra Raghu, and Jascha Sohl-Dickstein. Ex- plaining the learning dynamics of direct feedback alignment. International Conference Learn- ing Representations (ICLR)- Workshop track, 2017.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cog- nitive science, 11():233-63, 1987.
Geoffrey Hinton. How to do backpropagation in brain. In Invited talk at the NIPS2007 Deep Learning Workshop, volume 656, 2007.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-335) -366, 1989.
Daniel A Jiménez and Calvin Lin. Neural methods for dynamic branch prediction. ACM Transac- tions on Computer Systems (TOCS), 2064):369-397, 2002.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1:Xiv:1412.6980 2014.
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-100 dataset. online: http://www. cs. toronto. edu/kriz/ifar. html, 2014.
Yann LeCun. Learning process in an asymmetric threshold network. In Disordered systems and biological organization, pp. 233-240. Springer, 1986.
Daniel D Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factor- ization. Nature, 4076555))7788, 1999.
Qianli Liao, Joel Z. Leibo, and Tomaso Poggio. How important is weight symmetry in backpropa- gation? In AAAI Conference on Artificial Intelligence (AAAI), 2016.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7, 2016.
Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott. Feedback alignment in deep convo- lutional networks. arXiv preprint 1rXiv:18/2.000 06488, 2018.
10

---

Published as a conference paper at ICLR 2020
Javier R Movellan. Contrastive hebbian learning in the continuous hopfield model. In Connectionist models, pp. 10-17. Elsevier, 1991.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML), 2010.
Arild Nøkland. Direct feedback alignment provides learning deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2016.
Martin Riedmiller. Rprop-description and implementation details. Citeseer, 1994.
Martin Riedmiller and Heinrich Braun. direct adaptive method for faster backpropagation learn- ing: The rprop algorithm. In IEEE international conference on neural networks, 1993.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 323(6088):533, 1986.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22 252, 2015.
Farial Shahnaz, Michael W Berry, V Paul Pauca, and Robert J Plemmons. Document clustering using nonnegative matrix factorization. Information Processing & Management, 42(2))373-386, 2006.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In International Conference Learning Representations (ICLR), 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26- 31, 2012.
George Trigeorgis, Konstantinos Bousmalis, Stefanos Zafeiriou, and Bjoemn Schuller. A deep semi- nmf model for learning hidden representations. International Conference on Machine Learn- ing. pp. 1692-1700, 2014.
Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. piologically-plussibee learning al- gorithms can scale large datasets. International Conference on Learning Representations (ICLR), 2019.
Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
Jinshi Yu, Guoxu Zhou, Andrzej Cichocki, and Shengli Xie. Learning the hierarchical parts of objects by deep non-smooth nonnegative matrix factorization. IEEE Access, 6:58096-588 105, 2018.
Zhijian Yuan and Erkki Oja. Projective nonnegative matrix factorization for image compression and feature extraction. In Scandinavian Conference Image Analysis, pp. 333-344 342. Springer, 2005.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations (ICLR), 2019.
11

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 MONO-NETS ARE UNIVERSAL APPROXIMATORS
We denote
1
(xx) 1 exp(-xi)
the sigmoid function. the hyperbolic tangent and
tanh = 6" -
e² + e-
(1 if expr is true
1 [erpr] = (0 if ezpr is false
the indicator function.
From the definitions above we can derive the equalities
(xx
2 1 + tanh _
2))
(16)
tanh(x) tanh(-x)
(17)
which will need for proof.
Lemma (Universal Approximation Theorem). For any given f E C(Ta) and e > 0, there exist
function g In + of the form
N g(x) = U:o x + b.)
(18)
with B., u4 U ER, E and N < such that
(1(()) [( <.
In essence, the set functions g(x) of the form given in (18) dense in C((n)..
Proof. See Homii al. (1989
Theorem (mono-nets are Universal Approximators). For any given f C(1,) and e >> 0, there exist a function m : In - IR of the form
M
m(x)= = Σ tanh(w] x + bb) +
(19)
with R, M w. E R" D M and M oo such that
[l([x) fx))l <.
In essence, the set functions m(x) of the form given in (19 is dense in C(In).
Proof. By Lemma 1 we know that there exist g(x) with l1g(x) - f(x)l < that has the form
((x) Σ uoo + b.)
We will show that we can reformulate g(x) to the form in equation 19
Our basic idea to propagate all negative weight entries into to the first layer where negative weight values are allowed.
12

---

Published as a conference paper at ICLR 2020
g(x)
(w) x x+ b,)
N
(1
aan x bi)
2
N
+ 1 N
2
tanh
Wig + bi)
Σ 2
2 8mml
jml
N
1/; 2 0/012 tanh Σ
Wijjt + bi)
jml
+
Σ
1[0; <0jv; tanh
1
( Wi + bs)
2
1 N
22
2
i=1
[ 2 0j0 tanh (Σ Wis + be)
[[e 011222 tanh
Wissi b:)
1/s
2 1=1
Tte tanh (ws? T x + tbb)
2 tanh (w.Txx + (w( +
+
1=1
with
6 = [m 2 0/0 2 0
8/ 1/: < 0001 20
T
6. b;
Therefore, showed that there exist m(x) with a form as in equation 19 that satisfy
[l([x))
(x) <
This proof showed us how to translate any neural network with sigmoid activation function and one hidden layer of size N into mono-net with 2NN hidden units.
13

---

Published as a conference paper at ICLR 2020
A.2 FOR 1-DIMENSIONAL OUTPUTS MDFA COMPUTES THE SIGN OF THE GRADIENT
Lemma 2 (The gradient of a monotone layer is non-negative). Let m. R + R with m(x)
==
hr. h h22 bn har)) be a mono-net according Definition 7 Then
dhu
2 0,
(20)
dhu-i
ij
for any 1,i,j with 2 S & L and 1 <i S and S S 14-1
Proof. We have to distinguish two cases:
Case 1: == i.c. there is activation function.
We have
dhu
Wi 20,
(21)
dh
according to the definition in Equation TT)
Case 2: 1 < L, i.e. there is an activation function. We have
dhu
dhi-1 ig
=
is
(22)
Σ
Wi))
(diag()(WWhh bi)) k.j
(23)
where f' the derivative the activation function. Because f a monotonic function, its derivative is non-negative everywhere. As result we have a sum of a product of non-negative values. Ergo
dly
20
(24)
dhu
Lemma 3 (The gradient of a composition of monotone layers is non-negative). Let m R
IR
with m(x) == h2 hL-1 o .... h2 hı o ho(x) be mono-net according to Definition 7 Then
dhu
0,
(25)
drg
i.j
for any 1, 1,k,i, i,j with 2 <1 < L and 1 S k < and S sic S and 1 S S
Proof. By applying the chain rule we get
dhi dhu
-III
II m=k+1
dlt dhm-1
(26)
i.j
According to Lemma 2 we have product of non-negative values. Because product of non-negative values non-negative itself, we have
dhy dh ij
2 0
(27)
Theorem (For 1-dimensional outputs mDFA computes the sign of the gradient). Let Cly, hy) be loss function, m(z) := hr. e h .... h2 h1 o ho(r) be mono-net according to Definition i with parameters θ = [Wi bi/l 1.... L}. We denote 80 the update value computed by mDFA ac and 40 the gradient 36 for any 0 (Wi, bi) with < L. If = 1, it follows that
(50) is (Vo)
2 0,
(28)
is
for each coordinate (i.j) of
14

---

Published as a conference paper at ICLR 2020
Proof. We distinguish two cases:
Case 1: 1= i.c. is parameter of the last layer. From the definition mDFA we have
dC dhz
80
(29)
dh d0
For the gradient by applying the chain rule get
dC dL dh
(30)
d0
dh
d8
Thus, in the last layer the mDFA update equals the gradient.
Case 2: 1 L, i. is parameter of a hidden layer. From the definition of mDFA we have
(0)) (00).s
(ahb
d.
dhi do
B
(31)
dh
is
with B E IR," AL Next, we expand the multiplication,
nL
dc
ddis d(hi);
(8
=
dh
) Bu.
(32)
kml
d(hu);
d.
) BE,
(33)
d0s.s
dhp
We assumed that the output dimension is 1, i.c. n2 = Therefore,
(50).s
(50) ij
d(hi); dC
BI.
(34)
dOis dhi
For the gradient by applying the chain rule we get
v0 dC
= dL dhg
(35)
d0
dhg d0
dC dhg dhu
(36)
dhg dh, d0
Like above, we expand the multiplication.
dhg
d(hr)s
(V
)
(37)
dhi
dhu
dois
k=1
d d(bb);
nz.
dL
dh dri
)al
jk.s
(38)
d0t.s
dhi
kml
We assumed that the output dimension is 1, 1.c.c. n2 = 1. Therefore,
(vo)
= d(hi): dC
)( dhi
dhi )..
(39)
d0.s dhi
For Equation (88 we get by applying Lemma 3
50 )80),, ij
(vo).,
d(he); dL
) )1B1.)
((hi): dL
d0iss dhi )a( dhi
)
dhi 11,
(40)
d0i. dhi
dl d(ha)) dL
2
dhis
)1
B1.3
dhu 1,
(41)
dBi.j dhi
20
20
0.
(42)
n
15

---

Published as a conference paper at ICLR 2020
A.3 FOR & 2 2-DIMENSIONAL OUTPUTS MDFA UPDATES MAY NOT ALIGN WITH THE GRADIENT
Theorem (For k 2 2-dimensional outputs mDFA updates may not align with the gradient). Let (y. h be loss function, m(xx) = 11 1 o ....... o h2 o h o ho(a) be mono-net according to Definition / with parameters Ô = (Wi, by = 1,... LJ. We denote 60 the update value computed
by mDFA and V as the gradient #C the possibility that
86 for any A E {Wi bi} with S 2 < L. If ns. 2 2, there exists
(58) ij
(Vo) <0
(43)
ij
for at least one coordinate (i.j) of
Proof. We construct a minimal counterexample consisting of a network with two outputs and one
hidden layer of two neurons.
Let
0
W₂
(44)
1
0
0
B2
(45)
0
1
7c
(46)
Th
0
-1
Sh1
0
(47)
00
0
1
(48)
Then we have
Dc
W dhu
v0-
(49)
dh
do
= 0
i
()
!)
(50)
= -1 1 0
!
(51)
and
dC
dhi
80
B₂
(52)
dhg
dB
0 -1
0
(1
1
0
(1 !)
(53)
(1 -1
0
(54)
(55)
which are orthogonal.
B EXPERIMENT SETUP
B.1 NETWORK ARCHITECTURES
Dataset
Fully-comnected
Convolutional
CIFAR- 100
1024.1024
(96.5.2..96.3.2).(96333 (963:.1.(96.551))112223333 (19233.),(1233333,883331)
ImageNet 1024.1024,102,,1022
Network architectures, layers separated by commas. colle-eondeeted column specifies the number of neurons of each layer. Convolutional column specifies the number of filters, kernel size, and stride for each layer.
16

---

Published as a conference paper at ICLR 2020
B.2 n-CLASS IMAGENET
Set
Classes
BP
DFA
FA
mDFA
3
100.0 + 0.0%
59.6 7.2%
50,4 = 5.9%
51.4 + 4.4%
Training
5
100.0 + 0.0%
48,9 + 5.0%
41.4 5.5%
41.0 + 7.1%
10
99.7 0.5%
33.4 + 4.5%
24.7 3.3%
38.7 = 10.1%
3
74.5 + 8.8%
62.8 + 11.9%
64.2 + 10.9%
62.0 + 9.6%
Validation
5
63.0 7.4%
43.8 = 6.2%
42.3 + 7.1%
37.3 = 5.2%
10
41.2 + 5.0%
30.8 = 4.9%
26.0 + 4.5%
33.3 = 8.0%
Table 5: Mulit-class classification accuracy of fully-connected network with tanh activation on r2- class ImageNet. Mean and standard deviation.
Set
Classes
BP
DFA
FA
mDFA
100.0 = 0.0%
100.0 = 0.0%
35.8 + 0.7%
68.1 8.2%
Training
100.0 100.0 0.0%
21.4 1.44:0.45 0.4%
60.4 6.2%
10
100.0 + 0.0%
90.4 = 10.0%
11.3 + 0.7%
42.9 - 3.9%
68.5 + 13.0%
76.5 9.9%
53.3 + 1.7%
74.8 8.8%
Validation
5
61.6 + 7.3%
68.7 = 6.3%
29.0 = 2.0%
61.9 = 5.3%
10
52.8 + 4.6%
53.6 = 5.3%
10.6 + 0.8%
44.1 4.4%
Table 6: Mulit-class classification accuracy of Convolutional Neural Network with tanh activation on n-class ImageNet. Mean and standard deviation.
B.3 DISCUSSION ON HYPERPARAMETERS
We observed that FA algorithms yield a more stable convergence with "vanilla" stochastic gra- dient descent, i.e., post-processing of the FA updates, than with Adam (Kingma & Ba 2014) or Rmsprop (Tieleman Hinton 2012). This may be non-surprising as these acceleration methods have been developed for gradient- based optimization, whereas FA updates roughly align with the gradients at best.
Furthermore, observed that FA algorithms achieve a descent validation accuracy after the first few training epochs. However, in contrast to BP, these methods may require over hundred training epochs converge fully. This Lillicrap et al. 2016).
aligns with the observations made by
We found that FA, DFA, and mDFA perform best when all forward weights are initialized to zero. Notable exceptions are networks with ReLU activation function which express poor performance with the all-zeros initialization scheme. This poor performance with all-zeros initialization partially explains the poor observed performance of FA and DFA for ReLU networks.
In contrast Nøkland (2016), observed that backward weights initialized by a normal distribu- tion perform slightly better than the scaled uniform proposed by Nøkland (2016).
We confirmed the observation made by Bartunov et al. (2018), that feedback alignment algorithms are in general relatively sensitive to hyperparameter choice.
17


Reviews:
[{"review_id": "rJxE1TVAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the question of learning in neural networks with random, fixed feedback weights, a technique known as “feedback alignment”. Feedback alignment was originally discovered by Lillicrap et al. (2016; Nature Communications, 7, 13276) when they were exploring potential means of solving the “weight transport problem” for neural networks. Essentially, the weight transport problem refers to the fact that the backpropagation-of-error algorithm requires feedback pathways for communicating errors that have synaptic weights that are symmetric to the feedforward pathway, which is biologically questionable. Feedback alignment is one approach to solving the weight transport problem, which as stated above, relies on the use of random, fixed weights for communicating the error backwards. It has been shown that in some cases, feedback alignment converges to weight updates that are reasonably well-aligned to the true gradient. Though initially considered a good potential solution for biologically realistic learning, feedback alignment both has not scaled up to difficult datasets and has no theoretical guarantees that it converges to the true gradient. This paper addresses both these issues.\n\nTo address these issues, the authors introduce two restrictions on the networks: (1) They enforce “monotone” networks, meaning that following the first layer, all synaptic weights are positive. This also holds for the feedback weights. (2) They require that the task in question be a binary classification task. The authors demonstrate analytically that with these restrictions, direct feedback alignment (where the errors are communicated directly to each hidden layer by separate feedback weights) is guaranteed to follow the sign of the gradient.  (Importantly, they also show that monotone nets are universal approximators.) Empirically, they back up their analysis by demonstrating that in fully connected networks that obey these two restrictions they can get nearly as good performance as back propagation on training sets, and even better performance on tests sets sometimes. However, they also demonstrate (empirically and analytically) that violating the second requirement (by introducing more classes) leads to divergence from the gradient and major impairments in performance relative to backpropagation.\n\nUltimately, I think this is a great paper, and I think it should be accepted at ICLR. It provides some of the first rigorous analysis of feedback alignment since the original paper came out, and unlike those original analyses, it is not restricted to linear networks (which are certainly not universal function approximators). I have looked over the proofs, and they seem to all be correct. As well, I found the paper easy to read, which was nice. However, there are a few things that could be done to clarify the contributions and situate the work within the field of biological learning algorithms better:\n\n1) Though they do not include rigorous analyses, two previous papers have demonstrated empirically that feedback alignment works extremely well as long as the feedback weights share the same sign as the feedforward weights (see: Moskovitz, Theodore H., Ashok Litwin-Kumar, and L. F. Abbott. \"Feedback alignment in deep convolutional networks.\" arXiv preprint arXiv:1812.06488 (2018) and Liao, Qianli, Joel Z. Leibo, and Tomaso Poggio. \"How important is weight symmetry in backpropagation?.\" In Thirtieth AAAI Conference on Artificial Intelligence. 2016). Due to the requirement for monotone networks, this work is also providing a guarantee that the sign of feedforward and feedback weights are the same. That does not subtract substantially from the contributions of this paper, as the provision of the analytical guarantees is important. But, it is important for the authors to consider how their work relates to this past work. For example, could their analytical approach work equally well with nothing more than a sign symmetry guarantee? This should at least be discussed.\n\n2) It should be admitted somewhere in the paper that the second requirement on the networks for binary tasks is deeply unbiological. As such, it should be recognized in discussion that this paper provides some important contributions to our understanding of feedback alignment, but does not ultimately move the question of biologically realistic learning forward all that much. Indeed, the discussion at the end about applications notably ignores biology. But, rather than just ignoring it, the biological mismatch should be openly admitted.\n\n3) The results with the test sets are a little strange, at least for the tests with larger numbers of categories. In Bartunov et al. (2016), they reported not only better training set results with backprop, but also better test set results generally, than feedback alignment. Are the authors sure that their results, in say, Table 4, are not indicative of insufficient hyperparameter optimization?\n\nSmall notes:\n\n- Lillicrap et al.’s paper was eventually published in Nature Communications (see citation above), and the reference should be changed to reflect this.\n\n- The discussion on the impact of convolutions could be beefed up a little bit. In particular, it could be discussed relative to the results of Moskovitz et al. (above) who show that convnets work fine with nothing but guaranteed sign symmetry.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlPbgIsKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------update------\n\nAfter talking to other reviews and reading the rebuttal, I am convinced that the paper contributes sufficiently to the theoretical understanding of the FA algorithm and should be accepted as a conference paper. \n\nI hope that, in the next revision, the authors could include more about the limitation of their work and potential alternatives to improve the generosity of the proposed method.\n\n-------end of the update------\n\nThe paper presented a mono-net which has only positive weights and monotonically increasing functions in between layers except for the first layer, and it can be shown that the proposed mono-net is capable of modelling any continuous functions. The modified Direct Feedback Alignment (DFA) is applied where only signs are used to update the weights. There are several issues and concerns that leads me to the following comments and concerns:\n\n(A) Why is it necessary to construct a monotonic layer which is constrained to only be able to approximate monotonic functions? Please elaborate. If the concern is about the gradient update, then there is better way of constructing such a network with limiting it being only capable of modelling monotonic functions.\n\n(B) If, given the proof in appendix, that the proposed mono-net is indeed a universal function approximator, then why do all the layers on top of the first layer have to contain only positive weights?\n\nFollowing the proof, given a neural network with only one hidden layer and hyperbolic tanh activation functions, as long as the weights in the layer that is after the tanh functions are all non-negative or non-positive, the neural network is also a universal function approximator.\n\nSince the chosen activation function monotonically increases in the input domain, the sign of the update calculated by DFA is the same as the gradient calculated by the chain rule. \n\nThe aforementioned way of constructing a neural network allows it to be able to model to change the monotonicity of the function in between layers when the two sets of weights in consecutive two layers have opposite signs. Also, the update gives the sign of the gradient calculated by backpropagation.   \n\n(C) What is the different between the proposed network along with the update rule and a network with only non-negative weights in the layers above the first layer trained with RPROP? They look exactly the same to me. \n\nIf so, another perspective of the story is that the paper emposes a non-negative constraint on the weights in a neural network, and this could be used as a baseline.\n\n(D) The proof that gives the universal approximation theorem explicitly defines the squashing function/the activation function to be bounded and this paper follows the setting, which is reflected in the experiments where the proposed network with ReLU activation functions don't work well. \n\nHowever, the expressiveness of ReLU networks has been shown to be strong and they are also universal approximators. I'd believe that with (B), it might lift the constraints brought by ReLU in the settings proposed by the paper.\n\n(E) I am not sure why for now the number of output units matters that much. The training algorithm can easily ignore other output units when samples of a specific class is presented, then after learning, the algorithm can rank the output units to make predictions. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylesNeRKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach towards extending the capabilities of feedback alignment algorithms, that in essence replace the error backpropagation weights with random matrices.  The authors propose a particular type of network where all weights are constraint to positive values except the first layers, a monotonically increasing activation function, and where a single output neuron exists (i.e., for binary classification - empirical evidence for more output neurons is presented but not theoretically supported).  This is to enforce that the backpropagation of the (scalar) error signal to affect the magnitude of the error rather than the sign, while preserving universal approximation.  The authors also provide provable learning capabilities, and several experiments that show good performance, while also pointing out limitations in case of using multiple output neurons.\n\nThe strong point of the paper and main contribution is in terms of proposing the specific network architecture to facilitate scalar error propagation, as well as the proofs and insights on the topic.  The proposed network affects only magnitude rather than sign, and the authors demonstrate that it can do better than current FA and match BP performance.  This seems inspired from earlier work [1,2] - where e.g., in [2] improvements are observed when feedback weights share the sign but not the magnitude of feedforward nets.\n\nSummarizing, I believe that this research is interesting, and can lead to improvements in FA algorithms that could potentially be more biologically plausible, and offer advantages such as full weight update parallelization (although this is more related to the fixed weights rather than the method per-se given my understanding).  However, this also seems - at the moment - to be of limited applicability.\n\n===\nFurthermore, the introduction of the network with positive weights in the 2nd layer and on is remiscent of non-negative matrix factorization algorithms.  Can the authors establish a link to these methods, where variants with backprop have also been proposed?\n\n\n\n[1] Xiao W. et al. Biologically-Plausible Learning Algorithms Can Scale to Large Datasets, 2018\n[2] Qianli Liao, Joel Z Leibo, and Tomaso Poggio.  How important is weight symmetry in backprop-agation, AAAI 2016", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a rigorous analysis of feedback alignment under two restrictions 1) that all, except the first, layers are constrained to realize monotone functions and 2) the task is binary classification. Overall, all reviewers agree that this is an interesting submission providing important results on the topic and as such all agree that it should feature at the ICLR program. Thus, I recommend acceptance. However, I ask the authors to take into account the reviewers' concerns and include a discussion about limitations (and general applicability) of this work.
Decision: Accept (Poster)