DeepAGREL: Biologically plausible deep learning via direct reinforcement
Authors: Isabella Pozzi
Sander M. Bohte
Pieter R. Roelfsema
Venue:  (2020)
Keywords: biologically plausible deep learning
reinforcement learning
feedback gating
image claassification

Abstract: While much recent work has focused on biologically plausible variants of error-backpropagation, learning in the brain seems to mostly adhere to a reinforcement learning paradigm; biologically plausible neural reinforcement learning frameworks, however, were limited to shallow networks learning from compact and abstract sensory representations. Here, we show that it is possible to generalize such approaches to deep networks with an arbitrary number of layers.  
We demonstrate the learning scheme - DeepAGREL - on classical and hard image-classification benchmarks requiring deep networks, namely MNIST, CIFAR10, and CIFAR100, cast as direct reward tasks, both for deep fully connected, convolutional and locally connected architectures. We show that for these tasks, DeepAGREL achieves an accuracy that is equal to supervised error-backpropagation, and the trial-and-error nature of such learning imposes only a very limited cost in terms of training time. Thus, our results provide new insights into how deep learning may be implemented in the brain. 

Under review as a conference paper at ICLR 2020
DEEPAGREL: BIOLOGICALLY PLAUSIBLE LEARNING VIA DIRECT REINFORCEMENT
DEEP
Anonymous authors Paper under double-blind review
ABSTRACT
While much recent work has focused on biologically plausible variants of error- backpropagation, learning in the brain seems to mostly adhere reinforcement learning paradigm; biologically plausible neural reinforcement learning frame- works however were limited to shallow networks learning from compact and ab- stract sensory representations. Here, we show that it is possible to generalize such approaches to deep networks with an arbitrary number of layers. We demonstrate the learning scheme DeepAGREL
classical and hard image-classifen clasecclassfiication benchmarks requiring deep networks, namely MNIST, CIFARIO 10, and CIFARI 100 cast as direct reward tasks, both for deep fully connected, convolutional and locally connected architectures. We show that for these tasks, DeepAGREL, achieves an and the trial-and-error
accuracy that is equal to supervised grror-backpropagatii nature of such learning imposes only a very limited cost in terms of training time. Thus, our results provide new insights into how deep learning may be implemented in the brain.
INTRODUCTION
Among the learning rules for neural networks, reinforcement learning (RL) has the important virtue of occurring in animals and humans. Hence, RL by artificial neural networks can be used as a model for learning in the brain (Bishop et al. 1995). Indeed, previous theories have suggested how powerful RL rules inspired by artificial neural networks could be implemented in the brain (Roelfsema & Holtmaat 2018) and the methodology for shaping neural networks with rewards and punishments is an active area of research (Schmidhuber et al. 2011 Friedrich et al. 2010 Vasilaki al., 2009
O'Reilly & Frank 2006 Huang et al. (013)).
In current deep learning, deep artificial neural networks are typically trained using supervised learning, with variants of the eerrobbackpropggaaton (EBP) rule. EBP is method that adjusts synaptic weights in multilayer networks to reduce the errors in the mapping of inputs into the lower layer to outputs in the top layer. It does by first computing the output error, which is the difference between the actual and desired activity levels of output umits, and then determines how the strength of connections between successively lower layers should change decrease this error using gradient descent (Rumelhart et al. 1986).
Can the brain, with its many layers between input and output indeed solve this credit-assinnment problem in manner that is as powerful as deep learning? Similarly to deep neural networks, the brain of humans and animals are composed of many layers between the sensory neurons that register the stimuli and the motor neurons that control the muscles. Hence is tempting to speculate that the methods for deep learning that work so well for artificial neural networks also play a role in the brain (Marblestone et al.) 2016;; Scholte et al. 2017). A number of important challenges need to be solved, however, and some of them were elegantly expressed by Francis Crick who argued that the error- error-backpree caroo-backpropggatton rule is neurobiologically unrealistic (Crick) 1989). The main question is: how can the synapses compute the error derivative based on information available locally? In more recent years, researchers have started to address this challenge by proposing ways in which learning rules that are equivalent to error- backpropagation might be implemented in the brain (Urbanczik Senn SchieeSch et al. 2016 Roelfsema & Ooyen 2005 Rombouts et al., 2015 Brosch et al.; 2015 Richards & Lillicrap 2019 SSellier & Bengio| 2019 Amit 2018 [Sacramento et al. 2018), most of
which were reviewed in (Marblestone et al. 2016).

---

Under review as a conference paper at ICLR 2020
An important question is whether supervised learning, where a potentially high ditheniiensii computed outcome is compared to a target outcome of equal dimensionality, is a biologically plausible paradigm for animal learning (Marblestone et al., 2016): while examples of imitation learning exist, most animal learning would be classified as either unsupervised, self-supervised or of a reinforcement learning nature. Plausible deep learning approaches based on reinforcement learning however are lacking.
Here we provide the first RL approach deep biologically plausible learning that compares favorably performance-wise to supervised learning results found in the (recent) literature (Amit, 2018 Scellier & Bengio 2019). Using a direct-reward paradigm, we show how this framework can be linked precisely to error backpropagation, with g-values updated after every single action. We then show that this approach can be successfully applied to deep networks for image classification tasks.
As starting point, we focus on a particular type of biologically plausible learning rule for shallow networks known as AGREL (attention -atiioonngggdddoo reinforcement learning, reviewed in Richards et al.) (2019) and AuGMEnT (attention-gater memory tagging) (Roelfsema & Ooyen 2005 Rombouts ct al. 2015). These learning rules exploit the fact that in RL setting the synaptic error derivative can be split into two factors: reward prediction error which is positive if an action selected by the network is associated with more reward than expected, or, if the prospects of receiving reward increase while it is negative, if the outcome of the selected action is disappointing. In the brain, the RPE is signaled by neuromodulatory systems that project diffusely to many synapses so that they can inform them about the RPE (Schultz) 2 ; the second factor is attentional feedback signal that is known to propagate from the motor cortex to earlier processing levels in the brain (Roelfsema & Holtmaat 2018 Pooresmaeili al. 2014). When a network chooses an action, this feedback signal is most pronounced for those neurons and synapses that can be held responsible for the selection of this action and hence for the resulting RPE. These two factors jointly determine synaptic plasticity. As both factors are available at the synapses undergoing plasticity, it has been argued that learning schemes such as AGREL and AuGMEn'T are indeed implemented in the brain (Roelfsema & Holtmaat, 2018). However, the AGREL and AuGMEnTT frameworks have only been applied to networks with a single hidden layer, and modeled tasks with only a handful input neurons.
The present work has two contributions. The first is the development of biologically realistic learning rules for deep networks composed of multiple hidden layers in a RL setting: DeepAGREL. The second is the comparison of the efficacy of rrial-andeerrd learning to supervised learning with EBP, in challenging problems. We investigated how DeepAGREL copes with different datasets, specifically MNIST, CIFARIO and CIFAR 100, trained as direct reward RL tasks. This learning rule is mathematically equivalent to a version of error backpropagation in a reinforcement learning setting that trains one output unit at time selected by an e-greedy mechanism. We show that it achieves an accuracy that is essentially equivalent to EBP, at a very limited cost in terms of training time (1.5 2 times slower), which caused by the trial-and-error nature of RL.
2 BIOLOGICALLY PLAUSIBLE DEEP LEARNING THROUGH REINFORCEMENT
We here generalize and extend AGREL (Roelfsema & Ooyen 2005 Rombouts et al. 2012bb to networks with multiple layers. To achieve this, we assume that network nodes correspond to cortical columns which feedforward and feedback input (Roelfsema & Holtmaat 2018 Scellier & Bengio 2019). In the present implementation we use feedforward neuron and a feedback neuron per node, shown as blue and green circles in Fig. []]
Neurobiological findings suggest that feedforward activation of neurons gates feedback in the brain so that the activity of neurons that are not well driven by feedforward input are also hardly influenced by feedback connections (Van Kerkoerle et al. 2017] Roelfsema 2006 Treue & Trujillo 1999): we integrate such gating mechanism in our learning rule (see below), and use rectified linear (ReLU) functions as activation function of the neurons in the network. This simplifies the learning rule, because the derivative of the ReLU is equal to zero for negative activation values, and has constant positive value for positive activation values. Note however that the presented learning rule easily generalizes to other activation functions.
The network aims to select as "output action" the correct class for each image. Each computed output value is treated as a direct reward prediction (Q-value) and one class is selected. For this selected class (and only for this class 8) reward is assigned and reward prediction error 5 is
2

---

Under review as a conference paper at ICLR 2020
a) Forward pass
b) Action selection
Feedback neuron
Feedforward
c) Backward pass
d) RPE (8)
e) Update of the synapses
8
updated weights fixed weights
attention-gated signal
Figure 1: Schematic depiction of DeepAGREL. At each node, feedforward neuron (blue) and feedback neuron (green) are present; separate feedforward and feedback weights connect the nodes in the network.
computed as the difference between the predicted reward and the actual reward, 8 Q2. In the brain, this reward prediction error would correspond to the global release of dopamine (Schultz) 2002).
There are five phases upon presentation of an input image: the signal propagated through the network by feedforward connections to obtain activations for the output units where the Q-values are computed (Forward pass, Fig, [a), in the output layer one output unit wins in a stochastic, competitive action selection process (Action selection, Fig. [b)) the selected output unit causes (attention-like) feedback to feedback unit of each node (Backward pass, Fig. Te, note that this e.g.
feedback network propagates information about the selected action, just as in the brain, (Roelfsema & Holtmaat] 2018), rather than explicitly propagating error signals). A reward prediction error ó is globally computed (Fig. [d) after the outcome of the action is evident, and the strengths of the synapses (both feedforward and feedback) are updated (Fig, Te).
The proposed learning rule, DeepAGREL, has four factors:
Aw;,5 pre; posty fb..
where Aws, is the change in the strength of the synapse between units and j. pre, is the activity of the presynaptic unit, posty the activity of the postsynaptic unit and fb, the amount of feedback from the selected action arriving at feedback unit ý through the feedback network. This local learning rule governs the plasticity of both feedforward and feedback connections between the nodes.
We first consider learning by a network with two fully connected hidden layers comprised of ReLU units (as in Fig. D. and then derive how the proposed learning scheme can train networks with an arbitrary number of layers in a manner that provides synaptic changes that are equivalent to particular form of backpropapppo.ioon..
In the network with two hidden layers, there are N input units with activities I; The activation of the J neurons in the first hidden layer, y/ (1) is given by:
5 ReLU (a.) with
ui
34
where úi, is the synaptic weight between the i-th input neuron and the j-th neuron in the first hidden layer, and the ReLU function can be expressed as: ReI.U(x:) max(0.,))
3

---

Under review as a conference paper at ICLR 2020
Similarly, the activations of the K neurons in the second hidden layer, yk are obtained as follows:
(2)
(2) Vk
ReLU
(2) 0i
with
(2) aj
(1) jy 9/
with Uj,s as synaptic weight between the j-th neuron in the first hidden layer and the k-th neuron in the second hidden layer. The I. neurons in the output layer are fully connected (by the synaptic weights wx,) to the second hidden layer and will compute a linearly weighted sum of their inputs:
K
(4)
k=1
which we treat Q-values defined in Reinforcement Learning (Sutton al., 1998). from which actions (or classifications) are selected by an action selection mechanism.
For the action-selection process, implemented max-Boltzmana controller (Wiering & Schmid huber] [197): the network will select the output unit with the highest Q-value as the winning unit with probability e, and otherwise it will probabilistically select output unit using a Boltzmann distribution over the output activations:
exp q
P(21 Σ, exp q
(5)
After the competitive action selection process, the activity of the winning unit is set to one and the activity of the other units to zero, i.e. Zt=s and Rtys 0. The network then receives scalar reward r and a globally available RPE 8 is computed as 8 = r qs. where qs is the activity of the winning unit (see Fig. 1. i.c. the current estimate of reward expectancy. which seems to be coded for in the brain (Scuulzz 2002). leading to a global error E = _ In a classification task, we set the when the selected output unit corresponds the correct class, and we set the
direct reward reward to 0 otherwise.
Next, only the winning output unit starts propagating the feedback signal the other output units are silent. This feedback passes through the feedback connections with their own weights w' to the feedback neurons in the next layer, where the feedback signal is gated by the local derivative of the activation function, and then further passed to the next layer of feedback neurons through weights v', and so on. Hence only neurons that receive bottom-up input participate in propagating the feedback signal. We will demonstrate that this feedback scheme locally updates the synapses of the network in a manner equivalent a particular form of error- hackpropagation.
Given the learning rate a, the update of the feedforward weights Wk., between the last hidden layer and the selected unit s in the output layer (but the same rule holds for the corresponding feedback weights, indicated as w/ ( is given by:
Awk.a = a 8. 8.y% Za = AW and Awi.i = 0 = Awiy
(6)
The feedforward and feedback weights v and o' between the first and second hidden layer change as follows:
AUya = a
8. 1/0 -())) w.y 2. = 8 u/ 9(2), ffb Arks
1 if y > 0,
with 9(2)a
(8)
0 otherwise
Egoo
glo wh,a 21 w',k Res
(9)
4

---

Under review as a conference paper at ICLR 2020
a)
b)
c)
enabled plasticity
hindered plasticity
FR
FB
FB
Aw Aw
Aw'=
FF
FF
FF
FF
FB
FB
FF
FB
FB
FF
FF
FF
FF
Figure 2: DeepAGREL algorithm plasticity gating. a) Example hidden layers of network; b) when the activity of the feedforward neuron is above the threshold, the feedback signal propagated to lower neurons and plasticity enabled; c) when the input to the feedforward unit stays below the threshold for activation the feedback signal is not propagated the lower layer and plasticity is disabled.
where fb yb is the feedback coming from the selected output (O),, and 90) denotes the feedback gating. The weights between the inputs and the first hidden layer then adapted as:
Aug = a-- 5 5.
901)
Σ this 9(2) wa, 2
(10)
= 0 5. D2 9(1), Σ this 9(2) Sb Sby = a a 8. vs 9(1),
if >0,
with 901), otherwise,
(11)
5b Mo = Σ 9(2) ukj
fb p0
(12)
which is the feedback coming from the second hidden layer. fb 20 and fb fb,g") represent the activity of feedback neurons yk and y/ which are activated by the propagation of signals through the feedback network, once action has been selected.
For deeper networks, updates of feedforward synapses Awp.m from p-th neuron in the n-th hidden layer onto m-th feedforward neuron in the (n + 1)-th hidden layer are thus computed as:
Awp.m = a 8 of [(n+1) fb
(13)
and are equal the update of the corresponding feedback synapses Aw where the activity the feedback unit is determined by the feedback signals coming from the (n + 2)-th hidden layer:
fb (n+1) Σ 9(
Vy.m -fb (n+2)
(14)
with indexing the units the (n + 2)-th hidden layer.
The update of synapse is thus expressed as the product of four factors: the RPE 8, the activity of the presynaptic unit, the activity of postsynaptic feedforward unit and the activity of feedback unit the same postsynaptic node, as in Eq. [ of which eqs. (6). (7) and (1O) all variants. Notably, all the information necessary for the synaptic update is available locally, at the synapse. Moreover, simple inspection shows tat the identical update for both feedforward and corresponding feedback synapses (i.c., Awk. and Aw/ AUj, and Auk, and Aus,j and Au',)) can be computed locally.
The role the feedback units each node is to gate the plasticity of feedforward connections (as term, which determines the plasticity
well their of synapses onto the feedforward neuron. In the opposite direction, the feedforward units gate the activity of the feedback units. In Fig. examples of these interactions illustrated. Feedback gating
plasticity): fby acts plasticity-
5

---

Under review as a conference paper at ICLR 2020
9j is shaped by the local derivative of the activation function, which, for a unit with ReLU activation function, corresponds to an all-or-nothing gating signal: for ReLU feedforward units, the associated feedback units of node are only active if the feedforward units are activated above their threshold (Fig. 2b) otherwise the feedback units remain silent and they do not propagate the feedback signal to lower processing levels (ee..
There is neuroscientific evidence for the gating of plasticity of feedforward connections by the activity of feedback connections (Roelfsema & Holtmaat, 2018). Gating of the activity of feedback units by the activity of feedforward units is also in accordance with neurobiological findings: attentional feedback effects on the firing rate of sensory neurons are pronounced if the neurons are well driven by stimulus and much weaker if they are not (Van Kerkoerle et al., 2017 Roelfsemal [2066] Treue & Trujillo, 1999p. The distinction between feedforward and feedback nodes and the implemented gating mechanisms makes the update rule local at any depth within the network.
DeepAGREL is equivalent a special form of error- crror-bacprropggation (see Appendix A for the complete derivation):The standard error- pror-bacckrropggation equations for a weight between units p and m in layer n and n + respectively are:
A pmp..m = -ay, ym
(n+1) c(++1) with
en
1x+1) cm
OF (n+1)
Σ 9
Wm..g9
(2+2) dn+2), oq
(15)
dym
with q indexing the units of the (n + 2)-th hidden layer and yos indicating the derivative of v2 This corresponds to the DeepAGREL equations for the adjustment to the winning output when we set
O'
v = Mj and (0) aE -5, and i 0:
Jq.
Aup. = a 5 v/.²) 9m fb (m+1)
(16)
ym.
Compared to error-backproopggation, in the RL formulation of DeepAGREL only the error e for the winning action / is non-zero, and the weights in the network are adjusted to reduce the error for this action only. Depending on the action-selection mechanism, this trial-and-error approach will adjust the network towards selecting the correct action, while the Q-values for incorrect actions will only decrease in strength occasionally, when they are chosen. This contrasts standard error- backpropagation, which will continuously drive the values of the incorrect actions to lower action values. Hence, RI Q-values by DeepAGREL. is expected to be slower than learning with a fully supervised method such as pror-backprropggatin. We will test these predictions in our simulations.
EXPERIMENTS
We tested the performance of DeepAGREL on the MNIST, CIFARIO and CIFAR100 datasets, which classification tasks, and therefore simpler than more general RL settings that necessitate the learning of a number of intermediate actions before a reward can be obtained. These types of tasks This previous work
have been addressed elsewhere (Rombouts et al. 2015) with AGREL/AuGMedt used compact and abstracted sensory representations, and the present work therefore addresses much more complex input patterns.
The MNIST dataset consists of 60,000 training samples (i.e. images of 28 by 28 pixels), while the CIFAR datasets comprise 50,000 training samples (images of 32 by 32 pixels, and 3 values for the color at each pixel), of which 1,000 were randomly chosen for validation and left out of the training set. We use a batch gradient to speed up the learning process (but the learning scheme also works with learning after each trial, i.e. not in batches): 100 samples were given an input, the gradients were calculated, divided by the batch size, and then the weights were updated, for each batch until the whole training dataset was processed (i.e. for 590 or 490 batches in total), indicating the end of an epoch. At the end of each epoch, a validation accuracy was calculated on the validation dataset. An early stopping criterion was implemented: if for 20 consecutive epochs the validation accuracy had not increased, learning was stopped.
We ran the same experiments with DeepAGREL and with error- backpropppoooaaioonn for neural networks with with three and four hidden layers, as in (Amit) 2018). The first layer could be either convolutional or locally connected (ie, with untied weights), the second layer convolutional but with stride of
6

---

Under review as a conference paper at ICLR 2020
2 in both dimensions, to which a dropout of 0.8 (i.e. 80% of the neurons in the layer were silent) was applied, then two fully connected layers followed (with the last layer having a dropout rate of 0.3). At the level of the output layer (which had 10 neurons for MNIST and CIFARIO, 10, and 100 for CIFAR100) for proorbbackroopggation a softmax was applied and a cross-entropy error function was calculated. We decided to test networks with locally connected layers hecause such architecture could represent the biologically plausible implementation of convolutional layers in the brain (since shared weights are not particularly plausible, e.g. Cox al. (2005). However, since such layers introduce many more parameters in the network, hence we add them here to demonstrate the performance penalty of increasing complexity. Moreover, instead of using max pooling layers reduce the dimensionality of the layer following the convolutions, we substituted such layers with convolutional layers with equal number of filters and kernel size, but with strides (2.2), as described in (Springenberg et al. 2014). As argued by Hinton (Hinton et al.) (016), dropout is biologically plausible as well: by removing random hidden units in each training run, it simulates the regularisation process carried out in the brain by noisy neurons.
In summary, we ran experiments with the following architectures for the Conv respectively LocCon experiments in Table
a) conv32 3x3; conv32 3x3 str (2, ; drop.8; full 1, 0000 fu11500; 3,
b) 1occon32 3x3; conv32 3x3 str (2,2); drop.8; full 1, 0,000 ful1500; drop. with 10 different seeds for synaptic weight initialization. All weights were randomly initialized within the range [-0.02, 0.02] and the feedback synapses were identical to the feedforward synapses (strict reciprocity). For MNIST only we also performed a few experiments with fully connected networks, of which the weights were initialized in [-0.05, 0.05). For CIFARIO and CIFARIOR we also separately ran the same experiments with 10x standard data augmentation (zooming, shifting, Fermändez-Garche & König 2018).
vertical reflections)
4 RESULTS
Table presents the results of simulations with the different learning rules. We trained networks with only three hidden layers and networks four hidden layers; these networks had an extra hidden layer with 1000 units. We used 10 seeds for each network architecture and report the results as mean (slandard deviation). Our first result is that DeepAGREL. reaches relatively high classification accuracy of 99% on the MNIST task, obtaining essentially the same performance as standard error- backpropagation both with the architectures with convolutions and straightforward fully connected networks. The speed of convergence using DeepAGREL was a factor of 1.5 to 2 slower compared to using eroor-backroopggation for networks with convolutional layers, while it was factor of 2.5 slower in networks for locally connected layers, but performing slightly better than error- croorbackpropagationn
The results obtained from networks trained on the CIFAR10 dataset show that networks trained with DeRL reached the same accuracy (if not higher) than with groo-bbackroopggation The number of epochs required for the networks meet the convergence criterion was also comparable.
Table T also shows the results obtained from networks trained on CIFAR 100. Without data aug- mentation, the final accuracy obtained with DeepAGREL was somewhat lower than with error- backpropagation. However, DeepAGREL learns the CIFAR 100 classification task with a convergence rate only 2 to 2.5 times slower than proorbackkropagationn and the rate for CIFARIO. Adding in standard data augmentation, both algorithms perform much better, and DeepAGREL closes the gap with error backpropagation in terms of accuracy (also for CIFAR10). These results shows that such trial-and-error learning rule can scale up to a 10 times higher number of classes with relatively small penalty.
reinforcement To illustrate the learning process of networks trained with the reinfrrmme learning
approach, we show how the reward probability increases (Fig, 3 during the training, compared to how the error (plotted as 1 error) evolves throughout the epochs for 10 networks trained with error- backppoccproogttiionn (in both cases, mean 2o from 10 example networks is plotted as a function of the epochs), both for CIFAR1O 10 (left panel) and CIFARIO0 (right panel). Indeed, learning with supervision is faster than learning by rial-and-erroo a difference that is particularly pronounced for CIFAR 100 where the probability of choosing the right action starts only 1 16.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Results (averaged over 10 different seeds, the mean and standard deviation are indicated: in some cases indicated with "*" only out of 10 seeds converged). For CIFARIO and CIFAR 100, we also applied data augmentation where indicated.
Rule
1st layer
Hidden units
a
Epochs [ft]
Accuracy [%]
DeepAGREL
Full
1500-1000-500
5e-01
130 (54)
98.33 (0.09)
Error-BP
Full
1500-1000-500
1e-01
92 (11)
98.32 (0.04)
MNIST
DeepAGREL
Conv
21632-5408-1000-255
le+00
59 (21)
99.16 (0.16)
Error-BI
Conv
21632-5408-1000-22
1e-02
33 (12)
99.21 (0.17)
DeePAGREL
LocCon
21632-55088-100-5500
le+00
66 (15)
98.30 (0.22)
Error-BP
LocCon
21632-5408-1000-500
1e-02
24 (13)
98.73 (0.41)
DeepAGREL
Conv
28800-7200-1000-500
le+00
(23)
73.54 (1.35)
Error-BP
Conv
28800-7200-1000-500
1e-03
83 (21)
71.25 (1.08)
CIFARIO
DeepAGREL
LocCon
28800-7200-1000-33
le+00
173 (36)
64.37 (2.41)
Error-BP
LocCon
28800-7200-1000-500
1e-03
145 (16)
64.65 (1.16)
DATA ALGM
DeepAGREL
Conv
28800-7200-1000-
le+00
64(18)
79.97(1.09)
Error-Bi BP
Conv
11900-7200-10005500
1e-03
47(23)
78.39(0.67)
DeepAGREL
LocCon
28800-7200-1000-500
le+00
78(25)
70.87(0.70)
Error-BP
LocCon
28800-7200-1000-550
1e-03
56(8)
72.07(0.46)
DeepAGREL
Conv
28800-7200-1000-2
le+00
230 (30)
34.90 (1.49)*
Error-BP
Conv
18800-7200-1000-5000
1e-03
(24)
36.79 (1.78)
DATA CIFARIO0
DeepAGREL.
LocCon
28800-7200-1000-500
1e+00
343 (68)
29.39 (2.38)
Error-BP
LocCon
28800-7200-1000-500
1c-03
156 (13)
32.73 (0.78)
DeepAGREL
Conv
11800-7200-1000-500
le+00
93(18)
42.98(1.78)
Error-BP
Conv
28800-7200-1000-11
1e-03
52(16)
41.30(0.61)
DeepAGREL
LocCon
18800-7200-1000-550
le+00
123(25)
37.32(0.76)
Error-BP
LocCon
28800-7200-1000-500
1e-03
46(12)
38.66(0.49)
-1.0
0.35
0.00
-1.5
Reward
-0025
Reward
-0.50
0.20
0.15
2.5
0.2
CIFAR10
CIFAR100
-1.00
Q-AGREL
Q-AGEL
0.0
Error backeropogation
-1.25
0.00
Error bockprssopption
-3.5
60
100
100
150
Epoch
Epoch
Figure 3: Example of learning process with the two learning schemes for CIFAR10 and CIFARI00, without data augmentation.
DISCUSSION
We implemented deep, biologically plausible RL scheme called DeepAGREL and found that it was able to train networks perform the MNIST, CIFAR1O and CIFAR 100 tasks as direct reward problems with performance that was nearly identical error-backpee sroorbacckroopggation in particular when we included simple data augmentation. We also found that the fral-and-error nature of learning to classify with RL incurred very limited cost of 1-2.5x more training epochs to achieve the stopping criterion, even for classifying objects in 100 classes.
The results obtained with relatively simple network architectures (i.e. not very deep) and learning rules (no optimizers were used except for data augmentation where indicated). Such additions would almost certainly further increase the final accuracy of the DeepAGREL learning scheme.
8

---

Under review as a conference paper at ICLR 2020
The present results demonstrate how deep learning can be implemented in biologically plausible fashion in deeper networks and for tasks of higher complexity by using the combination of a global RPE and "attentional" feedback from the response selection stage influence synaptic plasticity. Importantly, both factors are available locally, at many, if not all, relevant synapses in the brain (Roelfsema & Holtmaat 2018). We used symmetrical weights and did not address a potential weight- transport problem (Bartunov et al. 2018) since, as argued in AArout et al. (2019), given reciprocal learning and weight decay specificity for feedforward and feedback weights emerges.
We demonstrated that DeepAGREL is equivalent to a version of error-backrropagation that only updates the value of the selected action. DeepAGREL was developed for feedforward networks and for classification tasks where feedback about the response is given immediately after the action selected. However, the learning scheme is a straightforward generalization of the AuGMeNT framework (Rombouts et al., 2012a 2015), which also deals with RL problems for which a number of actions have to be taken before a reward is obtained.
future work, we aim to to develop more architectures compatible with DeepAGREL and train more complex networks required for larger Similarly, as stressed by Richards et al. (2019), theory is needed to quantify the variance of the AGREL approach.
tasks like ImageNet-classificate
We find it encouraging that insights into the rules that govern plasticity in the brain are compatible with some of the more powerful methods for deep learning in artificial neural networks. These results hold promise for a genuine understanding of learning in the brain, with its many processing stages between sensory neurons and the motor neurons that ultimately control behavior.
REFERENCES
Mohamed MA Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas Tweed. Deep learning without weight transport. 2019.
Yali Amit. Biologically plausible deep learning. arXiv preprint arXiv:/812.07965, 2018.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lilli- crap. Assessing the scalability of diologically-motivee deep learning algorithms and architectures. In Advances in Neural Information Processing Systems, pp. 9368 9378, 2018.
Christopher M Bishop et al. Neural networks for pattern recognition. Oxford university press, 1995.
Tobias Brosch, Hciko Neumann, and Picter R Roelfsema. Reinforcement learning of linking and tracing contours in recurrent neural networks. PLoS computational biology, 2015.
David D Cox. Philip Meier, Nadja Ocrtelt, and James J DiCarlo. Orrcaking'position-ioi iosition-invariant invariant object recognition. Nature neuroscience, 8(9): 114,, 2005.
Francis Crick. The recent excitement about neural networks. Nature, (3766))) 1989.
Johannes Friedrich, Robert Urbanczik, and Walter Senn. Learning spike- based population codes by reward and population feedback. Neural computation, 22(7):1698-1711 2010.
Alex Hernández- and Peter König. Further advantages of data augmentation on convolutional neural networks. In Artificial Neural Networks and Machine Learning ICANN 2018, pp. 95-103, Springer International Publishing, 2018
Geoffrey Hinton et al. Can the brain do back-propagation. In Invited talk at Stanford University Colloquium on Computer Systems, 2016.
https: / www. yout ube com www.whh.wwwvaaaa... Wwtchhh-VVVCCCC
Tsung-Ren Huang, Thomas E Hazy, Seth A Herd, and Randall C O'RReilyy Assembling old tricks for new tasks: A neural model of instructional learning and control. Journal of Cognitive Neuroscience, 25(6):843-851, 2013.
Adam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience, 10:94, 2016.

---

Under review as a conference paper at ICLR 2020
Randall C O' Reillly and Michael J Frank. Making working memory work: computational model of learning in the prefrontal cortex and basal ganglia. Neural computation, 18(2):283-322 -328, 2006.
Arezoo Pooresmaeili, Jasper Poort, and Pieter R Roelfsema. Simultaneous selection by object-based attention in visual and frontal cortex. Proceedings of the National Academy of Sciences, pp.
201316181, 2014.
Blake A Richards and Timothy P Lillicraap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology, 54:28-36, 2019.
Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1711-770) 2019.
Pieter R Roelfsema. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci,, 29:203-227,
2006.
Pieter R Roelfsema and Anthony Holtmaat, Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 19(3):166, 2018.
Pieter Roelfsema and Arjen van Ooyen. Attention-gated reinforcement learning of internal
representations for classification. Neural computation, 17(10)2117-2214, 2005.
Jaldert Rombouts, Pieter Roelfsema, and Sander M Bohte. Neurally plausible reinforcement learning of working memory tasks. In Advances Neural Information Processing Systems, pp. 1871-1879, 2012a.
Jaldert O Rombouts, Arjen Van Ooyen, Pieter R Roelfsema, and Sander M Bohte. Biologically plausible multi--iimenional reinforcement learning in neural networks. In International Conference on Artificial Neural Networks, pp. 443-450 Springer. 2012b.
Jaldert O Rombouts, Sander Bohte, and Pieter Roelfsema. How attention can create synaptic tags for the learning of working memories in sequential tasks. PLoS computational biology, 11(3): e1004060, 2015.
David E Rumelhart, Geoffrey Hinton, James L McClelland, et al. A general framework for parallel distributed processing. Parallel distributed processing: Explorations in the microstructure of cognition, 1455-76)26, 1986.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pp.
2018.
Benjamin Scellier and Yoshua Bengio. Equivalence of equilibrium propagation and recurrent backpropagation. Neural computation, (atton,,n1122))) 31(2):312-329, 2019.
Mathieu Schiess, Robert Urbanczik, and Walter Senn. Somato-dendritic synaptic plasticity and emoo-backkroppggaioo in active dendrites. PLoS computational biology, 12(2)ee1004638, 2016.
Jurgen Schmidhuber, Dan Cireşan, Ueli Meier, Jonathan Masci, and Alex Graves. On fast deep nets for agi vision. In International Conference on Artificial General Intelligence, pp. 243-2446644 246. Springer, 2011.
Steven Scholte, Max M Losch, Kandan Ramakrishnan, Edward HF de Haan, and Sander M Bohte. Visual pathways from the perspective of cost functions and multi-task deep neural networks. Cortex, 2017.
Wolfram Schultz. Getting formal with dopamine and reward. Neuron, 36(2):241-263, 2002.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 1rXiv:1412.6806, 2014.
Richard Sutton, Andrew G Barto, Francis Bach, et al. Reinforcement learning: An introduction. MIT press, 1998.
10

---

Under review as a conference paper at ICLR 2020
Stefan Treue and Julio C Martinez Trujillo. Feature-basee attention influences motion processing gain in macaque visual cortex. Nature, 399(6736):575, 1999.
Robert Urbanczik and Walter Senn. Learning by the dendritic prediction of somatic spiking, Neuron, 813)):22--52, 2014.
Van Kerkoerle, MW Self, and PR Roelfsema. Effects of attention and working memory in the different layers of monkey primary visual cortex. Nat. Commun, 8:13804, 2017.
Eleni Vasilaki, Nicolas Frémaux, Robert Urbanczik. Walter Senn, and Wulfram Gerstner. Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS computational biology, 5 1(12))e1000586, 2009.
Marco Wiering and Jürgen Schmidhuber. Hq-learning. Adaptive Behavior, 6(2):219-246, 1997.

---

Under review as a conference paper at ICLR 2020
A APPENDIX DERIVATION OF ERROR BACKPROPAGATION
For error- proo-backproopggation in the same networks with error E computed as the summed square error over all output Q-values qi and target outputs ğt, E Ei â1)². if define Bq
ge
(gt - (t) == ef (0) where the superscript (O) stands for output layer, de-sstadarrdequuatoon for the synaptic updates are:
Awk, = - -oyk (2) 4
(17)
AUyAA -ay (2)
Σ Wiike (0) = -ay; ( (2) (2)
Uk
" c
(18)
Aui -cxay
(1)
Σ Uk (2)
Σ Wi (0)
aa
(2)' (1)'
ck
-xxyy
(19)
and in general, for weight between units p and in layer n and respectively are:
OE
Awp.m = -ory Vm "
(n) (x+1)' (n+1).
with
(x+1)
dym
(n+1)
Wm
(n+2)' (n+2)
Cq
(20)
with q indexing the units of the (n + 2)-th hidden layer and yoo indicating the derivative of 2
(-)
This corresponds to the DeepAGREL equations for the adjustment to the winning output when we set
y 2j and ei-
BE 8, and 1 0:
Dz.
Awk,I aduuk (2)
(21)
Anj.k ady, (1) fb (3)
(22)
un
Aui,j = adxigg Σ
gk fb = adxig,fb,.o
(23)
and, by recursion,
Awp.m =adyf"") fm (0-1
(24)
12


Reviews:
[{"review_id": "SkxelqZCFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents DeepAGREL, a framework for biologically plausible deep learning that is modified to use reinforcement learning as a training mechanism. This framework is shown to perform similarly to error-backpropagation on a set of architectures. The idea relies on feedback mechanism that can resemble local connections between real neurons.\n\nThis paper is an interesting approach to provide a reinforcement learning paradigm for training deep networks, it is well written and the experiments are convincing, although more explanation about why these specific architectures were tested would be more convincing. I also think the assumptions about feedback connections in real neurons should be visited and more neuroscientific evidence from the literature should be included in the paper. Do we expect feedback to happen at each level of a neuron-neuron interaction and between each pair of connected neurons? Is there a possibility that feedback is more general to sets of neurons, or skips entire layers of neurons? I think more neuroscience background would help this paper (and others on the topic). Nonetheless, I think the paper does offer an interesting proposal of a more biologically plausible form of deep learning.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJl1ApMw5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper generalizes the AGREL biologically plausible learning algorithm to deeper networks. It presents experiments to show that the method can scale to medium-sized tasks with only a modest slow down in training speed and essentially no change in classification performance.\n\nMajor comments:\n\nThis is an interesting paper which shows that a particular biologically plausible learning method can attain comparable performance to gradient descent on small and mid-size visual classification tasks. The proposed mapping to the biology is of a different flavor from many other recently proposed approaches. \n\nIt is striking that the CIFAR10 network trains about as fast as the EBP network, while the CIFAR100 network trains much more slowly. This is presumably because randomly guessing the right answer out of 100 possibilities is the bottleneck. The paper could be strengthened by studying the speed of learning as a function of the number of output classes. For this approach to scale to ImageNet, with a 1000-way classification (or to our human visual recognition abilities with far more classes), this is an important scaling dimension to consider. \n\nIt should be noted that other biologically plausible schemes like feedback alignment were able to solve CIFAR and other smaller image classification tasks, but struggled when applied to the larger scale ImageNet problem. The paper could be improved by pointing to this limitation of the present work, the possibility that performance could change on larger tasks, and the need to conduct larger experiments in future work.\n\nPersonally I think the statement at the beginning of the introduction that only RL occurs in animals and humans is overstated. Unsupervised learning occurs in some form in critical period plasticity, and in various unrewarded statistical learning paradigms, at a minimum.\n\nI would also caution against categorical statements of biological plausibility, for instance, in saying that shared weights in convolutions are biologically implausible (bottom of pg 6). The same criticism has been leveled at gradient descent learning, and as the present submission shows, these intuitive judgements can be misleading.\n\nThe distinction between RL and supervised learning is a bit blurred in the present submission because it is considering a classification setting in which exactly one out of a number of possible outputs is rewarded on each trial. This looks very similar to the supervised learning scenario, and relies on stochastic softmax-like competition to select a single output. This approach is very reasonable for mutually exclusive classification tasks, but this is a small subset of the tasks that full EBP could be applied to. \n\nThe paper is fairly clear but the explanation of the algorithm could be further streamlined and condensed. Is DeepAGREL equivalent to stochastically selecting a single unit in a softmax layer and then back propagating only its error? It seems so from what I understand, and this may be a straightforward way to explain the algorithm. \n\nTypos:\n\nThe text on page 7 describes MNIST performance as 99.17% but the table has 99.16%.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJegoPYvcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an backpropagation-free algorithm for training in deep neural networks.\nThe algorithm called DeepAGREL does not use the chain rule for computing derivatives and instead is based on direct reinforcement. \nAuthors conduct experiments on MNIST and CIFAR where they find their method to reach performance similar to BP.\n\nNovelty: the method is built upon existing ideas, however, the exact algorithm seems to be novel.\n\nClarity: although it is possible to understand the algorithm based on the provided textual description, it would not hurt to provide a more formal presentation of the main update equation (1). \n\nIt is not clear why authors provide two different expressions for updates in each layer (in terms of fb^{l} and fb^{l+1}).\n\nI would also appreciate a clear discussion on the theoretical properties of the algorithm, for example, is it guaranteed to converge to a critical point of some loss function? Can we derive the update rule from some known optimisation procedure, e.g. REINFORCE?\n\nQuality: Unfortunately I have a number of major issues with this respect.\n\n1. Authors do not clearly pose what are the properties of back propagation that they find biologically non-plausible and how exactly their algorithm is making progress on them. There seems to be some sort of consensus in the literature about these properties and if authors accept it, then it looks like DeepAGREL is not very plausible too, because it does not address the weight-transport issue at all.\n2. Many recent results are not even mentioned in the paper, for example, (Bartunov et al, 2019). From not so recent — the whole branch of research around target propagation algorithm (Lee et al, 2014).\n3. The experiments are not convincing to me, as the considered network architectures are quite shallow and thus the ability to perform credit assignment can not be demonstrated. Besides that, the locally-connected architecture has only the first layer locally-connected and it does not make much sense to me, as it does not factor out the impact of weight sharing. \n\nOverall, I think the paper is not ready for publication at ICLR.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an RL-based algorithm for training neural networks that is able to match the performance of backprop on CIFAR and MNIST datasets.

The reviewers generally found the algorithm and motivations interesting, but some had issue with the imprecision of the notion of "biologically plausible" used by the authors. One reviewer had issues with missing discussion of related work and also doubts about the meaningfulness of the experiments, since the networks were quite shallow.

For this type of paper, clarity and precision of exposition is crucial in my opinion, and so I recommend rejection at this time, but encourage the authors to take the feedback into account and resubmit to a future venue.
Decision: Reject