Biologically Plausible Neural Networks via Evolutionary Dynamics and Dopaminergic Plasticity
Authors: Sruthi Gorantla
Anand Louis
Christos H. Papadimitriou
Santosh Vempala
Naganand Yadati
Venue:  (2020)
Keywords: Biological plausibility
dopaminergic plasticity
allele frequency
neural net evolution

Abstract: Artificial neural networks (ANNs) lack in biological plausibility, chiefly because backpropagation requires a variant of plasticity (precise changes of the synaptic weights informed by neural events that occur downstream in the neural circuit) that is profoundly incompatible with the current understanding of the animal brain. Here we propose that backpropagation can happen in evolutionary time, instead of lifetime, in what we call neural net evolution (NNE). In NNE the weights of the links of the neural net are sparse linear functions of the animal's genes, where each gene has two alleles, 0 and 1. In each generation, a population is generated at random based on current allele frequencies, and it is tested in the learning task. The relative performance of the two alleles of each gene over the whole population is determined, and the allele frequencies are updated via the standard population genetics equations for the weak selection regime. We prove that, under assumptions, NNE succeeds in learning simple labeling functions with high probability, and with polynomially many generations and individuals per generation. We test the NNE concept, with only one hidden layer, on MNIST with encouraging results. Finally, we explore a further version of biologically plausible ANNs inspired by the recent discovery in animals of dopaminergic plasticity: the increase of the strength of a synapse that fired if dopamine was released soon after the firing.

Under review as a conference paper at ICLR 2020
BIOLOGICALLY PLAUSIBLE NEURAL NETWORKS VIA EVOLUTIONARY DYNAMICS AND DOPAMINERGIC PLASTICITY
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial neural networks (ANNs) lack in biological plausibility, chiefly because backpropagation requires a variant of plasticity (precise changes of the synap- tie weights informed by neural events that occur downstream in the neural circuit) that is profoundly incompatible with the current understanding of the animal brain. Here we propose that backpropagation can happen in evolutionary time, instead of lifetime, in what we call neural net evolution (NNE). In NNE the weights of the links of the neural net are sparse linear functions the animal's ' genes, where each gene has two alleles, 0 and 1. In each generation, population is generated dom based on current allele frequencies, and it is tested in the learning task. The relative performance of the determined, and the allele frequencies are updated via the standard population ge- netics equations for the weak selection regime. We prove that, under assumptions. NNE succeeds in learning simple labeling functions with high probability, and with polynomially many generations and individuals per generation. We test the NNE concept, with only one hidden layer, on MNIST with encouraging results. Finally, we explore a further version of biologically plausible ANNs inspired by the recent discovery in animals of dopaminergic plasticity: the increase of the strength synapse that fired if dopamine was released soon after the firing.
alleles of each gene over the whole population is
INTRODUCTION
In his Turing award lecture, neural networks pioneer Geoff Hinton opined that "evolution can't 't get gradients because a lot of what determines the relationship between the genotype and the phenotype outside your control" (Hinton] 2019). We beg to differ. Evolution does have what amounts to an effective oracle access to the (indeed, complex and intractable) mapping from genotype to pheno- type. The rell-established equations of population genetics governing evolution under recombina- tion (Bürger) 2000 Chastain et al., 2014) describe the way whereby the distribution of genotypes in the population is updated from one generation the next, informed by the empirical fitness of the phenotypes during lifetime; and these equations do bear similarity to gradient descent and, even closer, to no-regret learning (Chastain al., 2014)) this paper, we show that, in fact, quite effective training of neural nets can be carried out, without backpropagation, in evolutionary time.
The towering empirical success of ANNs has brought into focus their profound incongruity with what we know about the brain: backpropagation requires that and synaptic weights and plasticity be informed by downstream events. Clever versions of ANNs have been proposed recently that avoid this criticism: ANNs whose backward weights are random and fixed (Lillicrap et al., 2016) and a variant that also uses random feedback weights but with zero initial conditions (No kland, 2016), backpropagation interpretation of STDP (a widely accepted theory of plasticity) (Bengio itt al., 2015), unsupervised learning using STDP (Dichl & Cook] (0055). ANNs driven by neural competition (Krotov & Hoptield 2019), or ANNs with target value propagation at each layer rather than the loss gradient (Lee et al. 2015
Here take very different approach. We believe that, while forward neural computation is coterminous with life, backpropagation (i.c., feedback to individual neurons and synapses about their contribution to the overall performance of the eircuit) can be effectively carried out over evolutionary

---

Under review as a conference paper at ICLR 2020
time. Suppose that the brain circuitry for particular classification task, such "food/not food", is encoded in the animal's genes, assuming each gene to have two alleles O and 1. A (haploid) genotype is a bit string. Crucially, we assume that the weight of each link of the neural network is a fixed sparse linear function of the genes. Evolution proceeds in generations. At each generation, gene is an independent binary variable with fixed probability of 1. A population is sampled from this distribution of genotypes, and it experiences a sequence of inputs to the brain circuit. Fitness of each genotype depends. to some small degree, on the animal's success over its lifetime in the specific classification task. In the next generation. the allele frequencies will change slightly, depending on how each allele of each gene fared cumulatively (over both all inputs and all genotypes containing it) in the classification task. These changes follow the standard population genetics equations for the weak selection regime, see Bürger (2000); Chastain et al. (2014); weak selection means that the classification task is only one of the many biological functions (digestion, locomotion, other brain circuits, etc.) that affect the animal's fitness.
The question is, can competent brain circuits evolve this way? We offer theoretical evidence that this is indeed the case"] In Section 2 we prove that, if the classifier to be learned is linear, then evolution does indeed succeed in creating a neural network that classifies well, almost certainly and within polynomial (in the dimension of the problem) number of generations, individuals per generation, and neurons per individual. We also validate our theorem through experiments on the MNIST data set. Our experiments are not meant to compete with what is happening in the ANN world. We want to make the point that competent learning can happen in life: NNE with a single hidden layer already gives surprisingly good accuracy rates (more than 90% accuracy for classifying the MNIST digits 0 to 4).
There is a different way of looking at, and motivating, our results, namely from the point of view of the study of the brain in connection to evolution. "Nothing in biology makes sense except in the light of evolutionn" Theodosius Dobzhansky famously proclaimed. Neuroscientists have espoused this point of view, and evolutionary arguments come up often in the study of the brain, see for example (Bosman & Aboitiz (2015). However, we are not aware of a technical discussion in the literature of the obvious existential question: Is the architecture of the brain susceptible to evolution through natural selection? Can brain circuits evolve? Our mathematical and empirical results in this paper on NNE strongly suggest that, indeed, effective brain circuits specializing in classification tasks could have evolved.
We also propose a second biologically plausible ANN-like mechanism, based on dopaminergic plas- ticity. It was recently established experimentally (Yagishita al.] 2014) that weights in certain synapses (in this case from the cortex to the striatum in the mouse brain, but not only) are increased if dopamine was released within 0.5-2 seconds after the synapse's firing. Intuitively, this is rein- forcement plasticity mechanism that "rewards" synapses whose firing led to a favorable outcome. Inspired by this experiment, we define dopaminergic neural nets (DNN), in which the weight of link that fired (that is, both nodes fired during the current training example) is modified by multiple of (1 er where err is the error of the current training example. That is, links that fired are rewarded if the result was good, and actually also punished if it was not. Our experiments show that such DNNs can also learn to classify quite well, comparable to SGD.
we give rigorous proof that NNE with a single hidden layer succeeds in learning arbitrary linear target functions. Section 3 we discuss experiments with NNE and DNN on MNIST.
Our Contributions.
In Section 2
2 ANALYSIS OF NNE
A genotype can be viewed vector x (0,1)* A probability distribution over the genotypes is given by a vector p E [0, 1]" a genotype x is sampled by setting x(i) = 1 with probability p(i). independently for each The neural network corresponding to a genotype æ is a feed-forward neural network (FFNN) whose weights are computed as follows. For prediction network having m links, the weights of the links are given by Wx where W is an m n sparse weight generation
We note incidentally that NNE is of course very much distinct from neuroevolution (see the recent sur- vey Stanley al. 22019), which optimizes ANN architecture and hyperparameters through genetic algorithms.
2

---

Under review as a conference paper at ICLR 2020
matrix. We choose the entries of W to be random and i..d.: with probability B, (i,,)) is chosen uniformly at random from |-1, 1]. and is with probability 1-B.. 3.
The input to the network is vector y drawn from distribution D and has label (possibly real- valued) f(z). The output of the network on an imput is NNE, (v). In the simplest linear case (Section 2. y E IR' and NNE, (y) x W y. In experiments (Section 3.1 study the case when is the uniform distribution over MNIST, and NNE,, (-) is a 1-layer neural network with ReLU output gate (see Section 3.1 for formal definition).
For each genotype x, we measure its performance by computing the loss L(NNE, (v) (V()) (this could be squared loss, cross-entropy loss, etc.). For probability distribution over genotypes p. define the loss as
[(p)= Eg LNNE,, (v) L(y)).
We calculate the rewards (i) and f'(i) the expected negative loss whenever the allele is present and absent respectively.
f*(1) Exarpi EEyD LNNNE, (v))((y) |=(i) 1].
and
(1) Exange EgnD (NNE,,, [(y))] |x(() 0]. For the next generation we calculate,
and We normalize p and q to make a probability distribution. Thus the allele probabilities for the next generation will be,
= ef
(=
P p+q (+ + +e (1)(())))))
(3)
This is the standard update rule in population genetics under the weak selection assumption. The multiplier& captures the small degree to which the performance of this task by the animal confers an evolutionary advantage leading larger progeny.
Our first observation is that perfomance per allele is in fact a function of the gradient of the loss function.
Lemma
C(p')= - (()) and (i)
(46"))= = -
Proof.
C(p') EympEy-D [L(NNE, (W),(())))
= E (Eg~D (L(NNE, (v) E(v))] (x(()) =1]
+ (1 (10) [En~D [L(NNE, (v), (()))())) |x(1) =
= =p' (0) + (1- (-) -f)() -1(())(((()))-)))))) - (1)).
Here, the last line follows from equations
and 2 Now, taking the derivative w.r.t. p'(i) we get
(C(p') = -(f(()) (())
Op' (i)
We use this to prove the following theorem.
Theorem Fix 8 > 0. Suppose V²L(=) 3 H Vz E [0, Let U = su L(p) and
S. = (i [n] < (i) s -7. For c & min mmax(2U; 2/H,, 1), there is an 0 s.t.
E(C()+1)) < C(p') 7 Σ (v.((')))
iEs.
3

---

Under review as a conference paper at ICLR 2020
Proof. Using Equation 3 and Lemma
we get
p² + (1) - p1) - (f f())
c. c.)( - p1(1)) vi((g')
Ep' f(0)
1 - --((pp))
=-% V
where 7 is as defined above. For our choice of e, we have
1 1-e((p') 2 1- -U 2 and c-p'(i(1- S s 2H
Therefore, 7/ 5 11/H. Using Taylor's theorem, there exists = E [p', +1+ s.t.
(( ) - (y(+1 - v()
21 - (> -p')
<(p') - E ((((:C(D')) + H EEth (V
(Using V2((2) HT)
H
-(p') - ΣE (Ti 2 7 (V.L(p'))²
Since, vi s 1/H, we have 7/- 4 1.22 Therefore,
((p s [(p') - Σ
Vi (V.((p')) = C(p') 2(1 -c((T)) - -1(()((--100) (v.C('))
2
cō(1 5)
S L(p')
Σ (v=l(p'))²
2(1 - =BB iES.
where B := inf 1j. C(p). Therefore, the conclusion of the theorem holds for 17 co(1 6)/ (2(1 (B)).
2.1 LEARNING LINEAR FUNCTIONS
In this section, we show that in the case of linear target functions, with high probability, NNE converges to allele distribution p which is arbitrarily close to the correct linear labeling. Our NNE has m input gates connected to one output gate (i.c., no hidden layers). For a genotype x, the weights of the connections given by Wa. On input y, the NNE outputs x" w''g.
Theorem Let be the uniform distribution over vectors in an n-dimensional unit ball. Let a be fixed vector with ||a]| S 1, such that the label of y is Ely) == a² Let W have i.id. entries with W5 = + / m/d with probability d/m and O with probability with n = 0(m + log(1/6)), with probability at least 1 - 5. there exists an allele distribution p s.t.
(d/m). Then, for any 8 (0, 1},
Wp Moreover, with probability least 3/ for any (0, 1). with ((m(log(1/e)) (Wz)-a
there is an I {0, 13" s.f. [ ||]l 1
We remark that the above guarantee works for every linear target function in RT To learn, with high confidence, the target function from among d unknown (arbitrary) linear functions, above can be replaced by log d.
Proof. To have Wp- = a, with p(i) E [o, 1]. it suffices that the vector a lies the convex hull of the columns of W. This follows if the unit ball around the origin is contained in the convex hull of the vectors W(O, By duality, it suffices that every halfspacc tangent to the unit ball (and not containing it) has at least one of the W in it. For any single halfspace tangent to the unit ball, the probability that a random (( falls in is at least a constant factor - each W+( has squared length m in expectation and concentrated near it. Thus, the halfspace it defines carves out cap of constant measure. Next, by the VC theorem, n = 2(m + log(1/8)). with probability - 5 every such halfpsace will contain column of W. This establishes E(Wx)) x) a.
4

---

Under review as a conference paper at ICLR 2020
Generation t
Generation
p
update
(x p))
NNE, Ōz
ff
Training samples
networks (genolyyed)
Rewards to allcees
Figure I: Trainng NNE across generations
To bound the error, we consider the subset of columns of W that have nontrivial inner product with a and take their sum, i.c., let J {i w random variables:
1 all [ Z vm
and d [J], and consider the
y
Σ w and z d
iEJ
Then by the symmetry of the distribution of W, E(Y) points in the same direction (in all other directions, the truncated distribution remains symmetric and therefore has mean zero). For convenience. Then,
Var(
Σ SEJ
Var(w" a) S cllal|-
On the other hand,
E(I|Y)3)
1 E w S da
(dE(llw"
(P) (1))
d'E a)3) s cim
d +
d²
iJEJ
where e, c e2 are absolute constants. if d = ((m/e), then with large probability,
a ||ll|a 2 - O(e).
(4)
However, need this for every possible a. So take an (e/2)-net of the unit ball in R (which has size at most (3) For any fixed by taking d Ω m log(1/e] the Hoeffding bound tells us that the probability that (4) is violated is at most e 2dd/ < (c/4) Then, by a union bound, this bound d suffices for all Finally, with 22m (og(l//()))) (/t)/22) whp, every cap
{y a 2 1/ /m) has at least columns of W in it.
EXPERIMENTS
3.1 NNE ON MNIST
We study the effectiveness of NNE by evaluating its classification performance on the MNIST dataset.
To train an NNE via evolution of T generations of genotypes, we fix sufficiently large population size N. Each generation E [?'] consists of sample of A independently sampled genotypes from the allele distribution p', we denote this sample by Pt. This distribution is updated based on the task, in case, MNIST handwirtten digit recognition task. We let the allele distribution p' evolve over generations in this manner (see fig. T).
average performance fi) and f'()) of all the genotypes
5

---

Under review as a conference paper at ICLR 2020
Experimental setup. We use 200 training samples for each of the digits, drawn uniformly random from MNIST; we denote this set of training examples by S. p'. the allele distribution for the first generation, is sampled uniformly random from [0, 1|". We evaluate the performance of the alleles over N = 1000 genotypes.
Our network has 784 input units, one hidden layer of |h'| 1000 units with ReLU activation and an output layer of 10 units with softmax activation. We add a sparse random graph between the input layer and the hidden layer: between neuron in the input layer and neuron in the hidden layer, we independently add an edge with probability 0.1. The hidden layer is fully connected to the output layer. We choose function of only 3 fraction of the alleles. For the input sample y, {(y) is now a one-hot encoding of the label, and NNE,, (y) is the soft-max output of the network. We use the cross-entropy loss function, ,(NNE, (v) f(y)) = Eeeic E(y). log (NNE, (v)2)-
0.0025 for our experiments, i.e., each edge weight a sparse random
If classifier were to randomly guess the label of an input intance, its loss function value would be a : llg (1/10). We use the relative performance of the genotype w.r.t. to random guess for our updates. To this end, we define for a genotype I, 0z == B Eee[s] max {0, a² L(NNE, (v) ?) For each allele, we calculate the rewards f1( and (i) whenever the allele present and absent respectively.
('(1) Exere (1)(() Ezeps 2(1)
and (((1 E.Er 81 - x(i)) ΣxEri (1 xxi))
The allele distribution for the next generation is updated using equation
3
NNE described above achieves 78.8% % test accuracy the full MNIST test While this is somewhat far from the state of art in classification of MNIST images, our results demonstrate that very basic NNEs can perform reasonably well in this task.
Convergence of allele distributions. We repeat for many (hundreds of thousands) generations. As our theoretical results predict (see also Mehta et al. (2015). the vast majority of genes have allele probabilities that are very close to 0 and 1. Figure 2 shows the fraction of allele probabilities
that are at a distance [r. - 2r from 0 or 1, i.e., y is calculated as [i:miin(p" (0)
2 10K
1000
200K
2.0
0.6
Tracton of
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Distance (0.1)
Figure 2: Convergence of allele distribution after generations: x-axis shows the distance of the allele prob- abilitics from I and the y-axis shows the fraction of n allele probabilities that are a distance [ær, from or I.
NNE with output layer training. The biological implausibility objection of using stochastic gradient based updates less acute for the output layer, since in animal brains synaptic changes due to plasticity happen at the post-synaptic neuron, and for the output layer this is the output neuron. Even then, computing exact (or approximate) gradients is a nontrivial computational task; instead we consider using just the sign of the gradient for only the output layer lifetime training mechanism.
For the same network described as above, we randomly initialize the network weights using allele distribution learned using the NNE. We then calculate the sign of the gradient of the output layer weights and update the weights in the opposite direction (SignSGD), using a sufficiently small learning rate e', similar stochastic gradient descent. For in the hidden layer and j in the output

---

Under review as a conference paper at ICLR 2020
12
300
400
500
1.0
500
1000
2000
0.8
0.8
0.6
0.4
0.0
183
293
generations
(a)
(b)
generattios
Figure 3: Number of genes (n) vs performance of NNE: (a) Accuracy rates of NNE on MNIST - 4 showing the effect of number of genes on performance. (b) Similarly, we also plot the accuracy rates of NNE on MNIST 0 dataset while varying the number of genes. The accuracy trends show that more the number of genes, better the performance of NNE, but the cost of more training time.
model
MNIST to 4 4
MNIST to 9
NNE
92.1
78.8
NNE SignSGD
91.6
85.6
SGD
96 + 0.4
88.2 0.75
Table I: Accuracy rates of NNE on MNIST test digits.
layer, the update
Way := way - é' sign ((z) ((y),)h.)
(5)
where h; is output of the neuron i, and 2, softmax output of neuron (see appendix for the proof). SignSGD has been shown to effective for traning large deep neural networks (for c.g., see Bern- stein al. (018)).
We perform few hundred iterations this training using batch size 50. this experiment (NNE SignSGD), obtain 86.3% accuracy full MNIST test set. This further demonstrates that biologically plausible neural networks can perform reasonably well in this task.
Number of genes. A crucial choice for an NNE the number of genes. our experiments, we it is estimated that about 5, 000 genes
use few thousand genes; this is not unreasonable expressed in the cells of the mammalian brain. To investigate further, we compare the performance of our algorithm with increasing values of n (the number of genes). Figure presents the validation accuracy trends on the same network described above for five class ( classification and for full MNIST dataset. We observe that the accuracy rate of the network improves significantly with increase in the number of genes. However, requires much longer training time to achieve a desired accuracy rate.
Table compares the results of all the models along with the baseline, stochastic gradient descent trained on the same subset of MNIST.
3.2 DOPAMINERGIC NEURAL NETS (DNNs)
DNNs biologically plausible ANNs based dopaminergic plasticity. They learn by weak form of immediate reinforcement "rewarding" synapses whose firing led to favourable outcome. If connection between neurons has fired during training step, then its weight is increased the square error was low (less than 1). In this section, we demonstrate that simple DNNs can perform reasonably well for tasks like classifying the images in the MNIST dataset.
Experimental setup. For our experiments we use network consisting of an input layer, single hidden layer, and an output layer consisting of 784, h, and 10 neurons respectively. Each neuron in the input layer has link to each neuron in the hidden layer, and its weight is initialised by the popularly used Kaiming Uniform (more commonly called He initialisation (He et al.) (013)). These weights unchanged through out the learning process. Recent theoretical results suggest that

---

Under review as a conference paper at ICLR 2020
h
SGD
SignSGD
DNN
1000
90.34
86.84
84.84
100, 000
92.56
90.21
90.76
Table 2: Test accuracies of different models for different h.
large enough random layer sufficiently rich and efficiently trainable (Vempala & Wilmes 2019 (see also (Rahimi & Recht 2008)).
Each neuron in the hidden layer has link each neuron the output layer. The output layer outputs the softmax score. The weights of this layer are learned using plasticity based updates. On seeing an input y. the DNN tries predict the label of y; let us denote this by DNNw (v). the DNN got the prediction correct, i.e. the loss L(DNNw (u) (())) is at most co. then weight wiz get increased by small amount, provided the output neuron has low error (i.e. 12 - ((v))1 - 1/4) where 2j is the jth coordinate of DNN (s).
Formally, the update rule is as follows for in the hidden layer and j in the output layer.
max f0, 1 2; {(y))13 max {0, L(DNNw (v) ((V)) eg}
Wij = Wis + E1
[ [ 1²) (L(DNNw (v) [(w)) 6n)
Experimental results. To study the effectiveness of DNN in the 10-class MNIST digit classi- fication, we compare its peformance with some other standard baselines.
1. SGD: In this
the standard stochastic gradient descent (with the Adam optimiser Kingma & Ba (2155) based updates to train our network.
2. SignSGD: As before, we use the sign of the gradient for updates (equation 5
Table 2 shows the results for different h values. All results are after 500 epochs of training. As with NNE we use the cross-entopy loss for all the models. We found that en = 0.75 and 61 = 1 for the DNN gives reasonable performance. Our DNN gives encouraging results and is comparable to SignSGD in performance.
DISCUSSION AND FURTHER WORK
We have presented two biologically plausible mechanisms for the evolution of neural networks, motivated by the brain, and based on evolution. One feature of these mechanisms is that they process one input instance at time (i.e., unit batch size) and do not require the computation of gradients.
Our preliminary experiments with bio-plausible mechanisms suggest that they are promising alter- natives to backpropagation and the explicit use of gradients. The results raise several interesting possibilities:
In our current set-up, network weights are sparse linear functions of alleles. What if we used nonlinear functions (e.g., sigmoids ReLUs) to define the weights?
2. In our experiments, allele distributions approach 0/1 values for most coordinates. We could take advantage of this by fixing alleles that are sufficiently close 0 I and continue only on the rest.
3. Can this approach used, with greater success, for multi-layer networks?
4. Does NNE or DNN implicitly optimize the underlying architecture?
5. A recent model of memory creation and association in the mammalian brain is based on plasticity and inhibition (Papadimitriou & Vempala 2019). In this model, inhibition is implemented as cap, where the top & highest weighted input neurons of an entire layer are the ones that fire; the rest of the neurons in the layer are suppressed. Our experiments with DNN indicate that using k-cap for the hidden layer with & small as 256 when h = 100, 000 does not degrade performance (and even enhances slightly), while reducing computation. Can k-cap help with learning?
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, and Zhouhan Lin. Towards biologically plausible deep learning. arXiv preprint arxivv: 156, 2015.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. SIGNSGD: compressed optimisation for non-convex problems. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 559-5699 PMLR, 2018. URL html
mmr..
press//800/errstteein
Conrado Arturo Bosman and Francisco Aboitiz. Functional constraints in the evolution of brain circuits. In Front. Neurosci., 2015.
Reinhard Bürger. The mathematical theory of selection, recombination, and mutation, volume 228. Wiley Chichester, 2000.
Erick Chastain, Adi Livnat, Christos Papadimitriou. and Umesh Vazirani. Algorithms, games, and evolution. Proceedings of the National Academy of Sciences, 11 111029):10220-10623, 2014.
Peter Diehl and Matthew Cook. Unsupervised learning digit recognition using spike-timing- dependent plasticity. Frontiers in Computational Neuroscience, 9:99, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp- 1026-1034, IEEE doi: 10.1 D0.110//CCCV2211..222 URL https:
Computer Society, 2015. ISBN 978 1 4673-8331121 2015.122
//doi.org// www.orgg/11. 1109/ 1ww./11CV...... ICCV.
Geoffrey E. Hinton. Turing award lecture "the deep learning revolution". 1:27:43. 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations (ICLR), 2015.
Dmitry Krotov and John J. Hopfield. Unsupervised learning by competing hidden units. Proc. Nat.. Acad. Sci. U.S.A., 116(16):7723-7731 2019.
Dong- Hong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases, 2015.
Timothy P. Lillicrap, Daniel Cownden, Douglas Blair Tweed, and Colin J. Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. In Nature communications, 2016.
Ruta Mehta, Ioannis Panageas, and Georgios Piliouras. Natural selection as an inhibitor of genetic diversity: Multiplicative weights updates algorithm and a conjecture of haploid genetics. In Pro- ceedings of the 2015 Conference on Innovations in Theoretical Computer Science, 2015.
Arild Nø kland. Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems 29. 2016.
Christos H. Papadimitriou and Santosh S. Vempala. Random projection in the brain and computation with assemblies of neurons. In 10th Innovations in Theoretical Computer Science Conference, ITCS 2019, January 10- 12, 2019, San Diego, California, USA, pp. 57:1-57:19, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (eds.), Advances in Neural Information Processing Systems 20, pp. 1177-1184,
Curran Associates, Inc., 2008
Kenneth O. Stanley, Jeff Clune, Joel Lehman, and Risto Mikkkulainen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1:24-35, 2019.

---

Under review as a conference paper at ICLR 2020
Santosh Vempala and John Wilmes. Gradient descent for one-hidden-layer neural networks: Poly- nomial convergence and sq lower bounds. In Alina Beygelzimer and Daniel Hsu (eds.), Proceed- ings of the Thirty-Second Conference Learning Theory, volume 99 of Proceedings of Machine Learning Research, pp. 3115-3117, Phoenix, USA, 25-28 Jun 2019. PMLR.
Sho Yagishita, Akiko Sayyasi-TTakage Graham CR Ellis-Davies. Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. critical time window for dopamine actions on the structural plasticity of dendritic spines, Science, 345(6204):1616-1622)) 2014.
APPENDIX
Lemma neuron in the hidden layer and neuron j in the output layer, the gradient of the cross entropy loss with respect to the weights Waj in the final layer is
aL
(zj E(v)g)lu
Owi
where h2 is output of the neuron i, and 2j softmax output of neuron k.
Proof. For the hidden layer output and the final layer weights W, we compute the soft-max output as follows,
W h. so softmax(o).
True output ((y) is one-hot encoded vector and the output of the model is the softmax of the final layer input o. The derivative of loss function L w.r.t. is given by
OL
dl(y)s log(zx)
do,
0o,
1=1
a @log(zk)
1 8zy
(yy)
((y) 2x Oog
Øoj
E(y); Ozj
f(y)x Ozk
f(y)s 23 - 2z)
C
fly) (-2x2z)
Σ
=
2j 0oj
2k
0og
2j.
2k
kØj
kØj
=-((y), + {(y)szz
[ly)az) = - + Σ {(v)kz; -((y); + 2j Σ (ly) =, -
We know that 0j = E, Wiyh Hence,
0og h,,
Owsj
Then the gradient for the final layer weights wis calculated as:
aL
aL
doj
(z, - -(()))
owij
0oj
Twis
10


Reviews:
[{"review_id": "rygnNYcCFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that Artificial Neural Network (ANN) lack in biological plausibility because of the back-propagation process. Therefore, the authors provide an alternative approach, named neural net evolution (NNE) that follows evolutionary theory. This approach uses a large number of genotypes (in the form of vector with binary logits) that will evolve overtime during training. It does not require to calculate the gradient explicitly. The authors have conducted some experiments on MNIST using ANN with only one hidden layer. The experimental results show that the NNE can learn the classification task reasonably well considering that no explicit back propagation is used. \n\nI think overall the motivation to combine ANN with evolutionary theory is very interesting. The reviewer is not very familiar with evolutionary theory. So I judge this paper in the perspective of machine learning, from which I think the current approach is a week variant of back-propagation that still relies on gradient (see detailed comments below). Based on this, I give my rating. \n\nThe approach is formulated as NNE_x(y) = (x^T)*(W^T)*y. In traditional linear regression, W is the weight to be learnt. In this paper's formulation, W is named as a weight generation matrix, which is choosing to be random and i.i.d. with certain probabilities. The parameters to be optimized is x, which is named as a genotype that is viewed as a vector x \\in {0, 1}^n. So first of all, as W is fixed so the formulation is very similar to a traditional linear regression with an additional linear transform. The difference is that x is a binary vector with probabilities. These probabilities are optimized over time. \n\nFrom the Equations 1), 2) and 3), the probabilities are updated in a way to minimize the loss. This is kind of similar to back-propagation. Then the probabilities are updated and thus x is changed as well. In my understanding, this is still gradient-based optimization. I do not see it fundamental different to back-propagation. This is my main concern about this work. \n\nI did not check the details of Theorem 1. Could the authors please comment what is the purpose of Theorem 1 before proving it? This part is unclear to me in this paper. \n\nOne more question, for the W matrix,  the authors choice beta = 0.0025 in the experiment. Is there any particular reason for this choice? Or does it matter what value to choice as it is fixed anyway? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eZ0ExCuH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, I must confess that my knowledge is quite limited to read this paper. Perhap the authors present something that I can not catch up at the present.\n\nI conjecture the paper would like to bring the evolution in genetics and perhap brain cirecuits as well to define a novel neural net model, called NNE by the authors. \nThe paper is somewhat cumbersome in the introduction that makes the reader (here is myself) can not understand the main idea. At first the authors introduce about evolution in genetics and genomics that is a bit different from what I known. Then the authors claim that they can show that the brain circuits can evolved in their model.\nThere are so many mistake and/or typos in sentences and in mathematical formulation. These make me can not finish reading the paper. I have to stop reading at the end of Section 2.\nHere are my concerns and questions:\n1). What is \"STDP\" in the 2nd papragraph in Introduction?\n2). in the 3rd papragraph in Introduction: \"Suppose that the brain circuitry for a particular classiﬁcation task, such as “food/not food”,is encoded in the animal’s genes, assuming each gene to have two alleles 0 and 1\". This is realistics, the allels in animal is 0 1 or 2 if encoded.\n3). The authors use the words \"gene, genotype, phenotype\" in a special way that is different to what I known in genomics (in GWAS).\n4).  in the 3rd papragraph in Introduction: \"At each generation, a gene is an independent binary variable with ﬁxed probability of 1\". What do you mean by fixed probability of 1? I can not understand in anysense that I know.\n5). In Section 2, What is n? the authors start the mathematical formulation but I can not find out what is n? Is it the sample size?\n6). In Section 2, paragraph 2, you define y~ \\mathcal{D}, BUT then in all formulas later you denote y ~ D. What is D??? I can not understand.\n7).  In Section 2, paragraph 2, you define a label of y as \\ell(y), BUT then in the 1st sentence of the 3rd paragraph in Section 2 you wrote    L(NNE_x(t), l(y)). What is l(.) here ???\n8). In the equations (1) and (2), what is p^t  ???  You have NOT defined it.\n9). Right after equations (1) and (2), What is \\epsilon ???? Can NOt understand.\n10). The sentence right after the equation (3): \"This is the standard update rule in population genetics under the weak selection assumption.\" This is NOT trivial to me, and even the machine learning comunity, we do not know this rule, it is not obvious. PLEASE provide exact reference.\n11). the first equation in the PROOF of LEMMA 1 wrong  \\mathcal{L} (p^t)  should be equal to E_{x~p^t}  NOT p.\n12). in the PROOF of Theorem 1, I can NOT find out where \\gamma has been defined?\n13). What is d  in Theorem 2? is it the dimension? I make too many guesses !!!\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByedtDasKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose a method for training neural networks using evolutionary methods. The aim of developing this method is to provide a biological alternative to back-propagation. The authors prove that their method converges and with high probability succeeds in learning linear classification problems. Another method is also proposed which is linked to dopaminergic neurons.\n\nIn terms of presentation, the paper is generally clear and well-written. I was not able to assess the importance of the theoretical contributions of the work as my research is not in this area, so my comments are limited to the other aspects.\n\nWith regard to the biological plausibility of the method, it is unclear to me how the evolutionary method proposed here can enable learning in typical scenarios such as conditioning experiments in animals. The learning processes in animals typically occurs in short time spans (for example a few training sessions for conditioning to stimuli predicting food/no food) and therefore I don’t find it plausible to suggest evolutionary methods across generations are behind such forms of learning. Perhaps what the authors have in mind applies more to other forms of behaviour such as innate and involuntary responses in animals formed across generations rather than ongoing updates in synaptic plasticity as an animal adjusts its behaviour using environmental feedback. But then in this case the biological plausibility of the method seems fairly limited and not really an alternative to methods such as back-propagation.\n\nThe other biological aspect of the proposed work is the connection to dopamine and using the sign of gradients for updating the weights. I think connecting the current learning rule to the activity of dopamine neurons requires quantitative comparisons with experimental data, otherwise although I agree that the method is biologically inspired, but whether it is biologically plausible is not clear.\n\nBased on the above comments, I think the work will benefit from further developments before being ready for publication.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Unfortunately the paper is confusingly written, and there is only agreement by all reviewers on the rejection of the paper.  Indeed, if all reviewers and the area chair do not interpret the paper well, the authors' best response would be to rewrite the papers rather than disagree with all reviewers.

In the area chair's opinion, the current form the paper does not merit publication.  The authors are advised to address the reviewers' concerns, rework the paper, and submit to a conference again.
Decision: Reject