Learning with Plasticity Rules: Generalization and Robustness
Authors: Rares C Cristian
Max Dabagia
Christos Papadimitriou
Santosh Vempala
Venue:  (2021)
Keywords: meta learning
plasticity
local learning
deep learning
machine learning
neural networks
RNNs
backpropagation
perceptron
evolution
adversarial examples

Abstract:  Brains learn robustly, and generalize effortlessly between different learning tasks; in contrast, robustness and generalization across tasks are well known weaknesses of artificial neural nets (ANNs).  How can we use our accelerating understanding of the brain to improve these and other aspects of ANNs? Here we hypothesize that (a) Brains employ synaptic plasticity rules that serve as proxies for GD; (b) These rules themselves can be learned by GD on the rule parameters; and (c) This process may be a missing ingredient for the development of ANNs that generalize well and are robust to adversarial perturbations. We provide both empirical and theoretical evidence for this hypothesis. In our experiments, plasticity rules for the synaptic weights of recurrent neural nets (RNNs) are learned through GD and are found to perform reasonably well (with no backpropagation).  We find that plasticity rules learned by this process generalize from one type of data/classifier to others (e.g., rules learned on synthetic data work well on MNIST/Fashion MNIST) and converge with fewer updates. Moreover, the classifiers learned using plasticity rules exhibit surprising levels of tolerance to adversarial perturbations. In the special case of the last layer of a classification network, we show analytically that GD on the plasticity rule recovers (and improves upon) the perceptron algorithm and the multiplicative weights method. Finally, we argue that applying GD to learning rules is biologically plausible, in the sense that it can be learned over evolutionary time: we describe a genetic setting where natural selection of a numerical parameter over a sequence of generations provably simulates a simple variant of GD.

Under review as a conference paper at ICLR 2021
LEARNING WITH PLASTICITY RULES: GENERALIZA- TION AND ROBUSTNESS
A Anonymous authors
Paper under double-blind review
ABSTRACT
Brains learn robustly, and generalize effortlessly between different learning tasks; in contrast, robustness and generalization across tasks are well known weaknesses of artificial neural nets (ANNs). How can we use our accelerating understanding of the brain to improve these and other aspects of ANNs? Here we hypothesize that (a) Brains employ synaptic plasticity rules that serve as proxies for Gradient Descent (GD); (b) These rules themselves can be learned by GD on the rule parameters; and (c) This process may be a missing ingredient for the development of ANNs that generalize well and are robust to adversarial perturbations. We provide both empirical and theoretical evidence for this hesis. In our experiments, plasticity rules for the synaptic weights of recurrent neural nets (RNNs) learned through GD and are found to perform reasonably well (with no backpropa- gation). We find that plasticity rules learned by this process generalize from one type of data /laassiiii to others (e.g., rules learned on synthetic data work well on MNIST Fashion MNIST) and converge with fewer updatos. Moreover, the classifiers learned using plasticity rules exhibit surprising levels of tolerance to adversarial perturbations. In the special case of the last layer of classification network, show analytically that GD the plasticity rule recovers (and improves upon) the perceptron algorithm and the multiplicative weights method. Finally, we argue that applying GD to learning rules is biologically plausible, in the sense that it can be learned over evolutionary time: we describe genetic setting where natural selection of numerical parameter over sequence of generations provably simulates a simple variant of GD.
INTRODUCTION
The brain is the most striking example of learning device that generalizes robustly across tasks. Artificial neural networks learn specific lasks from labeled examples through backpropagation with formidable accuracy, but generalize quite poorly to different task, and are brittle under data perturbations. In addition, it is well known that backpropagation not biorealistic it cannot be implemented in brains, as it requires the transfer of information from post- to pre-synaptic neurons. This is not, in itself, a disadvantage of backpropagation unless one suspects that this lack of biorealism limits ANNs in important dimensions such as cross-task generalization, self-supervision, and robustness.
We believe that the quest for ANNs that generalize robustly between learning tasks has much inspiration ion to gain from the study of the way brains work. In this paper we focus plasticily laws controlling changes of the strength of synapse based on the firing history as at the post-synaptic neuron. We provide evidence, both experimental and theoretical, that (a) In the case of RNNs, plasticity rules can successfully replace backpropagation and GD resulting in versatile, generalizable and robust learning; and (b) These rules can be learned efficiently through GD on the rule parameters.
rules (Dayan and Abbott 2001)
Plasticity Rules. Hebbian learning ('fire together wire together" Hebb 19499 ) is the simplest and most familiar plasticity rule: If there a synapse (i.j) from neuron to neuron and at some point fires and shortly thereafter j fires, then the synaptic weight of this

---

Under review as a conference paper at ICLR 2021
X
9
(a) 3 ANN
(b) RNN
Figure 1: Feedforward networks vs RNNs
synapse gets an increment. Over the seven docades since Hebb, many forms of plasticity have been observed experimentally and/ /rr formalized analytically, many of them quite sophisticated and complex, see Dayan and Abbott (2001) for an exposition. All of them dictate change increment or decrement - in the synaplic weight. of synapse (i.j) provided neurons i and both fired some pattern. Intuitively, the decision for the application of a plasticity rule takes place at the post-synaptic neuron j, since receives information from the firing of both i and itself. This is consistent with our understanding of the molecular mechanisms that determine synaptic strength, all of which complex chemical phenomena. taking place at (the dendrite of) j.
In this paper consider plasticity rules objects that can be learned. This fits with the view that existing mechanisms have presumably changed over evolutionary time and are known to differ in their details from one animal species to another. We show experimentally that RNN can meta-learn a plasticity rule that allows it to learn to perform a classification task without backpropagation. This meta-learning is done by GD on the parameters of the rule. Interestingly, the same plasticity rule then performs well on very different tasks and data sets. There are many ways paramcterize a plasticity rulc, from full table specifications to small neural networks that take as input observed activation sequence at both ends of a synapse and output the change to the synaptic weight.
Why RNNs? RNNs are inspired by, and can model, recurrent activity observed in the brain; they are also especially well-suited to plasticity rules. To illustrate, suppose that we want to train the feed-for fordofr ANN in Figure 1(a) with plasticity rule. It is clear that the space of possible rules rather meager. In order to change the weight of link (i, after each labeled example, node j will decide the nature of the change based on local information, namely, whether or j or both fired during this epoch. Thus any learned plasticity rule must be some slight generalization of Hebb's rule
But suppose instead that the three hidden layers have been collapsed into one, resulting in the RNN shown in Figure 1 (bb, and this collapsed layer fires three times before readout, roughly simulating foedforward 3-layer network. Now node j knows much more about what happened to link (i,j) during these three epochs; such information was inaccessible in the feedforward setting. Any 2 x 2 matrix of reals is a possible plasticity rule, where 2 the number of possible firing patterns the second, fired in the third," or "101" - for each of i and j, and the entries of the matrix denote increments/ decrements, additive or multiplicative, of the weight of link (i. one a task, it is possible that this rule may be an
such as "fired in the first round, did not fire in
updates the entries of this rule by training adequate proxy for the update calculated by backpropagation, Furthermore, we might hope that this rule may even generalize well, performing far above baseline on very different tasks. Evolution. We proposed to replace GD in deep learning by biorealistic plasticity rules, and then use GD to learn the plasticity coefficients. Are contradicting oursclves? After all, the brain did not develop its plasticity rule(s) through GD, but through evolution. But since
We could update all incoming links to node j based on the firing status of all of them: Zenke al. (2015) suggests that such complex rules may be indeed at work in the animal brain. See the discussion for more on this intriguing research direction.
2

---

Under review as a conference paper at ICLR 2021
GD apparently produces good plasticity rules, the question arises, is evolution at all like GD In Section C we address this question analytically. In particular, we prove a general result (which is of some interest by itself in Evolution) stating that the evolution of any real parameter of the phenotype affecting fitness (such as the parameters of the plasticity function) is approximately equivalent to simple (and suboptimal) variant of GD, as long as the parameter is expressed as the sum of large number of small genetic contributions (as is known to be the case for many common traits, such as height in humans); the full details are in Appendix [ Hence, it is reasonable to assume that the tuning of such paramcters could have been achieved over evolutionary time.
Summary of Results. Could such plasticity rules serve as effective learning algorithms? As we show in the following sections, the answer is affirmative: in the special case of the simplest possible network, with no hidden layer and applied to binary classification task, learning the plasticity rule through GD recovers two classical supervised learning algorithms, the Perceptron algorithm and the Multiplicative Weights (or Winnow) algorithm. We proceed to experiment with learning more complex plasticity rules in a general RNN, establishing that learning plasticity rules leads to performance that is quite good. Even though the performance is not at the same level as ANNs, our experimens show that learning through plasticity has three imporlant benefits: (1) It generalizes well across learning lasks; (2) its convergence to a good classifier is more rapid, iee., the number of updales (measured by the total number of samples) needed is significantly fewer; and (3), and perhaps more striking, classifiers learned this way appear to be considerably more robust to adversarial perturbations than classifiers learned using GD. An intriguing result here is that the robustness appears to increase significantly with the depth (number of rounds) of the RNN.
1.1 RELATED WORK
Plasticity Rules. Motivated by the brain, learning with plasticity rules has also been studied in machine learning. Early work of Bengio al. (1900) suggested genetic algorithms for doing so, and later [engio et al. (1992) explored gradient-based methods as well Hall Floreano and Urzelai (0000) applied evolving Hebbian plasticity rules to randomly initialized weights for a robot navigation task, while Miconi et al. (2018) introduced differentiable plasticity with a plasticity parameter for every edge of a network, which also evolves over time, and applied this to large, high-dimensional data sets. More recently, work by Yaman et al. (2019) is in similar spirit as ours but with important differences: they apply plasticity rule updates to a specific small 2-layer ANN and find it beneficial; we focus on how rules learned for one task on one network apply to other tasks on other networks, and on the robustness properties of learning through plasticity.
Other Update Schemes. There is a variety of mechanisms other than plasticity available to modulate the brain's synaptic weights. Rather than trying to learn more complex plasticity rules, Lillicrap et al. 2020 argue that hand-designed local update rules are sufficient in the presence of feedback connections, and that these are plausible mechanism for learning in the brain. Whereas backpropagation directly calculates each parameter's 's contribution to the overall loes, a feedback path with appropriate learned weights can approximate this signal in its stead (such feedback paths are known to be present in the visual cortex). In particular, Metz et al. (2018) learned an update rule which trains both the forward and backward paths and generalizes effectively across tasks, while a body of previous work (sce Sacramento ct al. (2017); Guerguiev et al. (2017)) has demonstrated that well-known mechanisms from neurobiology can coordinate these lorward and backward paths to learn in an online fashion.
Taking a different tack, Wang et al. (2018) train an RNN to implement general reinforcement learning algorithm, which bears some conceptual similarities to our scheme of learning a general plasticity rule. Here, the meta-learning procedure by which the network's weights are updating is analogous to the action of the dopamine system on the neurons of prefrontal cortex, but when applied to novel tasks the network's 's weights are frozen. Finally, Andrychowicz et al. (2016) and more recently Maheswaranathan et al. (2020) parameterize gradient- based optimizer and then optimize these parameters, which 18 similar in implementation to our strategy for learning plasticity rules.
3

---

Under review as a conference paper at ICLR 2021
Meta-Learning with Evolution. The plausibility of optimizing meta-learning parameters through evolution has been studied in the literature under the term neuroevolution, see Floreano et al. (2008); Stanley et al. (2019). In particular, evolving plasticity rules is Truitful line of research in its own right (see Soltoggio et al. (2018) for a review). although it uses genetic algorithms to explicitly evolve better architectures and learning rules. Here we use standard models in population genetics to show that the evolution of numerical parameter can be done in GD-like fashion.
Adversarial Robustness. Lastly, the existence of adversarial perturbations, and in particu- lar learning to avoid them, has been an active topic in recent years, beginning with Gooddellow et al. (2014) and continuing with Madry et al. (2018); : Iyys et al. (2019). Crucially, these methods achieve robust classification by explicitly regularizing the objective function of the network to counter an adversarial attack. We focus on learning methods which by themselves
happen to converge to minima that are robust to adversarial perturbations without explicitly searching for them.
LEARNING (WITH) PLASTICITY RULES
Define the RNN plasticity rule r: {0,1)T 1) x {0, - R to be a function that maps a pair of binary vectors to a real number. The binary vectors correspond to the firing patterns of two neurons i,j connected by a synapse (i.j) in a T-round recurrent network. Similarly the output layer plasticity rule is defined by ro {0, 1)" x {0, 1), the binary vector again describing the firing pattern of a neuron, and the 0/1 value describing whether node in the output layer corresponds to the true label or not. The functions r, to indicate the change to the synарse weight, which can be additive or multiplicative. For example, Hebbian plasticity correspouds to the AND function with T = 1. During supervised learning, the plasticity rules are applied independently to each synapse. There are two alternatives here: (1) apply plasticity rules only in the event of disagreement between the network's 's output and the true label of the training example. That is, we assume that, besides the local firing information, the plasticity mechanism also receives a signal about the loss of the current training example; it is known from animal experiments such as Yagishita et al. (2014) that this does happen in the mammalian striatum and cortex through the excretion of dopamine. (2) we apply training rules on all training examples. This requires even lesser coordination, and the time-scales of dopamine action are not an issue. In our experiments, we find that both modes perform equally well (see Fig, in the Appendix). Moreover, the second mode incorporates error information only at the oulput layer (where the correct label is known), making it completely unsupervised throughout the rest of the network. To learn a plasticity rule, we select model and dataset to train with, and then randomly initialize rule. We apply a standard loss function to the output of this network (e.g. cross-entropy loss for classification), but as function of the parameters of the rule. GD can then be used to update these parameters to minimize the loss function.
Training. Our architecture is similar to an RNN. The network consists of an imput layer connecting the imput to a directed graph G = GVV,E), E), and a fully-connected output layer connecting G to the output nodes. We generate G at random, choosing each edge with probability Let E 3dxx[V denote the weights of the input layer, W E RVVII the weights of G, and U RIVIx the weights of the output layer. Over the course of T rounds, we maintain a hidden vector h E R initialized to zero, and updated as h t cu (a - Ax)) where x E Rd is the input, a is ReLU activation function, and C 32IV + 38V is notion of a cap, a biologically plausible activation function implementing the excitatory-inhiiitor balance of a brain area, see [Papadimitriou and Vempala (2019). Given a vector u, c₁ (u) returns a copy of with only the highest. k entries remaining; the rest are set to zero. If at the end of round h; is nonzero, we say that the corresponding unit has fired. The output layer consists of linear combinations U of the hidden vector components (one output per label), and final softmax is then applied. We will refer to this particular architecture as the simple R.N. Given plasticity rules, we train a network as follows. For each individual example in the dataset, we run the forward pass and keep track of the firing sequences of each node. Using these firing sequences, we update the graph using the RNN rule r, and the output layer according to ro described previously.
4

---

Under review as a conference paper at ICLR 2021
Landscape of rules. Any function which maps appropriate binary vectors to real numbers defines a rule. An RNN rule can be any function r : {0, 13" x {0,1)" - R, and the output rule can be any To {0, x {0. 1) - R. We consider two different parameterizations: (1) Table: r and ro are look-up tables of size 2 x 2 and 2 x 2, respectively. The entries are the parameters we learn. (2) Small NN: and ro are defined by small auxiliary neural networks. These networks take as imput the activation sequences, say the concatenation of 82, and output the update value, r(s1, 52). In this case, the weights of the auxiliary network are the parameters we Iearn.
Efficiency. Using tables to represent the plasticity rules is more expressive but requires an exponential number, (2 of parameters. On the other hand, the complexity of the second method depends only on the size of the auxiliary network, which is independent of the simple RNN size, and its input, the activation sequence, grows linearly 27. We found that training using plasticity rules converges with a significantly smaller number of updates compared to GD. See Fig 2 for comparison of the two methods.
Convergence Accuracy
80
Accuracy (%)
60
50-
40
30-
TuainTr RNN
20
GD-Trained RNN [batch-1)
10
GD-Trained RNN (batch-200)
20000
40000
60000
80000
100000
120000
Number Updates
Figure 2: On the standard MNIST data set, we trained the same underlying RNN with T |V| = 1000 with an output layer plasticity rule, and separately with GD (using the standard Adam optimizer, learning rate 10-2) on the output weights. Note that we did not optimize hyperparameters such as batch size and learning rale. This is only meant to show that plasticity- based training is competitive with gradient methods.
Data sets. We use six different datasets. In the first four, 10,000 points are generated from a 10-dimensional normal distribution and assigned binary labels by linear threshold function (Halfspace), two different ReLU networks each with a single hidden layer of width 1000 and randomly initialized weights (ReLUI and 2), and simple RNN with T = 3, 3/|V|= 100, A 50. The last two datasets are the MNIST and Fashion MNIST benchmarks.
3 CROSS-TASK GENERALIZZAIION WITH PLASTICITY RULES
GD is general method of optimization, capable of improving the performance of any model for which gradients can be computed. The obvious question is whether plasticity rules offer similarly general strategies for updating the weights of a network. We find that rules learned from simple, low-dimensional datasets generalize to accurately classify data in higher dimensions labeled by much more complex functions, see Fig 3 First, we examine the empirical evidence, and exhibit experiments which demonstrate the remarkable capability of these rules to generalize across tasks. Then we analyze output-layer rules, capturing well- known provable methods for learning linear threshold functions. To test the generalization abilities of these plasticity rules, we learn a rule for a particular network and dataset, and then use it to train other architectures to classify other datasets. In the first experiment, we separately learn output and RNN rules for small networks. With these fixed rules in hand, we then re-train a feedforward and recurrent network of both small and large sizes on all six of our datasets. The results are clear (see Fig 3) all four models perform well on other datasets, although the large recurrent network consistently outperforms the other
5

---

Under review as a conference paper at ICLR 2021
100%
Small
T=1
Test Accuracy
75%
Large
Small
50%
Large
Halfspace
MeLU1
PeeLUZ
Feshion-MNIST
Figure 3: Comparison of various models on different datasets trained using the same plasticity rules. We first. learned a plasticity rule for the output weights small feedforward network (i.e. [V] 100, T = on the Halfspace dataset, and plasticity rule for the graph weights of a small recurrent network (i.e. [V| = 100, T' = 3) on the ReLUl dataset, We then hold these rules fixed and use them to re-train both of the small models and additionally large models ([V) = 1000) on all six datascts, restricting MNIST and Fashion-MNIST to only a random 10,000 training examples in the interest of fair comparison. Averages of 10 re-trainings for each model/dataset dataset combination are shown above.
three models. We have empirically observed a significant increase in accuracy as compared lo network of the same size when using recurrent network with ils weights updated by plasticity rule (Fig. 3h, with the improvement most obvious on the more nonlinear datasets. Significantly, certain datasets the small recurrent network even outperforms the large feedforward network, suggesting that learned recurrent weights can compensate for fewer neurons. Moreover, a rule learned on one dataset appears to generalize well to others. Thus, it appear that an appropriate RNN plasticity rule represents general strategy for producing separable representations, although an explanation of how these rules work, let alone whether they are optimal, remains elusive.
3.1 ANALYZING THE OUTPUT LAYER PLASTICITY RULE
We first examine update rules for the output layer alone, with the goal of learning a synaptic plasticity rule to update the output layer weights. It is well-known that training just the output layer minimize well-known loss functions is a convex optimization problem that can be solved efficiently; GD provably works with specialized variants under different assumptions on the data. It has also been established that training just the output layer feedforward network, with random weights and a sufficiently wide penultimate layer can provably achieve high classification accuracy (Rahimi and Recht 2008 Vempala and Wilmes 2019).
The classical perceptron algorithm for learning linear threshold function f(x) sign(w" the following iteration, starting with w 0:
While there misclassified example x, w <- w +x((x).
This is guaranteed converge
halfspace consistent with all the labels in most | max, || (min, ||w x=])² iterations Rosenblatt (1962); Minsky and Papert (1969). To map this our setting, we learn a network with single output neuron, and assume each I; E 1]. Then this corresponds to the output layer rule in Fig. 1 which depends on the (incorrect) prediction value p(z) = sign(w æ). This an additive update rule. The
Perceptron (additive) -1
MW (multiplicative)
p(x)
p(x)
p
p(x) =
2;
2;
T;
3;
Table 1: The plasticity rules for the Perceptron and MW algorithms
Multiplicative Weights algorithm
Littlestone 19877 can be mapped to similar multiplicative

---

Under review as a conference paper at ICLR 2021
plasticity rule. Recall that MW only acts examples where the current hypothesis predicts incorrectly, and then on variables that are "ON", doubling the corresponding weight if the true label is 1, and halving if the true label is -1.
Our first theorem is that very similar plasticity rules for the output layer can be automatically discovered in general setting, i.c., an effective output layer rule can be provably meta-learned.
Theorem 1. GD initialized to zero, converyes to a rule with sign pattern [-,+; -].
odditive output rule, from any starting rule, and network weights
In fact, GD provably optimizes the output layer rule. The next theorem follows from the observation that the cross-cntropy loss is a convex function of the outer layer weights, which are linear functions of the output layer rule for any fixed graph and sequence of examples. The problem of finding the output layer update rule that minimizes the cross entropy loss is conver optimization problem.
Theorem 2.
To explain the generalization itself, we offer a modest (but rigorous) guarantec. In the next section, we will extend this to data that is not perfectly separable.
Theorem 3. Let r -== [-aa, a; bb] be an oulpul layer plasticily rule with b > 0. For data in 1]" that are strictly linearly seporable by a wnit vector w with Σi= w = O, applying this rule to the weights of linear threshold network converges to a correct classifier.
4 ADVERSARIAL ROBUSTNESS OF LEARNING WITH PLASTICITY RULES
A prevalent attack method is the Fast Gradient Sign Method, first proposed in Goodiello let al. (2014), which uses the following single step update: I x+ a sign (,, y)) where L is the loss function, is the input wish to perturb, and y is the true label. We use more powerful adversary, allowing for (1) multiple gradient steps as in Madry ct al. (2018), (2) moving directly in the direction of the gradient, instead of using only its sign, in RRoza et al. 2016), and (3) targeting a specific class y' that we wish to misclassify the image with:
x t+1 Hs+s(
(LL(x,)))
where S is the set of allowed perturbations, and (1) is the projection of a vector onto the set x + S. For a given network and image, we generate nine adversarial images, one for each value of y' * y. If any of the nine resulting perturbations become misclassified, then we count the original image as misclassified under perturbation (sce Appendix A.3 for details). For MNIST, we restrict to perturbations that lic within an ball around the original x, and to pixel values in the interval [0, 1]. We generate an adversarial dataset for both plasticity and gradient trained networks for increasing values of Figure 4 shows adversarial images for a rule-trained network need to be considerably more noisy than their GD-trained counterparts.
2
3
4
/
2
3
4
Notutee
2
3
4
Figure 4: Adversarial perturbations on MNIST (left) and FashionMNIST (right) for GD-trained network, and a taraiie network. Original images are in the top row.
Figure clearly shows that plasticity rules create more robust classifiers than GD. Madry et. al. (2018) explored the relationship between model capacity and adversarial robust.esss
7

---

Under review as a conference paper at ICLR 2021
Accuracy Against Perturbed Examples
300
Rulc Leoming (T=3)
Rute Leaming (T=1)
GD Leariig (1w5)
Accusscy
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Pertarbatton Magnitude
Figure The same network with |V| 1000, cap 500 was trained with plasticity rules for T = 3, and with GD. For cach trained network, we generate adversarial data-scts with increasing perturbation magnitude.
noting that a larger capacity is needed in order be robust than to simply classify benign examples. We observe that deeper networks are more robust than shallow ones.
How to explain this robustness? One possible explanation is the following: the RNN finds a rich representation, one in which the examples with different labels can be separated with large margins. More precisely, for most. correctly labeled data points, e-balls around them are also classified with the same label. Large margin learning, a celebrated success of Support Vector Machines (Cortes and Vapnik] 1995 Vapnik] 1998), could explain robustness if large margins exist in suitable kernel space. We show that similar result holds for a small plasticity based learning, provided we also update on correctly classified examples that are within small margin of the threshold (sec Theorem 4 in the Appendix). The success of RNN plasticity rules in finding a representation amenable to more robust classification is intriguing and merits rigorous explanation.
DISCUSSION
Learning is the modification of the long-term state of an organism or other system caused by experience; such modification is effected by the system's learning mechanism. Meta-learning then must be the structure or parameters of the learning mechanism that remain invariant across learning experiences. In animal brains, synaptic plasticity is just about the only mechanism that qualifies; done through plasticity.
learnin happens in the animal brain, we propose that it is
Can these lessons be useful for ANNs? Here we focus on RNNs, because they afford a richer space of synaptic plasticity mechanisms, and we demonstrate that plasticity rules can be learned through GD which (1) achieve reasonably effective learning on a variety of training data without backpropagation; (2) the same rules learned on data set also perform quite well on new data a different sort, and on a graph with a different wiring; and (3) these rules can train models which are naturally and significantly more robust to adversarial attacks. Furthermore, in the case of the rules for the output layer, GD produces plasticity rules which recover basic learning algorithms such as the Perceptron and Winnow. We also make the point that learning plasticity rules through GD is biologically plausible, in the sense that learning of any parameter through evolution is, under assumptions, possible through a process which is lantamount to variant. of GD.
We believe that our ideas and results point to a rich and promising field of inquiry. Plasticity in the input layer would probably enhance learning, but would it hurt generalization? Can plasticity rules more complex than the output rule also be dissected analytically? Can plasticity rules work for feed-forward networks? Our observation in Footnote makes this direction worthy of further experimental exploration. Are there ways to combine plasticity with backpropagation to enhance generalization while maintaining learning performance? In our experiments, much of the improvement in accuracy is achieved by the output layer, yet. learned RNN rules still provide small but constitent. increase in accuracy over random
8

---

Under review as a conference paper at ICLR 2021
# Rounds
MNIST Acc.
Fashion MNIST
Robustness on MNIST
T = 3
87%
77%
2 60%, = 4: 36%
= ==1
93%
81%
e 2 12%, = 4:
85%
70%
= 2 00%, = 4: 0%
Table 2: Each experiment uses graphs with [V] = 1000, k = 500 and 2 epochs of training.
We ran two separate runs for == The starred entry has all entries of the input weights equal to one (normally, we let these be random from normal distribution). It is unclear why such an initialization produces such a stark improvement in accuracy on MNIST.
fixed weights; we suspect that larger experiments with more than two iterations of the RNN would result in higher accuracy through RNN plasticity. On evolution, are there more general, and more parsimonious, schemes for which evolution is Lantamount to GD? Does plasticity also enable self-superviision for example, the creation of powerful represenlations from unlabeled corpora, as happens both in the language module of the human brain and in modern NLP? Output-layer plasticity can be interpreted in terms of familiar learning algorithms. How can we interpret the much more complex RNN plasticity rules learned? So far, we have been able only make partial sense out of them. Finally, what is the full range of algorithms that can be realized as synaptic plasticity rules? Docs this view, motivated by neural plausibility, yield an interesting complexity-theoretic viewpoint?
REFERENCES
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfan, D., Schaul, T., Shillingford, B., and De Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. Tn Advances in neural information processing systems, pages 3981-3989.
Bengio, S., Bengio, Y., Cloutier, J., and Geesei, J. (1992). On the optimization of a synaptic learning rule. In Preprints Conf. Optimality Artificial and Biological Neurol Networks, volume Univ. of Texas.
Bengio, Y., Bengio, S., and Cloutier, J. (1990). Learning a synoptic learning rule. Citeseer.
Chastain, E., Livnat, A., Papadimitrion, C., and Vazirani, U. (2014). Algorithms, games, and evolution. Proceedings of the National Academy of Sciences, 111(29):162001002) 10623.
Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3):273 297.
Dayan, P. and Abbott, L. F. (2001). Theoretical neuroscience, volume 10. Cambridge, MA: MIT Press.
Floreano, D., Dürr, P., and Mattiussi, C. (2008). Neuroevolution: from architectures to learning. Evolutionary intelligence, 1(1):47 62.
Floreano, D. and Urzelai, J. (2000). Evolutionary robots with on-line self-organization and behavioral fitness. Newral Networks, 13(4-5):431-44))
Freund, Y. and Schapire, Machine learning, 37(3):277-29) -96..
(1999). Large margin classificaioon using the perceptron algorithm.
Goodfellow, I., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXio 1412.6572.
Gorantla, S., Louis, A., Papadimitriou, C. H., Vempala, S., and Yadati, N. (2019). Biologically plausible neural networks evolutionary dynamics and dopaminergie plasticity. NewrIPS workshop on Real Nearons and Hidden Units.
Guerguiev, J., Lillicrap, T. P., and Richards, B. A. (2017). Towards deep learning with segregated dendrites. ELife, 6:e22901.
Hebb, D. O. (1949). The organization of behavior: nearopsychological theory. Wiley, New York.
Hinton, G. F. (2019). Turing award lecture "the deep learning revolution", 1:27:43.

---

Under review as a conference paper at ICLR 2021
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. (2019). Adversarial examples are not bugs, they are features. In Advances Neural Information Processing Systems, pages 125-136.
Lillicrap, P., Santoro, A., Marris, L., Akerman, C. J., and Hinton, G. (2020). Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12.
Littlestone, N. (1987), Learning quickly when irrelevant attributes abound: A new algorithm (extended abstract). In FOCS, pages 68-77.
Madry, A., Makelov; A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In Indernational Conference on Learning Represenlations.
Maleswaranathan, N., Sussillo, D., Metz, L., Sun, R., and Sohl-Dickstein, J. (2020). Reverse engi- neering learned optimizers reveals known and novel mechanisms. arXiv preprint arXie:2011.0111
Metz, L., Maheswaranathan, N., Cheung, B., and Sohl-Dickstein, (2018). Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv:1804.0022
Miconi, T., Stanley, K. O., and Chune, J. (2018). Differentiable plasticity: training plastic nenral networks with backpropagation. In ICML.
Minsky, M. and Papert, (1969). Perceptrons: Introduction Computational Geometry. The MIT Press.
Nagylaki, T. (1993). The evolution of multilocus systems under weak selection. Genefics, 134(2):627- 647.
Papadimitrion, C. H. and Vempala, S. (2019). Random projection the brain and computation with assemblies of neurons. In 10th Innovations in Theoretical Computer Science Conference, ITCS 2019, January 10-12, 2019, San Diego, California, USA, pages 57:1-57:19.
Rahimi, A. and Recht, B. (2008). Random features for large-scale kernel machines. In Platt, J. C., Koller, D., Singer, Y., and Roweis, S. T., editors, Advances in Newral Information Processing Systems 20, pages 1177 1184. Curran Associates, Inc.
Rosenblatt, F. (1962). Principles of Neurodynamics. Spartan Books, Washington, DC.
Rozsa, A., Rudd, E. M., and Boult, T. E. (2016). Adversarial diversity and hard positive generation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 25 32.
Sacramento, J., Costa, R. P., Bengio, Y., and Senn, W. (2017). Dendritic error backpropagation in deep cortical microcircuits. arXie preprint Xrxiv:1001.0000 00.000000......
Signer-Hasler H., Flury, C., Haase, B., Burger, D., Simianer, H., Leeb, T., and Rieder, S. (2012). A genome-wide association study reveals loci influencing height and other conformation traits in horses. PLOS ONE, 7(5):166.
Soltoggio, A., Stanley, K. O., and Risi, S. (2018). Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. Neural Networks, 108:48 67.
Stanley, K. O., Clune, J., Lehman, J., and Müikkulainen, (2019). Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):24-35.
Vapnik, V. N. (1998). Statistical Learning Theory. John Wiley and Sons Inc., New York.
Vempala, S. and Wilmes, J. (2019). Gradient descent for me-hhideen-aee layer neural networks: Polyno- mial convergence and sq lower bounds. In Beygelzimer, A. and Hsu, D., editors, Proceedings of the Mhirt--Sccone Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 3115 3117, Phoenix, USA. PMLR.
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., and Botvinick, M. (2018). Prefrontal cortex as a meta-reinforcement learning system. Nature neuroscience, 21(6):860-88) 868.
Yagishita, S., Hayashi Takagi, A., Dlvies,avies, G. C., Urakubo, H., Ishii, S., and Kasai, H. (2014). A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 345(6204):1616-1662 1620.
10

---

Under review as a conference paper at ICLR 2021
Yaman, A., Iacca, G., Mocanu, D. C., Coler, M., Fletcher, G., and Pechenizkiy, M. (2019). Evolving plasticity for autonomous learning under changing environmental conditions. arXi preprint 1eXi::1904.00 01709.
Zenke, F., Agnes, J., and Gerstner, (2015). Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories spiking neural networks. Nature communications, 6(1):1-13.
11

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL METHODS
In this section, we give complete details of our experimental procedure. The accompanying code can found here: htttps://github. http/BBainnletwor//raaaaaaae
Next, we experimentally show that a rule learned on a specific data set generalizes to new data sets. We learn rules on simple data sets, such as data labelled by a linear threshold function, then use this rule to train simple RNN on more complex data-sets generated by ReLu networks, simple RNN's, MNIST and Fashion-MNIST.
Then, networks trained by plasticity rules are empirically shown to more robust than ones trained by GD. Furthermore, as depth increases, so does robusiness to adversarial attacks.
Finally, We also describe alternative, arguably more bio-plansible schemes for updating weights during training.
A.1 TRAINING AND TESTING PROCEDURE
Rule- based training. First suppose that already given output layer plasticity rule To : {0, 13" {0, and an RNN rule r: [0,1)² x [0, + R. We can now take any simple RNN with T rounds and any data X (2 ...,,,(()), and train using these rules. Of course, in the case that 1, there would no RNN rule.
1. In the case of additive updates, initialize the graph weights W and output layer weights U to zero. In the case multiplicative updates, initialize these to 1.
2. For each example 2,, perform the forward pass, and keep track of the firing sequence of the nodes.
3. Given the firing sequences of each node, update W according RNN rule and U, according to output layer rule ro- We scale down the magnitudes the rule updates by factor of y, the step size.
4. The final weights provide the trained simple RNN.
GD to Learn a Rule. We now want to learn rule specific to a particular data set. For this, we do the following.
1. For each epoch, randomly split the data into batches (we used size 100 or 1000).
2. For each batch, train a network using the current rule as described above.
3. Using the resulting network, compute the cross entropy loss this batch.
4. Compute the gradient of this loss with resp of choice ect to the parameters of the rules.
5. Update the rules according to the optimizer choice.
The experiments we have used the Adam optimizer, with regularization (with constant of 0.01).
A.2 GENERALIZATION EXPERIMENTS.
We used six different data sets: Halfspace data is labeled by a simple linear threshold function. ReLU1 and ReLU2 data are labeled by two different ReLU feedforward networks, each with single hidden layer of width 1000 and randomly initialized weights, and two output neurons, and the argmax of the two output neurons generated by a random simple RNN with 3, IV 100, 50, = 0.5. Each dataset has both training and testing data, each consisting of ten thousand examples. Lastly, we used the standard benchmark MNIST and Fashion- MNIST datasets, with their 28x28 pixel images vectorized to 784 dimensions, where selected ten thousand random images out of the sixty thousand in each of their training
taken to label each example. The simple RNN data was
We began with a simple RNN with [V] = 100, k = 50, p = 0.5. Using this network with == 1, we learned an output layer plasticity rule using GD on the Halfspace dataset.
Next, we used network of the same size with 7'= 3 and ReLU1 data to train, this time learning an RNN plasticity rule parameterized by single-hidden layer neural network, in addition to the output layer rule.
12

---

Under review as a conference paper at ICLR 2021
For each of the two models, created new network with larger graph, [V|= = 1000, k 500, 0.5. We did not learn new rules specific these particular graphs, but rather retained the previously learned rules.
Using each model's respective rule(s), we trained the models on the ten thousand training examples from each of the six data sets. Note that this training only consists of initializing the weights of the graph and output layers 0, and for each misclassified example, update the weights according to the rule, completely without using GD. We did this ten times for each data-set, with the order of examples randomly shuffled each time. We reported the average testing and training accuracy in the figure. In every experiment learning rate of n = 10-2 was used, corresponding to weighting the weight update proposed by the rule by factor of 17.
A.3 ROBUSTNESS EXPERIMENTS
We generated a simple RNN with [V] 1000 and cap of 500, and trained it separately with plasticity rules and with GD.
We performed two experiments with rule, for = and one for T= 3. In both cases, the same perceptron-style output rule was used. For T' = we utilized small two layer feedforward network to act as the rule. This had a hidden layer of size 20. train this auxiliary network, we used the method described earlier, however we did so on a smaller simple RNN with [V] = 200, and cap of 100.
Once trained each of the three networks, hundred random images chosen be adversarially perturbed. For given network and image, we generate nine adversarial images according to the following multi-step attack method previonsly described, one for each value of o # y:
11++s - V.L(Z,y'))
If any of the nine resulting perturbations become misclassified, then we count the original image mis- classified under perturbation. We perform this process for each 0.0, 0.5, 1, 1.5, 2.0, 2.5, 3.0, 4.0, allowing a perturbation up to a magnitude e in the la norm.
A.4 ALTERNATIVE SCHEMES
Updates each example. Instead of applying the rules only when we misclassify an example, a more biologically plausible updating scheme would to perform the rule updates for every example, regardless the current model's prediction.
Experimentally, this approach has had results very similar to those when updating only for mis- classified examples. For instance, Figure D is comparison of the accuracy curves MNIST when applying the same perceptron update rule all examples, and only misclassified examples.
Update misclassified examples (T=3)
90
80
75
Accuracy
70
65
55
Update on examples
Update on Misclassited Examples
50
0
5000
10000
15000
20000
25000
30000
Numbr Samples
Figure 6: Updating on all examples provides similar results as updating only on misclassified data for T 3, [V| 1000, At 500 simple RNN on MNIST.
Updates to all edges. Our output layer rule only updates edges which lead either to the node corresponding the true label of the example or the prediction. Instead, we could apply the rule
13

---

Under review as a conference paper at ICLR 2021
to all edges the first column of the rule indicating the updale to the edge leading to the correct label, and the second column indicating the update the remaining edges. Note that this only affects the multi-label case. For the MNIST data set, we have again had comparable results.
Using both schemes. Using both schemes described above (updating at each example, and updating all weights), we provide computationally efficient analytic solution to finding the optimal output rule (given all else is fixed) with respect to mean-squared error loss (MSE loss). See Section B
In the case of our binary classification data, the accuracy we achieve with this optimal rule comparable to that of our original model, which would update only on misclassified data.
However, when combining both schemes on MNIST data, we begin to see decline in accuracy. The usual perceptron rule which originally achieved 92% accuracy is now only reaching 88-89%. The optimal rule reached similar 89% accuracy.
Note that this "optimal" rule is only optimal with respect the MSE loss, which general is particularly well-suited for classification tasks. Additionally, this rule is not necessarily of the same sign pattern
B MATHEMATICAL PROOFS
The next theorem is inspired by Freund and Schapire 1999). Theorem 4. Let
(Im, Hmm) be traiviing data set in R with binary labels such that ||=||| S R, and D² = Ei ma, y:(w" .x) for some unit vector w". Suppose we sequentially apply an oulput layer rule of the form [-o, a.,,ab b; -b], with b 2 a O to any ezample whose label is incorrectly predicted or with y of the threshold. Then the number of incorrectly predicted labels is bounded by
o (0x
R² +2
72
Proof of Theorem 1 For analysis, we assume that we compute the loss after applying the update rule to a random example. For the cross entropy loss, we minimize
[(r, W) B2-D (- log (r, W,x) Ip(x) & {x))
Let fe(x) = if ((x) and (=(x) = 0 otherwise. Pc(x) is defined similarly for the prediction of æ. Since the rest of the network is fixed, we can view L and as functions of just the output layer weight matrix W, consisting of weight vectors we for each output class Now fe is the output neuron value for class c, ie., the result of softmax applied linear combination previous layer outputs. So we have,
(
fa(r, W2
where is the vector of penultimate layer outputs and we(() is the weight after the rule update, Le.,
we(r) v 17 Σ Vi
Σ
r(a, Pr(y/ p.pp blp(2) o {(x)).
(,,E(o,13
E With f(z) ((=)=e*/>>) 11,e,, , have
In fxx))
(In
Σ 2h) lne
02j
Ozj
e
x(i j).
Σ t
We then compute the gradient of with respect to r:
aL
Olog (r w.
BrnD
(p(x) # {(xz)
(r(a,b)
Ōr(a,b)
((wu.lr)
(w.(() (felr, x(e= x(c=((s)) (m(x) + ((s)
BrnD Σ
br(a,b)
14

---

Under review as a conference paper at ICLR 2021
In the case of two labels, gel:
Ōr(a,b) BEnD (fesssr w.
AL
(Ow -u Ör(a,b)
(wecs)(r) :) pp()) + d(x))
Or(a, b)
where ((r) is label opposite ((x). Note that
@w.(r) v Ōr(a,b) = L (xll a.pp(a) = (x(x)) # b))..
Therefore, 91 ov(a..)
is
BBZ-D (te)(r,Wa)) Σ vi (Pr(N a, = = bl (x(2) * b) - Pr(s a,pe(a) bl bc(())) * b1)
From this, we can get the sign of each entry the rule matrix. First, it is clear that the entries for first and second columns (corresponding to b = 0, 1, i.e., the updates to the "correct" and "incorrect" labels) have opposite sign. Next if the gradient for (0,b) is positive, then the gradient for (1,b) is negative, since entries in the second Then, since we use standard squared Euclidean norm regularizer, at optimality, the overall gradient a matrix with the above sign pattern plus the current rule matrix r. For this to be zero (at a point with zero gradient), the rule r and the gradient must have the opposite sign pattern. Let P(a, c) Pr(y!- = a, ((x) c). Then, since every v. used to update is misclassfied, each coefficient r(1, 0) and multiplier P(1.e) - P(1,c) must have the same sign, so we have r(1, 0) negative, then the term negative and the gradient with a = 0, = has positive sign and the rule has negative sign. The signs of the other entries follow similarly.
are negations of the first row minus a positive constant.
Proof of Theorem 3 The proof is inspired by the classical proof of the Perceptron algorithm. For data labeled by an unknown linear threshold function sign(w" .x) with margin y- we consider the invariant w w² /|]w[|2- Then on a misclassified example x whose true label is 1, the update
if 34
Wa
Wi
+ b
if = 1.
Therefore the numerator goes from w² w to
w"
Σ w + Σ wi
son =1
=w" w + alw" r) + (b
a) Σ wi.
Then, since x has label 1 we have Eix 1i + Ex wi 2 7. Also, by assumption, E, wz = Em w + Ema wi = O. Therefore, Σ Dix,= w 2 7/2. follows that the increase in w" w in iterations is at least tay.. On the other hand, consider the squared norm of the denominator. After one updated it goes from ||al|2
Σ =-
(w:
a)² + Σ
(w: + 6)²
sary=1
<||w||
2 1n 26
Wi
Σ ws
k=;=1 m1
C: -
=||w0|
+
2(b -a) a) Σ
wi + 20( - x)
<||w|| 2 2(b a)
Σ
Wv
fr,11
where the last step uses the fact that is misclassified and w- < 0. We thus bound
the increase [| in t iterations by Ct for some constant < 62n 2(b a) Now since
In0" - / < 1, we must have
td222 <0 1C
26
15

---

Under review as a conference paper at ICLR 2021
OPTIMAL OUTPUT LAYER RULE FOR MEAN SQUARED ERROR LOSS
derive an analytic solution to finding the optimal output layer rule given that fixed. By optimal, we mean the rule minimizing the mean-squared error loss of the model after training:
I(r) = 1 - t=1
(edr, - -
where we have data points (1)) (n)) and labels. Unlike previously, we do not apply final softmax to the output,
fc(r,x10) = y
where again y is the vector of penultimate layer outputs corresponding to x0 and w.(r) is the final weight vector corresponding to label
Previously, we have only updated the weights of the output layer if our prediction was incorrect. In this situation, we will instead updating the weights for every example. Doing so, the final output weights will be independent of the order of the data. Given that the initial weights are initialized at 0, the final weights can be explicitly described by
w.(r) = Σ r(a,b) a.bE(U.1)
Σ x"(v")) lo(x(()(b)
where x" (v) is the standard indicator function. That (x' (y)); = if y * 0 and (x' = otherwise. On the other hand, (x") 1 if y = 0 and 0 otherwise. Note that must be nonnegative since it is the result of a ReLu activation.
(*))
For instance, consider the r(1,0), and term contributing to w.(r):
r(1,0) (f
Σ (2(
Recall that r(1, 0) describes update of an edge (1,j) node fired, and node j is the output node corresponding to the true label. We have le(x) # whenever the true label of is equal to c. And, we are updating the weight (w.(r)), by r(1,0) whenever the jth node fires, as expected.
Now, we compute the gradient T with respect to r:
Or(a,b) n-1 1.EE(w (w.(r) 10) (((x(())) (w)(() ( Or(a,
Notice that for each the last term independent r, and evaluates to a real number:
(r). (0) Σ
())))
0r(a,b)
Finally, note that the remaining term we(r) (r).)))))))))) (()) ()) ( is linear combination of the entries in
plus some constant. Hence, so must be arta.b)
at To find the rule r minimizing the loss, we simply set the gradient to zero. Since each 3r(a. = is a linear equation in r, we have a simple x system linear equations. Its solution the optimal rule.
Furthermore, it is computationally efficient to determine the optimal rule, taking O(n d) time, where d is the dimension of the penultimate layer, y. This can be done by directly computing the 4 4 linear system described above. Solving the system afterwards simply takes constant time.
16

---

Under review as a conference paper at ICLR 2021
C EVOLUTION CAN SIMULATE (A VARIANT OF) GD
We have shown that plasticity rules can be computed though GD in RNNs, and learning is enhanced significantly as a result. On the other hand, plasticity animals evolues. Can we demonstrate analytically that, indeed, plasticity rules can also be learned through evolution? And is there a connection between these two paths on plasticity, namely evolution and GD? Could it be that evolution simulates GD in this c
Here we show, using the standard mathematical models of population genetics and evolution, that any real parameter such as each of the plasticity coefficients can be adapted by evolution by having such a parameter be the sum of many genetic contributions. This is rather common genetics for example, height in mammals seems to be effected additively by over 200 genes, hence the Gaussian nature of height distributions, see Signer-Hasler et al. (2012). Furthermore, we show that the evolution equations ultimately point to GD!
Consider model in which a haploid organism has genes ......1 each with two alleles {+c,- -c}, and suppose that a parameter Y the phenotype - for example, a coefficient of the plasticity rule is represented as the sum of these n values. To study the evolution of such organism, consider a sequence of generations indexed by f. where at each generation we denote by of the frequency of allele i in the population, and thus for each individual in the population the expectation of Y is Y = c. E,E, (2x, 1). At each generation, population is sampled from this distribution, and each individual's performance on the learning lask partly determines the individual's fitness - intuitively, its expected number of offspring. We assume that the contribution of this particular parameter to fitness is small - this is reasonable, as there are many other traits contributing to fitness, such as locomotion and digestion. This is known the weak selection regime of evolution Nagylaki (1993); Chastain et al.] (2014), and the population genetics equations of how the x; (the genetic make-up of the species) evolve are:
Z (2) 0. -L(0)
(A similar equation holds for the frequency the other allele, (1 (1-x,)--1.) z a normalizer to be defined soon, L(Y) is the expected loss of the test data when the parameter is Y, and 0 assumedd to be a very small positive number, is the amount by which aptitude in this learning task will enhance the individual's chance of surviving and procreating. That is, the frequency of the i-th gene changes by 0 times the difference between some reference expected loss, taken to be L(Y), and the expected loss when the i-th gene of parameter Y conditioned to be +e. The function of z-+1 is to keep the allele frequencies adding to one: zz+1 = + ((z;((()) L(Y(+e)) + (1 (LY) - L(M--))]-
Since 1 1 a O(a²), the above expression is within O(02) equal to: Ita
0 (LL((L( (()+ 2²) (L(Y) L(Y x.)]
Now notice that Ye, the expectation of conditioned on the value of the gene being +e, (Y -8-((1 e(1
-2x,)) + e. To see this, the parenthesis is the expectation of the remaining genes besides
gene i, and then is added to that; and similarly y = Y -2ex.
Fimally, we can approximate the difference (L(P) - L(Ý - <(1 - xt)) by @h ob x²) + O(E², 18/4)11 l), and similarly for the other difference, to finally obtain, by the chain rule and the fact that or 2e, Uı,
ØL du
8LL ayz
2 x
2x 0(6" +
Notice now that, ignoring the error term, which is by assumption small, this is GD on gene frequency 2,, with the extra factor 2 22,, factor which slows the GD at large values of x, and accelerates it at small values. Alternatively, this equation is precisely GD the new variable 2; 2r; af, the integral of the factor 2x; note that, appropriately for a variable change, the defining function of z; is strictly monotone for æ; in 1].
This result holds for the scenario in which each plasticity coefficient is represented by the additive contributions of many genes. What happens in the setting, less wasteful genetically, in which these genes shared between the plasticity coefficients? That is, let us assume that each coefficient is a random linear function of a random subeet of these coefficients. That situation is much harder to analyze and compare to GD, but it does work as an effective evolutionary mechanism, see Gorantla et al. (2019), Theorem 1.
Reeaal that Geoff Hinton opined in his Turing award lecture get gradients."
Hinton 2019 that "evolution can't
17


Reviews:
[{"review_id": "yYQGNHh7zp9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to encapsulate the update rule for a neural net into a look-up table specifying weight changes for each combination of \"pre-synaptic\" input to the weight, and \"post-synaptic\" activation of the unit receiving that incident connection. They learn the elements of this matrix by gradient descent, and then use that learned update rule to train neural nets on a new task. This is motivated by a separation of timescales biologically, wherein learning rules might be evolved over long timescales, and then act within each brain over shorter ones.\n\nThere is a nice discussion of related previous work, but it misses a few key items that, to me, diminish somewhat the novelty of this work. That's okay: being first isn't everything. But I think it is important to point out to readers what is new and better about this work vs previous work.\n\na) The auto ML zero paper from Quoc Le et al. (arXiv 2003.03384). They learn both architectures and learning rules via simulated evolution\n\nb) Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., ... & De Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems (pp. 3981-3989). They use GD to learn plasticity rules.\n\nc) A few recent papers on bio-plausible backprop-type algorithms.\ni)  Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Biorxiv: https://doi.org/10.1101/2020.03.30.015511 from Naud and colleagues\nii) Guerguiev, Jordan, Timothy P. Lillicrap, and Blake A. Richards. \"Towards deep learning with segregated dendrites.\" ELife 6 (2017): e22901.\niii) Sacramento, João, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. \"Dendritic cortical microcircuits approximate the backpropagation algorithm.\" In Advances in neural information processing systems, pp. 8721-8732. 2018.\n\nAside from the relation to prior work, I have a few technical and conceptual questions / comments:\n\n1) Fig. 2: were all three nets given the same initialization? That could matter for comparing the training curves of accuracy vs. training time because a good initialization could give one learning rule an apparent advantage. And given the accuracy at t=0, it doesn't look like they are the same.\n\n2) I like that the authors studied generalization of the learned rule between tasks: that is important (although, SGD also generalizes well). I'm a bit less impressed by the performance obtained in the MNIST and fashion MNIST tasks. At the same time, using two-factor rules (update is a function just of pre- and post-synaptic inputs) to solve MNIST sounds hard given that there's no credit assignment signal. I think that the authors would be well served to read up on the papers on bioplausible deep learning, and consider variants of this work that include a credit assignment signal.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qDhrqfop0fo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis is a fascinating paper which push the bounds of artificial neural network knowledge and understanding, while questioning the typical approach of considering various neural networks branches in isolation. Concretely, it shows that by using concepts from neuroevolution together with deep learning concepts, we can learn how to learn learning rules (i.e. plasticity rules) which can generalize across tasks and can train neural networks which are more robust to adversarial attacks than typical networks trained with stochastic gradient descent. \n\nStrong points:\n\n•\tIn my opinion, the paper is visionary. It answers few questions, while opening the path for a large number of new research directions and new unanswered questions.\n\n•\tThe paper has a very well-balanced content of novelty, math, computer science, neuroscience, and even philosophy.\n\n•\tThe paper is very well written and anchored in a multidisciplinary literature. It has the potential of becoming a “must read” paper in the future.\n\n\nWeak points:\n\n•\tThe datasets used, including MNIST and Fashion-MNIST are rather simple. It would be very interesting to see how the approach behaves on more complex datasets.\n\n\nDuring the discussion phase, I would recommend to the authors to address the following comments:\n\n1. In the limit of time, try to perform experiments also on CIFAR 10/100. I believe that it would be interesting to see on CIFAR 100, the behavior of three types of learned plasticity rules: (1) plasticity rules learned on the simple datasets, (2) plasticity rules learned on CIFAR 10, and (3) plasticity rules learned on a subset of the CIFAR 100 training set.\n\n2. Are you encountering problems with ReLU activation in recurrent networks such as exploding or vanishing weights? Does your approach work also with hyperbolic tangent?\n\n3. I believe that it would help the paper clarity if you can add a table towards the end of the paper to summarise the main results in terms of accuracy, training time, etc.\n\n4. Perform a proof-read of the whole paper to improve the English usage and the presentation. For instance: typos (e.g. “rule, The next theorem”), unit measures for axis labels (e.g. figure 5 – accuracy [%]), etc.\n \n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lf2BL0qd9c9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper uses meta-learning to search for novel, local learning rules in artificial neural networks. This is done by parameterizing local learning rules for feedfoward and recurrent neural networks, then using mini-batch gradient descent to update the parameters of the learning rule.  The authors argue that this is a promising strategy for discovering the learning rules used by biological systems, with three main contributions: (1) they provide proofs that this approach does what we would hope it would do when applied to a single linear layer; (2) experiments demonstrate meta-learning for simple non-linear and recursive architechtures; and (3) the authors provide an argument that evolution could replace gradient descent as the method of searching over possible learning rules. The authors also show through experiments that models trained with these non-gradient methods are more robust to (gradient-based) adversarial attacks. \n\nOverall, I thought this paper was well-written and provided interesting arguments and proofs. However, the experiments are not enough to support the main claims, so I think this paper is borderline. \n\nPros: \n\nWhile the main idea of using meta-learning to search for new bio-realistic learning algorithms is not new, the particular formulation used here with recursive neural networks was new as far as I know, and I found this idea very interesting. Most of the work in this area has been focused on feed-forward networks, but as the authors emphasize, recurrent neural networks add a whole new dimension to the space of biologically-plausible, local learning algorithms. An example of such an algorithm is Recirculation, described in Hinton and McClelland 1987, \"Learning Representations by Recirculation\", which is closely related to Feedback Alignment (Baldi and Sadowski 2018, \"Learning in the machine: Recirculation is random backpropagation\"). \n\nCons:\n\nMy main criticism is that the experiments are not enough to show that the \"discovered\" learning rule does anything useful in the RNN. As the authors admit, one can achieve good performance in a multi-layer NN by fixing the random weights in the hidden layers and by training only the output layer. I would have really liked to see how the meta-learned algorithm compared to fixed, random representations. The experimental performance of the meta-learned algorithm on MNIST is quite poor (~80% test accuracy, Figure 3), so its unclear what is going on. \n\nThe results of the adversarial robustness experiments are not surprising. Adding additive or multiplicative noise during training will also make the trained networks more robust. It think these experiments actually distract from the main ideas of the paper. It would have been better to more carefully explore whether the learning algorithm can learn more difficult functions.\n\nSection 2 describes two possible alternatives for the plasticity rules: one that incorporates information about the error, and one that does not. I think it is important to highlight the fact that the latter is an unsupervised plasticity rule --- the meta-learning algorithm has access to the target output, but the local plasticity rule does not. So while a plasticity rule trained on Dataset 1 does have some information about the Dataset 1 targets (by way of the meta-learning updates), when it is applied to Dataset 2 it never receives any information about Dataset 2's target, and is thus unsupervised. This is an important distinction between the two approaches.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9M7q1YXVJ2-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method for meta-training plasticity rules, allowing networks to learn new instances of a given domain quickly and efficiently. The method consists in implementing plasticity as an arbitrary function of the past few timesteps of local activity at a synapse (input and output). Experiments show that the method finds reasonably successful rules, that these rules generalize across (some) domains, and that the learning seems more robust to (some) adversarial attacks than plain gradient descent.\n\nWhile meta-training plasticity rules is not new, I believe the method is novel and quite interesting. I also appreciate the experiments to demonstrate cross-domain generalization and robustness to adversarial attacks.\n\nA possible caveat is that the experiments, though diverse, are still a bit limited. The paper only uses relatively small feedforward networks and recurrent networks with only 3 timesteps. In terms of \"real\" datasets, only MNIST and Fashion-MNIST are considered. Similarly, the trained plasticity rule seems robust to a certain type of adversarial attack, but is it more robust to other forms of distortion - such as plain noise, deformations, etc. ? I suppose this is tolerable for an introductory paper.\n\nMinor comments: \n\n- Bengio  et al. 1992. “On the Optimization of a Synaptic Learning Rule.”, used gradient descent to meta-learn plasticity rules and should be included in the Related Work section. \n\n- Similarly, Metz et al. 2019: https://openreview.net/forum?id=HkNDsiC9KQ. \n\n- Sometimes, in the figures, it is not clear what exactly is shown. E.g. Which type of plasticity rule is used (network-based or lookup tables)? Also in Figure 4, how exactly are these image generated - what's the criterion for a sufficient error?\n\nSpeculative comment: IIUC, the networks in this paper are binary and can be seen as spiking networks.  Interestingly, the space of rules explored by this method seems to include standard models of biological plasticity, such as spike-timing dependent plasticity, as well as more complex \"triplet\" rules (see e.g. https://www.pnas.org/content/108/48/19383). There's probably some interesting work to be done in this direction (for future work of course, not for this introductory paper).\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores meta-learning of local plasticity rules for ANNs. The authors demonstrate that they can meta-learn purely local learning rules that can generalize from one dataset to another (though with fairly low performance, it should be noted), and they provide some data suggesting that these rules lead to more robustness to adversarial images. The reviews were mixed, but some of the reviewers were very positive about it. Specifically, there are the following nice aspects of this work:

A) The meta-learning scheme has interesting potential for capturing/learning biological plasticity rules, since it operates on binary sequences, which appears to be a novel approach that could help to explain things like STDP rules.

B) It is encouraging to see that the learning rules can generalise to new tasks, even if the performance isn't great.

C) The authors provide some interesting analytical results on convergence of the rules for the output layer.

However, the paper suffers from some significant issues:

1) The authors do not adequately evaluate the learned rules. Specifically:

- The comparison to GD in Fig. 2 is not providing an accurate reflection of GD learning capabilities, since a simple delta rule applied directly to pixels can achieve better than 90% accuracy on MNIST. Thus, the claim that the learned rules are "competitive with GD" is clearly false.

- The authors do not compare to any unsupervised learning rules, despite the fact that the recurrent rules are not receiving information about the labels, and are thus really a form of unsupervised learning.

- There are almost no results regarding the nature of the recurrent rules that are learned, either experimental or analytical. Given positive point (A) above, this is particularly unfortunate and misses a potential key insight for the paper.

2) The authors do not situate their work adequately within the meta-learning for biologically plausible rules field. There are no experimental comparisons to any other meta-learning approaches herein. Moreover, they do not compare to any known biological rules, nor papers that attempt to meta-learn them. Specifically, several papers have come out in recent years that should be compared to here:

https://proceedings.neurips.cc/paper/2020/file/f291e10ec3263bd7724556d62e70e25d-Paper.pdf https://www.biorxiv.org/content/10.1101/2019.12.30.891184v1.full.pdf https://proceedings.neurips.cc/paper/2020/file/bdbd5ebfde4934142c8a88e7a3796cd5-Paper.pdf https://openreview.net/pdf?id=HJlKNmFIUB https://proceedings.neurips.cc/paper/2020/file/ee23e7ad9b473ad072d57aaa9b2a5222-Paper.pdf

And, the authors should consider examining the rules that are learned and how they compare to biological rules (e.g. forms of STDP), if indeed biological insights are the primary goal.

3) The paper needs to provide better motivation and analyses for the robustness results. Why explore robustness? What is the hypothesis about why these meta-learned rules may provide better robustness? There is little motivation provided. Also, the authors provide very little insight into why you achieved better robustness and insufficient experimental details for readers to even infer this. This section requires far more work to provide any kind of meaningful insight to a reader. What was the nature of the representations learned? How are they different from GD learned representations? Was it related to the ideas in Theorem 4? Note: Theorem 4 is interesting, but only applies to a specific form of output rule.

4) In general, the motivations and clarity of the paper need a lot of work. What are the authors hoping to achieve? Biological insights? Then do some analyses and comparisons to biology. More robust and generalisable ML? Then do more rigorous evaluations of performance and comparisons to other ML techniques. Some combination of both? Then make the mixed target much clearer.

5) The authors need to tidy up the paper substantially, and do better at connecting the theorems to the rest of the paper, particularly for the last 2 theorems in the appendix. Also, note, Theorems 2 & 4 appear to have no proofs.

Given the above considerations, the AC does not feel that this paper is ready for publication. This decision was reached after some discussion with the reviewers. But, the AC and the reviewers want to encourage the authors to take these comments on board to improve their paper for future submissions, as the paper is not without merit.
Decision: Reject