Do What Nature Did To Us: Evolving Plastic Recurrent Neural Networks For Generalized Tasks
Authors: Fan Wang
Hao Tian
Haoyi Xiong
hua wu
Yang Cao
Yu Kang
Haifeng Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Evolving Plasticity
Learning to learn

Abstract: While artificial neural networks (ANNs) have been widely adopted in machine learning, researchers are increasingly obsessed by the gaps between ANNs and natural neural networks (NNNs). In this paper, we propose a framework named as Evolutionary Plastic Recurrent Neural Networks (EPRNN). Inspired by NNN, EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based Learning all in one meta learning framework for generalization to different tasks. More specifically, EPRNN incorporates with nested loops for meta learning --- an outer loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to specific tasks. In the inner loop of EPRNN, we effectively attain both long term memory and short term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for the formation of memories in NNNs.  The inner-loop setting closely simulate that of NNNs, which neither query from any gradient oracle for optimization nor require the exact forms of learning objectives. To evaluate the performance of EPRNN, we carry out extensive experiments in two groups of tasks:  Sequence Predicting, and Wheeled Robot Navigating. The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning based approaches in the tasks. The experiment results suggest the potential of EPRNN to generalize to variety of tasks and encourage more efforts in plasticity and recursion based learning mechanisms.

Under review as a conference paper at ICLR 2022
Do WHAT NATURE DID To Us: EVOLVING PLASTIC RECURRENT NEURAL NETWORKS FOR TASK GENER- ALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
While artificial neural networks (ANNs) have been widely adopted in machine learning, the gaps between ANNs and biological neural networks (BNNs) are receiving increasing concern. In this paper, we propose framework named as Evolutionary Plastic Recurrent Neural Networks (EPRNN). Inspired by BNN, EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based Learning in one meta-learning framework for generalization to different tasks. an outer
More specifically, EPRNN incorporates nested loops for meta- meta-learning loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to specific tasks. In the inner loop of EPRNN, we effectively attain both long-term and short-term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for memristance in BNNs. The inner-lopp setting closely simulates BNNs, which optimization
neither gradient-basse objectives. To evaluate the performance of EPRNN, we carry out extensive experi- ments in two groups of tasks: Sequence Predicting, and Wheeled Robot Navigating. The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-an based on plasticity and recursion while yielding comparably good performance against deep learning-baat -basid approaches in the tasks. The experiment results suggest the potential of EPRNN to generalize variety of tasks and encourage more efforts in plasticity and coursion-based learning mechanisms.
require the exact forms of learning
INTRODUCTION
ANNs have achieved great success in handling machine learning tasks. Despite being initially inspired by Biological Neural Networks (BNNs), there apparent gaps between ANNs and BNNs. Mainstream ANNs use gradient- based optimizers to minimize learning objectives. Shreds of evidence show that BNNs learn through plasticity (Gerstner al., 1993) without explicit learning objectives, among which Hebb's 's rule (Hebb) 1949) is most well known. Though gradient descent methods are the most efficient optimizers for ANNs, their side effects are also noticed, including the problems of catastrophic forgetting, over-consumption of data, and the requirement for manual efforts in designing objective functions. Those challenges are becoming an essential impedance to the further development of machine intelligence.
Recent studies show the learning mechanisms of BNNs, such as plasticity (Soltoggio al. 2008
Najarro & Risi 2020) and model-based learning (Santoro et al. 2016) Mishra et al. 2018). under appropriate meta-parameter optimization, can effective alternative for task generalization in ANNs. Unlike gradient- based methods, these mechanisms simulate the learning behaviors of BNNs and don't require any explicit-form learning objectives. More recently, authors in (Miconi et al. 2019) proposed a plastic recurrent neural network for lifelong learning of ANNs, where implements Hebbian plasticity with differentiable objectives and gradient- based optimization. Though the above studies have investigated learning of ANNs using the two mechanisms derived from BNNs with gradient-based methods (Miconi et al. 2019) optionally, in this work, we aim at further verify the path of discovering those rules evolutionarily, simulating that of BNNs.

---

Under review as a conference paper at ICLR 2022
Backgrounds. Though learning in BNNs has not been fully understood, some of the learning mechanisms and rules, such as plasticity (Gerstner et al. 1993) and recursion (Pollen 2003). have been observed in brains and adopted by ANNs. Typically, Model-based learning employs recurrent neural networks (RNN), LSTM (Hochreiter & Schmidhuber 1997). and self-attention (Mishra et al.
2018 Chen al., 202T) layers learners. Learning is based memories within the feed-forward pass. The information is updated in the hidden states instead of the parameters. Model-basch learners found be sample efficient in generalized supervised tasks (Santoro al. (016), zero-shot generalization in language (Brown et al. 2020), and reinforcement learning (Mishra et al. 2018 Chen al. 202T) when compared with various type of gradient descent methods. So far, among model-based learners, though self-attention-basee learners such as Transformers have state-of-the-art performance, the ( (where short sequences. On the other hand, recurrent learners such as RNN and LSTM have O inference costs but suffer from poor asymptotic performances. That is, when sequences are getting longer, performances longer improve even deteriorate. It partly due to the limitation of the memory spaces. For instance, an recurrent neural network of hidden size n has a memory of (n)). In contrast, its parameters scale with (n?)), making parameter-updatie more powerful learning mechanisms than recursion-only.
the sequence length) makes them only available to relatively
XX
Inner-Loop Learning Mechanism
Decision & Action
300
DDC
D
* 300
200
DDC
Genomes
Observation & Feedback
D00
DeC
Doc
D000
Fitness
Natural Evolution (Outer Loop)
Individual Lifetime (Inner Loop)
Figure 1: An illustration of the natural evolution: The evolution takes place in the outer loop, where the genomes are mutated and selected, and the population either thrive or become extinct based on the Fitness. Lifetime of each individual composes the inner loop. At the beginning, the genomes decide the learning mechanisms and initial neural configurations of the brain in the born life. As the neural networks interact with the environment through actions and observations, its connections and hidden neuron states are further updated to better adapt the environment. Plasticity are believed to be important part of the learning mechanisms. The fitness depends on the learning and adapting capability of each individual.
In addition to recursion- learning, evolving plasticity (Soltoggio al., 2008 2018 Lindsey & Litwin-Kumar 2020 Yaman et al. 2021) has been proposed to reproduce the natural evolution and plasticity in simulation, as shown in Figure Implementing plasticity not straightforward: unlike gradient-basse learning methods, plastic rules are not universal but have to be optimized beforehand, which is not possible without a further outer-loop optimizer over the inner-loop learning. Evolutionary algorithms (Zhang et al.) [011]] Salimans et al. (017a) are typically applied in the outer loop to search for meta-parameed carried by genomes during evolution. Those optimized plasticity rules are then applied in the inner loop to further tune NN's parameter for better adaptions to the environment. Another line works tries to bring gradient-based learning algorithms plasticity rule optimization (Miconi al. 2018 2019). found that evolution can more efficient in cases of long-horize in reinforcement learning (Salimans al. 2017b Stanley 2019).
shaping
learning rules, which can regarded as information
Miconi et al. 2018 2019) that improve recursive neural networks using plastic rules for capacity of learning, we propose
Our Works. Inspired by the previous works (Cabessa & Siegelmann, 2014
2

---

Under review as a conference paper at ICLR 2022
Methods
Inner Loop
Outer Loop
Memory Augmented NN. (Santoro et al. 2016)
Recursion
Gradient
MAML, (Finn al.,
2017)
Gradient
Gradient
Conditional Neural Processes,
Garnelo et al. 2018
Average
Gradient
SNAIL, (Mishra et al.
2018)
Attention
Gradient
ES-MAML. (Song et al.)
2019)
Gradient, Hill Climbing
Evolution
EPMLP, (Najarro & Risi
20200
Plasticity
Evolution
Differential Platicity, (Miconi et al. 2018
Plasticity & Recursion
Gradient
Backpropamine, (Miconi et al. 2019)
Plasticity & Recursion
Gradient
EPRNN (ours)
Plasticity & Recursion
Evolution
Table 1: A brief review of meta-learning methods
a novel meta-learning framework namely Evolutionary Plastic Recurrent Neural Networks (EPRNN) for task generalization. Specifically, this work makes contributions follows.
We study the potential of learning plasticity and recursion rules through the natural evolution in task generation. We show that recursion and plasticity-based rules can surpass gradient- based methods as inner-loop learners.
We present investigations and analyses on the learned rules and parameters, showing that the learning framework discovers plasticity rules that effectively update the connection weights according to the learning tasks. The differences between the transformation of hidden states and parameters are also shown, verifying the efficacy of combining recursion with plasticity.
The most relevant works
study are (Miconi et al. 2018 2019 Lindsey & Litwin-Kumar 2020
Yaman et al. 202T). Compared to (Miconi et al. 2018) that leverage gradient oracles to efficiently search plastic rules, the proposed EPRNN could work well in gradient-free settings. Compared to (Lindsey & Litwin- Litwin-Kumar [0020] Yaman et al.] (202T) that use evolutionary strategies to learn plastic rules, EPRNN also incorporates an RNN-based inner loop for couusionnbbsedd learning. The work (Miconi et al. 2019) also uses recursion (i.e., RNN) and differentiable plasticity in nested loops train self-modifying neural networks, EPRNN replaces the outer loop with evolutionary strategies to generalize tasks with non-differentiable objectives. Though EPRNN is not as competitive as gradient-bast based methods, which can optimize advanced neural networks with large datasets, our work still demonstrates the potential of using plasticity and recursion for meta-learning through natural evolution.
2 RELATED WORKS
Meta-Learning. Meta-learning aims at building learning machines that gain experience using task- specific data over the distribution of tasks. Inspired by human and animal brains that are born with both embedded skills and the capability of acquiring new skills, meta-learning implements two nested learning loops: The outer learning loops optimize the meta- peramearssesers that typically involves initial
parameters (Finn al. 2017 Song al. 2019). learning rules (Zoph & Le 2017 Najarro & Risi
2020: Pedersen & Risi 2021 model structures (Soltoggio et al. 2008: Li & Malik| (0166) and all
of three (Real al. (020) over distribution of tasks; The inner learning loops adapt the model to specific tasks by utilizing those meta- parammtar.ssess.... According to different inner-loop optimizers, we roughly classify the methods into model-base and
methods. The model-based methods do not update the parameters in the inner-loop, where only hidden states is updated (e.g.. recursion); The purameter-updaiing methods re-modify the connection weights in the inner-loop (c.g., gradient descent (MAML), plasticity). From this point of view, method can be classified into both groups. A brief review of the typical meta-learning paradigms is presented in Table [
Plasticity-based Learning. The proposal of the learning mechanism of BNNs is initially raised by Hebb's rule (Hebb) 1949). the most prominent part of which is "neurons fire together wire together". It is further polished by Spike Time-Dependee that the signal of learning is dependent on the temporal patterns of the presynaptic spike and post- synaptic spike. Learning could also appear in inhibitory connections, also known as anti-Hebbian
Plasticity (STDP) (Gerstner al. 1993) indicating
3

---

Under review as a conference paper at ICLR 2022
(Barlow 1989). Also, relationships between STDP and memory are investigated Barrann
& Seeramo-Gotarredooe 2009). Since many of those rules are related to spiking neural networks
(Ghosh-Dastidar & Adchi 2009), apply them to ANNs, simplified rules are proposed (Soltoggno et al. 2008) instead: given the pre-synaptic neuron state X and post-synaptic neuron state Y, the connections between X and Y are updated by
8w m|A C
(1)
m is the output from neuron modulators that adjust the learning rates of plasticity. Most of the existing rules are sub-classes of Equation| [ For instance, some works neglect the neural modulator
m (Najarro & Risi 2020; Miconi et al. 2018), others have set B, C, and D to 0 (Miconi et al. 2018 2019). The learned rule A, B, C, D will inevitably be dependent on initial parameter of W however, learning plastic rules that is not dependent the initial parameters was also investigated (Najarro & Risi 2020 Yaman et al. 2021).
ALGORITTMS
Problem Settings. We consider an agent (learner) that dependent on meta-parameter 0. It has the capability of adapting itself to a distribution of tasks T3 E T by interacting with the environment Tj through observation 1 and action a,. In K-shot learning, the agent is allowed to first observe samples of length K (this stage can be referred as mela-traiinnng-traiinngg see BBaaulieu et al. 2)200), then its fitness is calculated in mela--ruuningg-estiee tasks (GSL), the observations typically include features (I²) and labels (y:) in meta weeta-trainnng-ee traainnng stage (i+ = (It. 3/1 }), and the labels are left out for predicting in the senet-traiinng-tettitee (Santoro et al., 2016, Garnelo et al. 2018). In Generalized Reinforcement Learning tasks (GRL), the observations typically include states (st), actions (o₂-1) 1), and feedbacks (rt-1) (6. = {st, Pt-1)- sometimes r=-1 1 can not be observed) (Mishra et al.) 2018). The goal of meta-training is to optimize O such that the agent achieves higher fitness in mela-training- learned parameters are given weeta-esstnng-traiiing and pella--esting-estiine in order, the performances in meta-testing- -estting are evaluated.
rollouts. In Generalized Supervised Learning
stage
In meta-testing, similarly, the
Plastic Recurrent Neural Networks (PRNN). Given sequence of observations é ... it, .... we first consider an recurrent neural network (RNN) that propagates forward and yields sequence of outputs a, following:
h = oW, br + W4 b),
= f(h++1)
where he is the hidden states at step f. In PRNN, we kept W; stationary, but we set We to be plastic, so that we add subseript / to mark different W. at different steps. Regarding ht as pre-synaptic neuron states, and h4+1 as post-synaptic neuron states, by applying Equation I we update W, with:
W1+1 = W2- + SWe
(4)
dW;- WA O (he-1 hTT) + WB 3 (m, h2)) + Wo O (het1 12) + WD m
(5)
hx41 = 17 ② he+1,
(6)
where we use C and to represent "element-wise multiplication" and "matrix multiplication" respec- tively. h and 1 are column vectors. WA, WB, We. WD are collection of plastic rules of A, B, C. D from Equation [] which has the same shape as W,. m, is the neural modulators that adjusts the learning rates of plasticity. We calculate m, by applying neuron modulating layer denoted with:
m = (W((m)
h4 + w((m) i4 - ((m)
sketch of PRNN is presented in Figure 2 The main difference between PRNN and naive RNN is that PRNN updates both the hidden states and the connection weights during the forward pass.
Evolving PRNN Given task T3 E T,, by continuously applying Equation 2 training and wwwt--raiinnnettiiii learning rules, and the sampled task T, which is denoted as:
to 7 over meta- traiiinn
the fitness is eventually dependent on the initial parameters,
FO T) = Fitness i +1, 41+1, ik +2, ak aK+2....,
(8)
W,, Wo, WA WA, Wc, WD. w(()) w(()) b, (m)) E
(9)
4

---

Under review as a conference paper at ICLR 2022
Neuron Modulator (m₂)
Plasticity rule: WA, Wa, Wo WD
h
ht
ht+1
ht+2
W
W
W
Observations (te)
Figure 2: A sketch of the information flow in plastic recurrent neural networks (PRNN). The red connections are plastic, the black connections are static.
Following Evolution Strategies (ES) (Salimans et al. 2017a). in kth outer-loop iteration, we sample different tasks from T, and meta-parameters 84,; (i E [1, nl, from the neighbourhoods of 6g. We evaluate the fitness of sampled meta-parameters, and update the meta-parameters by applying:
B = 0g - a- Fir(Op. (e) - 0%)
(10)
Why Recurrent Neural Networks? As stated in Equation | plasticity in rewd-forward-onl NNs allows NNs to gain experiences from single-frame frame observation only. In cases of non-sequential GSL, the plasticity has chances to tune the connection weights the specific task by relying on observing one single frame of data (i, {It, V/ }). since its information of the feature and the supervision is complete. However, in general cases, learning can effective without summarizing sequences of observations. For instances, a human driver getting used to new car through continuously interacting and observing. Moreover, in GRL, there are time lag between the observation states and feed- backsback. Recursion helps to summarize historical observations to give the correct update for the connection weights.
Although, compared with naive RNN, there obviously bunches of more sophisticated neural structure such as GRU and LSTM, we believe it is more desirable to start from simplest recurrent structure study the potential of combining recursion and plasticity.
10
(a)
(b)
(c)
Figure 3: Demonstration of the tasks.(a) Two tasks sampled from Sequence Predicting (221,K=5, the red lines training sets and the green lines testing sets. (b) One task sampled from Sequence Predicting (2=3,, (c) A trajectory generated by an agent in Wheeled Robot Navigating task.
5

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
4.1 TASKS FOR EVALUATION
In generalized tasks, we have each task T3 dependent on some configuration parameters that are hidden from the agent. Below we introduce two groups of generalized tasks that we experiment on.
Sequence Predicting (Generalized Supervised Learning tasks). We randomly generate sequences of vectors (() = (y1 (1). ((()), where y: (1) = A,sin( 2 + Φ;), = 1,2,3,.. and A, ~ 4411 (1),, n;
71 ~ ((10, 100), Φ; ~ 4(00 2T), and a[ b) represents the uniform distribution between a and b. A, and ø are hidden from the agent. The front part of the sequence y11), ((), .... y (K) is exposed to the agent, and the left part y K 1), y(K + 2).... y(N) is to be predicted. The fitness the opposite of mean square error (MSH) between the predicted sequence and the ground truth.
We test the methods for comparison in four groups of tasks including (1 = -, 1,K = 10, N = 20),
1, 25, N 50), (L 3, K 10, N 20), and (l 3, K 25, N 50) (sce
Figure B (b))...
Wheeled Robot Navigating (Generalized Reinforcement Learning tasks). The agent is to navigate two-wheeled robot to randomly generated goal in 2-D space g (9x, We assume that there is a signal transmitter on the goal and a receiver on the robot. The robot observes the signal intensity decided by A, Ao k log (de/da) + (inspired by the attenuation for electromagnetic wave, see Friis (1946)), where de the current distance between the robot and the goal, (N o) the white noise in the observation. Ius Py: Ag, & are environment related configurations that is hidden from the agent. For each task, we sample configurations by Jv. In ~ ~ 0.5), Ao ~ (0,0, and k ~ (0.11, 0.5). The action is the rotation speed of its two wheels that controls the orientation and velocity of the robot. The reward at each step is f₁ = -d,, an episode terminates when the robot approaches the goal or steps reaches the maximum of 100. We also hide its own position and orientation from the agent, such that the agent relies on recording its own action the signal strength A, to locate itself. We investigate three types of navigating circumstances with different level of noises in the observed signal: Low Noise (a = 0.01), Median Noise (a = 0.05), and High Noise (a 0.2) (see Figure c)).
4.2 EXPERIMENT SETTINGS
We add the following methods into comparison, the methods share exactly the same outer loop and differ the inner loop.
ES-MAML (Song et al. 2019 We use four gradient descent steps in the inner loop, the learning rate of each step treated as meta-parameted which is to be optimized by the outer loop. As MAML can not utilize instant observation in zero-shot case, we show results of both ES-MAML (zero-shot) and ES-MAMI. (one-rollout) in Wheeled Robot Navigating. Except for ES-MAMI. the other methods are measured with zero-shot meta-testing score only in Wheeled Robot Navigating.
ES-RNN: Vanilla RNN as the inner loop learner.
ES-LSTM: Long Short Term Memory (Hochreiter & Schmidhuber, 1997) as the inner loop learner.
EPMLP (Soltoggio al. 2018): Multi- Perceptrons (MLP) with plasticity rules implemented.
EPMLP (Random) Najarro & Risi 2020
The main difference with EPMLP is randomly
setting the parameters of the plastic layers at the beginning of each inner loop instead of using fixed trainable initial parameters.
EPRNN (w/o m): Removing neuron modulator (m²) from the plasticity rule in PRNN.
We add additional noa-ptastic fully connected layers before and after the plastic layers to increase the representation capability of the models. For faimess, kept those layers identical for all compared methods. For every 100 outer loops in meta training. we add an extra meta testing epoch, evaluating the average fitness of current meta-parameters testing tasks. Each run includes 15000 outer

---

Under review as a conference paper at ICLR 2022
loops and 150 meta-testing epochs. Each result is concluded from independent 3 runs. Our code relies on PARI parallelization. We leave the detailed illustration of the model structures and hyper-parameters in the Appendices.
4.3 RESULTS
We present the experiment results in Figure (In Table and Table 3 also list the summarized performance by averaging the Top-3 meta-testing scores in the latest 10 meta-testing epochs of each run over 3 independent runs). Generally, we can conclude that PRNN performs substantially better when compared with naive RNN. In some cases, it even produces better results compared with LSTM, despite the simpler model architecture. is also interesting notice that the gap between RNN and PRNN are smaller in shorter sequences or low-noise environments, but larger more challenging tasks with longer sequence or higher noise (In Wheeled Robot Navigating tasks, higher noise pushes the agent to maintain longer memory in order to filter the noise and figure out the way goal). This phenomenon reaffirms the lack of long-tore Long-term memories in RNN, and shows that PRNN significantly improves this drawback. Comparing EPRNN (w/o m) with RNN and EPRNN clearly demonstrates that simple ABCD rule (without the neural modulator) may also work to some extent, but introducing the neuron modulator my can further benefit the learner.
Table 2: Summarized performances comparison in Sequence Predicting tasks
Methods
1=,,K=10,
|=1,K=25,N=50
1=3,K=10,N=20
1=3,K=25,N=50
ES-RNN
-0.385 + 0.060
-1.228 + 0.191
-1.273 = 0.009
-1.811 + 0.015
ES-LSTM
-0.165 0.014
- 0.013
-1.229 = 0.010
-1.475 0.016
ES-MAML
0.299
-1.747 0.064
-1.218 = 0.013
-1.796 = 0.013
EPMLP
-0.732 + 0.031
-1.185 0.045
-1.339 + 0.004
-1.735 0.013
EPMLP-
-1.233 + 0.115
-1.319 + 0.035
-1.489 = 0.023
-1.788 + 0.003
(Random)
EPRNN
0.11440.012
00.20810.025
-1.107
-1.430+0.019
EPRNN (w/o m)
0.135 = 0.026
-0.351 0.058
-1.22 = 0.014
-1.520 0.041
Table 3: Summarized performances comparison in Wheeled Robot Navigating tasks
Low Noise
Median Noise (a = 0.05)
High Noise (a = 0.2)
Methods
(a = 0.01)
ES-RNN
16.90 + 1.30
- 118.99 + 0.14
-31.23 +: 3.95
ES-LSTM
14.04 0.08
-1.50 + 0.58
-22.53 + 0.45
ES-MAML (zero-shot)
37.1880..
-37.91 + 0.80
-37.90 = 0.05
ES-MAML (1 rollout)
-23.10 0.46
-29.21 + 0.46
EPMLP
-21.24 + 6.08
-16.65 + 0.39
-23.75 2.02
EPMLP (Random)
13.02 0.19
1.225
28.85
EPRNN
-12.71 0.75
- -15.07 + 0.03
-23.79 0.55
EPRNN (w/o m)
14.93 = 0.49
-16.72 + 0.33
-24.11 + 0.34
Among plasticity based methods, show that recursion is advantageous than MLP in evaluated tasks. It is also worth noticing that EPMLP and EPMLP (Random) perform steadily beyond the gradient-based learner MES-MAML). In ES-MAML. the gradient can only be calculated after an episode is completed, while EPMLP is able to perform sequential learning even though no feedback available during its life time. This demonstrates the possibility of surpassing numan-designen gradient-based learning rules with automatically learned unsupervised rules. Moreover, comparison between EPMLP and EPMLP (Random) validate the proposal of Najarro & Risil (2020), implying the possibility of discovering global plastic learning rules instead of rules coupled with the initial

---

Under review as a conference paper at ICLR 2022
parameters. Yet optimization of the initial parameters is still advantageous, which also be validated by evidences in nature that the newborn lives already have certain embedded skills (e.g.. Newborn human babies have reflexes of suckling and grasping; Foals can stand shortly after being born).
-25
-15
-23
425
(a) Sequence Predicting (==1K=10.N=20)
(b) Sequence Predicting (I=3K==0.N=20)
-44
+10
-32
(c) Sequence Predicting
(d) Sequence Predicting
(E=1,K=25N==50)
(==,K=25,N=50)
43
-15
300
(e) Wheeled Robot Navigating (Low Noise)
(f) Wheeled Robot Navigating
(g) Wheeled Robot Navigating (High Noise)
(Median Noise)
Figure 4: Plotting meta-testing scores against meta-training iterations.
4.4 ANALYSIS AND DISCUSSION
To investigate whether plasticity rules update the connection weights as expected, we test the trained model with different tasks and record the updating trajectories of plastic connection weights Wa and hidden states h2z We run t-SNE visualization to map those tensors (W,s and h2S) to 2-D space and show their trajectories in Figure 5 The connection weights W, typically start from the same position and gradually move in different directions depending on the task configurations. The final weights effectively capture environmental configuration that was hidden from the agent. Particularly, in Figure d) for Wheeled Robot Navigating tasks, we can see that W, captures only the signal transmission patterns (Ao, k), but neglects are important stationary patterns that helps the agent to interpret the observed signal strength ( (A₂). while the absolute position of goal is less important as its relative position to the robot changing continuously. This demonstrates that plasticity has performed meaningful updates on the weights of the connections depending on the tasks. On the other hand, the hidden states (o) are noisier and less distinguishable among different tasks. The small repeated circles in the trajectory of hidden states might correspond to short-term patterns such as the periodicity of Sequence Predicting task and the circular movements in Wheeled Robot Navigation tasks. (see Appendices A.2
position of the goal (g., 9y)) We guess that Ao,
8

---

Under review as a conference paper at ICLR 2022
(a) W. for different A
(b) W, for different n
(c) W2 for different
fedt andiee
42
(d) ht for different A
(e) hr for different
(1) he for different Ø
POLEE WERLOD
-80
W, for diffferent Ao. k; and (h) by for different A, ks and
goals
goals
Figure 5: t-SNE visualization of the transformation of the connection weights (W,) and hidden states (he) the inner loop. (a)(b(()))(e(() are from Sequence Predicting (1=1,K==5.N=50) task, (g)(h) from Wheeled Robot Navigating (Low Noise) task.
Note that we did not compare EPRNN with many gradient- solutions such as differentiable plasticity (Miconi et al. 2018 2019), since the classic Hebbian plasticity has been observed and confirmed from neurons for more than half century (Hebb) 1949) - there is not evidences that the nature do differentiation or backpropagation (Lillicrap et al. 2020) update plastic rules. The purpose of our work is to reveal the potential of natural evolution of plasticity, and we don't hope to claim that EPRNN could outperform gradient- baaeed solutions on machine learning tasks.
CONCLUSIONS
In this paper we present EPRNN a nature-inspired learning framework composed of Evolution Strategies, Plastic rules, and Recursion. Experiment results show that plasticity can be effectively forged with recursion enhance the learning capability. The proposed framework can achieve equivalent or even better performances compared with more sophisticated neural structures, by applying the simplest recurrent neural structures. Moreover, we also show that under proper meta parameters, plasticity has chance to surpass gradient descent methods.
We believe the learning framework of Figure can be extended to more sophisticated plastic rules and model structures, uncovering better learners in the future. Also, it would be more interesting such framework can be validated in complex environments such natural language processing tasks and vision-related tasks. Finally, we are looking forward that this work can shed light to new paradigm of building intelligent machines and inspire efforts in this line of research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Horace B Barlow. Adaptation and decorrelation in the cortex. The computing neuron, 1989.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney, Learning to continually learn. In ECA/ 2020, pp. 1001. IOS Press, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Jeremie Cabessa and Hava T Siegelmann. The super-turing computational power of plastic recurrent neural networks. International journal of neural systems, 24(08): 2014.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint 1r..i::2006.11:5: 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Harald T Friss. A note on simple transmission formula. Proceedings of the IRE, 34(5):254-256, 1946.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1713. PMLR, 2018.
Wulfram Gerstner, Raphael Ritz, and J Leo Van Hemmen. Why spikes? hebbian learning and retrieval of time-resolved excitation patterns. Biological cybernetics, 69(5):503-515, 1993.
Samanwoy Ghosh-Dastidar and Hojjat Adeli. Spiking neural networks. International journal of neural systems, 19(04))295-308 2009.
Donald Olding Hebb. The organization of behavior; neuropsycholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780) 1780, 1997.
Ke Li and Jitendra Malik. Learning optimize. arXiv preprint arXiv:1606.01885, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
Bernabé -atares-Barranco and Teresa Cerrano-Gotarredone Memristance can explain spike-time- dependent-plastiictyy in neural synapses. Nature precedings, pp. 1-1, 2009.
Jack Lindsey and Ashok Stwin-Kuman Learning to learn with feedback and local plasticity, arXiv preprint arXiv:2006.095 2020.
Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, pp. 3559-3568.
PMLR, 2018.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URUhtttss:/oppeeeeeee
wwwp:/tttt/ta...tttmm d= A15AAiSYY
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks. arXiv preprint a 2xXiv:2007.02668 2020.
10

---

Under review as a conference paper at ICLR 2022
Joachim Winther Pedersen and Sebastian Risi. Evolving and merging hebbian learning rules:
Increasing generalization by decreasing the number of rules. arXiv preprint arXiv:2104.07959, 2021.
Daniel A Pollen. Explicit neural representations, recursive neural networks and conscious visual perception. Cerebral Cortex, 13(8):807-814 2003.
Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning, pp. 8007-8019. PMLR, 2020.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv:/703.03880 2017a.
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. CoRR, abs/ ftss/1703.03866 2017b. URL URLpttt://arrxiooog wtb./b//////a///ooo///o///...//a 1703. 03864
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In International conference on machine learning, pp. 1842-1850. PMLR, 2016.
Andrea Soltoggio, John A Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Floreano. Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In Proceedings of the Ith international conference on artificial life (Alife XI), number CONF, pp. 569-576. MIT Press, 2008
Andrea Soltoggio, Kenneth O Stanley, and Sebastian Risi. Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. Nearul Networks, 108:48-67, 2018.
Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao Tang. Es-maml: Simple dessian-free meta learning. In International Conference Learning Representations, 2019.
Kenneth O Stanley. Why open-endedness matters. Artificial life, 25(3):232-235, 2019.
Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, Matt Coler, George Fletcher, and Mykola Pechenizkiy. Evolving plasticity for autonomous learning under changing environmental conditions. Evolutionarr computation, 29(3):391 393-441414444 2021.
Jun Zhang, Zhi-hui Zhan, Ying Lin, Ni Chen, Yue-jiao Gong, Jing-hui Zhong, Henry SH Chung, Yun Li, and Yu-hui Shi. Evolutionary computation meets machine learning: A survey. IEEE Computational Intelligence Magazine, 6(4):68- 2011.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.

---

Under review as a conference paper at ICLR 2022
A APPENDICES
A.1 MODEL ARCHITECTURE AND TRAINING DETAILS
To maintain fairesss for comparison, the model architectures of the compared methods are similar, where the differences lie in only one of their layers (Figure (6. We use 3 hidden layers for Sequence Predicting tasks and hidden layers for Wheeled Robot Navigating tasks. For ES-MAML, we replace the PRNN layer with fully connected layer; For ES-RNN and ES-LSTM, we replace it with RNN and LSTM respectively; For EPMLP, we replace with plastic fully connected layer, where the plasticity rule is stated by Equation [ and the neural modulator m is calculated by an additional dense layer with sigmoid activation. The hidden sizes of all the hidden layers are 64 (for LSTM 64 hidden states and 64 cell states).
outpot
< 64
ouppul
Fully Connected RELU
Fully Connected RELU
+ 64
Fully Connected RELU <>
Fully RELU
PRNN
C
PRNN
RNN
Pully Connected RELU
LSTM
Fully Connected RELU
61
LSTM
Fully Connected RELU
Plastic Hully Connected
Plastic Fully Connected
e
uput
ingut
(a) Sequence Predicting
(b) Wheeled Robot Navigating
Figure 6: A sketch of the model architectures for evaluated tasks
For Sequence Predicting tasks, the input observation and the output action has the dimension of I. In the meta-training-Iruinee observation; In the the meta-training-Ee test and meta-testing-tent stages, its previous action (a,-1) are taken as inputs. For Wheeled Robot Navigating tasks, the output action is the control command of its two wheels (a length 2 vector), the input observation is the concatenation of its previous action and the current observed signal intensity (At).
and meta-lesting stages, we use the ground truth y:-1 input
Running Lnner-Lopp (ir(Ow) T)
Worker/ 1-rer/1
Worker d
Worker +
Worker m.
(i)(O..)
(T.....Thh Besss
Sumpling Tiskk (T)
In Neigabochoods Sumple Meta Parameter Ber. Update with: tees 9h Michhe (.) (i)()
Server
Figure 7: A sketch of the parallel training framework.
The training process is accelerated by utilizing the paralleling mechanism of PARL. We employ 400 workers (400 Intel(R) Xeon(R) CPU E5-26550 running inner-loops for 400 off-springs in each generation of the evolution, and additional 1 CPU to perform evolutionary update (shown in Figure (b). takes 3 to 12 hours for each run depending on the length the inner loop and model architectures. Following the previous work (Salimans et al. 2017a). we rank normalize the fitness among 400 workers. The learning rate a is set to 0.2. The mutation is performed by adding independent Gaussian noises each parameter. During the meta-ngaining,
sample B different tasks and 400 meta parameters. Each worker run B meta-trett eeeta-n graining-training and B meta-training- testing given the assigned meta-parameter, then the fitness is averaged. We set B 16 for Sequence
12

---

Under review as a conference paper at ICLR 2022
Predicting tasks, and B = 4 for Wheeled Robot Navigating tasks. For each meta-testing epoch, evaluated the current meta parameters in newly sampled 1600 tasks.
A.2 SUPPLEMENTARY EXPERIMENTAL RESULTS AND ANALYSES
We show more I-SNE visualization of the connection weights and hidden states in Figure X which can be compared with Figure The transformation of hidden states in ES-RNN (Figure 8 (b)(c and (e)) are even less distinguishable among different tasks compared with that EPRNN Figure/5dd)e)) and (h)), showing naive RNNs fail to capture task configurations. The weights of EPMLP can also capture some hidden configurations (Figure d)) but are less accurate compared with that of EPRNN
(Figure 5 g)) For instance, the gold and grey line share the same hidden configuration of A 0.5 and h = 0.5. The connection weights effectively captures this information and the two lines overlap with each other in EPRNN, but it does not happen in EPMLP. Similar phenomenon can be observed for the red and the green lines (A = 0.5 and h = 0.1).
AALE POck 4AL with
6-12 plo unaa
(a) he for different A in ES-RNN agent
(b) by for different in ES-RNN agent
(c) he for different é in ES-RNN agent
-30
12
(d) W2 for different Ao. and goals (e) h, for different Ag. and goals
EPMLP agent
ES-RNN agent
Figure 8: Supplementary t-SNE visualization of the transformation of the connection weights (W;) and hidden states (h.) in the inner loop. (a)(b)(c are from Sequence Predicting (1=1,K=25,N=50) task, (d)(e) are from Wheeled Robot Navigating (Low Noise) task.
We also plot the trajectories of the robot in Wheeled Robot Navigation tasks. All the agents struggle to find the goal by continuously shifting their directions, which more frequent in cases of higher observation noise. A reasonable guess is that in higher noise cases, the agents are less confident in their estimations. They have to more frequently adjust their directions to acquire diverse observations for more effective denoising.
13

---

Under review as a conference paper at ICLR 2022
EPRNN (Low Noise) (b) EPRNN (Median Noise) (c) EPRNN (High Noise)
(d) EPRNN (w/o m) (Median Noise)
(e) EPMLP (Median Noise)
ES-RNN (Median
Noisc)
(g) ES LSSM (Median Noisc)
(h) Random Controller
Figure 9: Trajectories of the agents from Wheeled Robot Navigating task with the task configuration
of (A = 0.5, & = 0.2, (g.r, 9y,) (-0.3.-00 -0.3)). We show the trajectories of the wheeled robot in black and red lines and the goal in green circles.
14


Reviews:
[{"review_id": "wCwhz74fkYv", "reviewer": "Reviewer_RBhw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n- Optimizing learning rules is interesting and it's a promising approach.\n\nWeaknesses:\n- The proposed method is not new (the combination with recurrent neural networks may be, but it is still of very limited novelty).\n- The language of the paper has many mistakes and is a bit informal in some parts (\"researchers are increasingly obsessed\", etc).\n- The parallel Biological Neural Networks is interesting, but very weak.\n", "summary_of_the_paper": "The authors explore the interesting (although not novel) topic of using derivative-free optimization / evolutionary algorithms to optimize learning rules.\nThe authors propose a method, EPRNN (Evolutionary Plastic Recurrent Neural Networks) that uses Evolution Strategies to learn a Hebbian learning rule for a recurrent neural network.", "summary_of_the_review": "The main issues with the paper are language and the minimal novelty of the proposed methods.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "evK4tLGkt5H", "reviewer": "Reviewer_mN1j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas behind the paper are interesting and investigating more biological-inspired methods that could potentially solve problems without gradient information are exciting. However, the contributions of this particular paper could be further elaborated on and the particular test domains are all rather simple. Beyond the approach introduced by Najarro&Risi (2020) and Soltoggio et al. (2018), the main addition is to also use a recurrent network. This by itself would be fine, but then the approach is only applied to a simpler RL domain than what Najarro&Risi investigated previously. Did the authors also try the approach on a more complex domain with higher input space? How did the approach perform there? That would be a more convincing demonstration of the power of RNNs+plasticity rules.\n\nAdditional comments:\n- The visualization of what the weights learn are interesting. What did the recurrent hidden states learn?\n\n- Why is the robot in Figure 3c navigating in these circles? It would be great to show trajectories before and after learning.\n\n- Is the distance signal used as input to the neural network, or only used for the reward (as in the point nagiation task in the original MAML paper)? If used as a network input, how well would it do without it? \n\n- \"In some cases, it even produces better results compared with LSTM, despite the simpler model architecture. “ -> I’m not sure I would agree that the architecture is necessarily simpler.\n\n- Why is only W_d mulitplied with the modulation signal in Equation 5?\n\n- \"Recursion-based learning employs recurrent neural networks (RNN), LSTM (Hochreiter & Schmidhuber, 1997), and self-attention (Mishra et al., 2018; Chen et al., 2021) layers as learners.” -> It would be great to have an exact definition of what the authors refer to as recursion.\n\n- Figure 3 should mention what the different colors mean.", "summary_of_the_paper": "In this work, the authors use evolutionary strategies to train recurrent neural networks with Hebbian plasticity rules. They test the system on two tasks, sequence prediction and a simple RL tasks that involve robot navigation. The approach is compared against previous work that uses plasticity but without recurrent connections and other approaches such as LSTMs. For the problems presented in this paper, the proposed approach outperforms most methods used in the comparison.", "summary_of_the_review": "An interesting paper but currently the approach would need to be tested on more complex problems to more fully demonstrate the advantages of using an RNN+plasticity architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JOtdUTcqM-q", "reviewer": "Reviewer_byce", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new flavor of meta-learning, which is an interesting and promising research avenue currently trending. The proposed approach particularly aims at joining plasticity learning, evolutionary computation and recurrent neural networks (more trending topics), in a blend yet unpublished.\nThe state of the paper however is pretty rough, and it would require considerable work before it can be considered for publication in a major venue. The experimental setup particularly is inconclusive: the tasks are overly simplistic, custom but unmotivated, and addressed with an unnecessarily complex system, leaving to much room as to where positive or negative performances come from a specific component.\n\nI would like to start by pointing out a few words or sentences taken from the paper that the authors should necessarily address -- just find them on the paper, I am confident the improvement can be evinced easily from the context. As a suggestion, focus on quantitative rather than qualitative remarks, and avoid claims unsustained in the literature (if not straight wrong).\n- Researchers are increasingly obsessed...\n- ANN have achieved great success [...] due to the strong capacity of handling large datasets\n- Gradient oracles\n- ANNs usually need overconsumption of datasets\n- In a mimic manner\n- Learning takes place in the feed-forward pass\n- Recursion are found to be extremely sample-efficient\n- The outer learning loop optimize the meta [hint: meta is an adjective, meta-learning is a type of learning, not \"learning a meta\"]\n- [...] model to specific task by utilizing the meta [yep it was not a typo]\nThere are plenty more, but I am confident that the authors can take it from here. I strongly suggest to involve an English native speaker for general proofreading and edit, as this work could gain significantly from an exposition targeted to an international audience.\n- It is found that evolution can be more efficient in cases of very long horizon in reinforcement learning [citing Salimans; this conclusion is not in the paper as far as I remember; would be more likely linked to the work by Stanley on open-endedness evolution]\n- Meta learning aims at building learning machines that gain experience\n- There are obviously bunches of more sophisticated neural structure\n- Each result is concluded from independent 3 runs\n\nHere is some feedback on the exposition. Also these are all requirements.\n- It is unnecessary to italicize and underline the already Title Case words for the EPRNN acronym. Nor to bold it 7 times in the abstract. Nor in the rest of the paper.\n- Some vocabulary which is uncommon for a ML audience and should be explained before usage, such as 'memristance' and biological 'plasticity'\n\nFinally approaching the technical perspective: there are serious claims that need to be backed by a scientific process.\n- Figure 1 illustrates a simplification of natural evolution as learning loop. The external loop is the hereditary passing of genetic material (left), the internal loop depicts the lifetime of an individual. The implication in this double-loop setup is that experience obtained by an individual through interacting with the environment is then passed genetically to its offspring, which incidentally would require self-modification of the DNA to incorporate the changes during the individual lifetime. Please confirm whether you stand by this interpretation, and provide citation to work done in verification and support to this hypothesis.\n- The list of contributions shows 3 points. The first is a study that I could not find in he actual paper. The second describes the proposed meta-learning framework without clearly showing the contribution (which as far as I understand, is limited to having put these three pieces together, without even supporting the case as to why it should be advantageous). The third point is the fact that you run experiments (unless it is meant to highlight how the tasks were custom-made, though the paper should highlight the relevance and importance of these two tasks as it is unclear). In conclusion, the list of contribution needs re-writing to correctly represent a list of actual contributions sustained in the paper.\n\nReference to past work is incomplete (if not even biased) from a ML perspective. As a consequence the arguments on this paper are often unnecessarily complex or simply incorrect, again from the perspective of ML literature. This is also a requirement for publication.\n- The literature review (and Table 1) should mention at the very least the work of Quoc Le and Esteban Real in AutoML, as it is the most active and recognized work in meta-learning of artificial neural networks.\n- Evolutionary Algorithms are referenced from Zhang et al. 2011, pointing to a lesser known paper with a dubious title. The only evolutionary computation work that is utilized in multiple places is the OpenAI ES, a single paper published on ArXiv without peer review, which itself lacks a correct representation of the long-standing field of evolutionary computation. A few names to build a better awareness of the field include foundation work from Koza and Banzhaf, then Miikkulainen into Stanley, and absolutely Hansen or even Glasmachers.\n- Even in something as fundamental as ANNs, the concepts of network architecture, recurrent connections, sample efficiency, gradient-based vs gradient-free, etc. are not as mysterious and vague as presented in this paper. I believe a more deep understanding of the underlying concepts can help the authors constructing a more compelling (and correct) argument.\n\nAs a concluding note: the authors should learn about neuroevolution of recurrent networks for continuous control tasks, as the literature has been going strong for over 20 years. Adding plasticity rules is not original per se, and wrapping into a meta-learning framework only automates hyperparameter optimization, itself minimal in modern evolutionary algorithms.\nThe proposed tasks are incredibly simple. As a requirement to change my mind on my recommendation, and to study the effective complexity of the tasks, I propose the authors to motivate extensively their need to construct their own custom tasks. As a baseline, I also see it necessary to include results from Random Weight Guessing of minimal-sized networks. For example, the first task (regression of a rescaled and translated sinusoidal with noise) can likely be solved by literally guessing (through subsequent random initializations) the weights of a neural network composed of only one layer of one recurrent neuron, or at worse one layer of 10 recurrent units followed by one feed-forward predictor. The architecture proposed in Figure 6 of two layers of 64 RELU sandwiching one layer of recurrent/plastic neurons is overly complex, highly inefficient, and not motivated.\nThe second task is even easier: a wheeled robot with power control on the wheels and a sensor estimating the distance from the goal. This class of problems is typically solved by a linear controller, as it requires no memory nor non-linearities, so I would expect to see the results of RWG on a network with just two feed-forward neurons (the ouput layer) again with no hidden layer. Once again the proposed structure, identical to the previous but for adding one extra 64-neurons fully-connected layer, is entirely out of scale for the problem, and unmotivated in the text.", "summary_of_the_paper": "The authors put together their own flavor of meta-learning using off-the-shelf plasticity learning, evolutionary computation and recurrent network. Experiments are conducted on two overly-simplistic, custom-designed tasks, and the results lack impact.\n", "summary_of_the_review": "While the introduction of yet another meta-learning framework should be supported, as the area is extremely promising, and the combination of plasticity learning rules plus evolutionary computation plus recurrent networks shows great potential, this paper is not in a shape proper for publication in a major conference, and the results on the overly-simplistic custom benchmarks are far from conclusive.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HyC4vjs2SQ", "reviewer": "Reviewer_oHzr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe model appears to perform better than alternative models in the two tasks tested.\n\nWeakness:\nIn my opinion, the main weakness of this work is the lack of intellectual insights or impressive empirical results. The main contribution is to apply evolutionary algorithm to meta-learn plasticity rules for RNNs. Given that there are quite a few papers in this area that involve two of the above elements (evolutionary algorithm, plasticity rules, RNNs), it is not too hard to combine the three.\n\nGiven the somehow arbitrary nature of the artificial tasks studied by the authors, it is difficult to tell whether the empirical results are impressive.\n\nMinor concerns:\nIntro: plastic rules --> plasticity rules\nIntro: “plastic rules, aka Hebb’s rule”, Hebb’s rule is only one kind of plasticity rule\nIn general, the text can be edited to improve clarity \nShould cite Confavreux Neurips 2020\n", "summary_of_the_paper": "The authors applied evolutionary algorithms to meta-learn plasticity rules for recurrent neural networks. They show this approach performs better than alternative meta-learning approaches on two artificial tasks (sequence prediction and wheeled robot navigation).", "summary_of_the_review": "The main challenge to this manuscript is the lack of intellectual insights, except a new combination of several existing techniques, and a lack of strong empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper the authors demonstrate the use of meta-learning in plastic recurrent neural networks with an evolutionary approach, avoiding gradients. They show that this approach can be used to develop networks that can solve problems like sequence prediction and simple navigation.

The reviews for this paper all had scores below the acceptance threshold (3,5,3,3). The principal concerns were:

(1) The lack of novelty. Other papers have taken very similar approaches (e.g. Najarro & Risi, 2020 or Miconi et al., 2019), and fundamentally this paper simply ties together different elements in one package.

(2) Lack of demonstration of the approach beyond some very simple tasks.

(3) Lack of connection to the related literature on neuro-evolution and ML. 

(4) General clarity and style of writing issues.

The authors responded to the reviewers, but the responses did not convince the reviewers enough to increase their scores past threshold. Given this, a reject decision was reached.
Decision: Reject