DO WHAT NATURE DID TO US: EVOLVING PLASTIC RECURRENT NEURAL NETWORKS FOR TASK GENER-
Year: 2021

Abstract:
While artificial neural networks (ANNs) have been widely adopted in machine learning, the gaps between ANNs and biological neural networks (BNNs) are receiving increasing concern. In this paper, we propose a framework named as Evolutionary Plastic Recurrent Neural Networks (EPRNN). Inspired by BNN, EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based Learning in one meta-learning framework for generalization to different tasks. More specifically, EPRNN incorporates nested loops for meta-learning — an outer loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to specific tasks. In the inner loop of EPRNN, we effectively attain both long-term and short-term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for memristance in BNNs. The inner-loop setting closely simulates BNNs, which neither use gradient-based optimization nor require the exact forms of learning objectives. To evaluate the performance of EPRNN, we carry out extensive experiments in two groups of tasks: Sequence Predicting, and Wheeled Robot Navigating. The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning-based approaches in the tasks. The experiment results suggest the potential of EPRNN to generalize to a variety of tasks and encourage more efforts in plasticity and recursion-based learning mechanisms.

text: While artificial neural networks (ANNs) have been widely adopted in machine learning, the gaps between ANNs and biological neural networks (BNNs) are receiving increasing concern. In this paper, we propose a framework named as Evolutionary Plastic Recurrent Neural Networks (EPRNN). Inspired by BNN, EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based Learning in one meta-learning framework for generalization to different tasks. More specifically, EPRNN incorporates nested loops for meta-learning — an outer loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to specific tasks. In the inner loop of EPRNN, we effectively attain both long-term and short-term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for memristance in BNNs. The inner-loop setting closely simulates BNNs, which neither use gradient-based optimization nor require the exact forms of learning objectives. To evaluate the performance of EPRNN, we carry out extensive experiments in two groups of tasks: Sequence Predicting, and Wheeled Robot Navigating. The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning-based approaches in the tasks. The experiment results suggest the potential of EPRNN to generalize to a variety of tasks and encourage more efforts in plasticity and recursion-based learning mechanisms.

heading: 1 INTRODUCTION
text: ANNs have achieved great success in handling machine learning tasks. Despite being initially inspired by Biological Neural Networks (BNNs), there are apparent gaps between ANNs and BNNs. Mainstream ANNs use gradient-based optimizers to minimize learning objectives. Shreds of evidence show that BNNs learn through plasticity (Gerstner et al., 1993) without explicit learning objectives, among which Hebb’s rule (Hebb, 1949) is most well known. Though gradient descent methods are the most efficient optimizers for ANNs, their side effects are also noticed, including the problems of catastrophic forgetting, over-consumption of data, and the requirement for manual efforts in designing objective functions. Those challenges are becoming an essential impedance to the further development of machine intelligence.
Recent studies show the learning mechanisms of BNNs, such as plasticity (Soltoggio et al., 2008; Najarro & Risi, 2020) and model-based learning (Santoro et al., 2016; Mishra et al., 2018), under appropriate meta-parameter optimization, can be effective alternative for task generalization in ANNs. Unlike gradient-based methods, these mechanisms simulate the learning behaviors of BNNs and don’t require any explicit-form learning objectives. More recently, authors in (Miconi et al., 2019) proposed a plastic recurrent neural network for lifelong learning of ANNs, where implements Hebbian plasticity with differentiable objectives and gradient-based optimization. Though the above studies have investigated learning of ANNs using the two mechanisms derived from BNNs with gradient-based methods (Miconi et al., 2019) optionally, in this work, we aim at further verify the path of discovering those rules evolutionarily, simulating that of BNNs.
Backgrounds. Though learning in BNNs has not been fully understood, some of the learning mechanisms and rules, such as plasticity (Gerstner et al., 1993) and recursion (Pollen, 2003), have been observed in brains and adopted by ANNs. Typically, Model-based learning employs recurrent neural networks (RNN), LSTM (Hochreiter & Schmidhuber, 1997), and self-attention (Mishra et al., 2018; Chen et al., 2021) layers as learners. Learning is based on memories within the feed-forward pass. The information is updated in the hidden states instead of the parameters. Model-based learners are found to be sample efficient in generalized supervised tasks (Santoro et al., 2016), zero-shot generalization in language (Brown et al., 2020), and reinforcement learning (Mishra et al., 2018; Chen et al., 2021) when compared with various type of gradient descent methods. So far, among model-based learners, though self-attention-based learners such as Transformers have state-of-the-art performance, the O(T 2) (where T is the sequence length) makes them only available to relatively short sequences. On the other hand, recurrent learners such as RNN and LSTM have O(T ) inference costs but suffer from poor asymptotic performances. That is, when sequences are getting longer, performances no longer improve or even deteriorate. It is partly due to the limitation of the memory spaces. For instance, an recurrent neural network of hidden size n has a memory of O(n). In contrast, its parameters scale with O(n2), making parameter-updating more powerful as learning mechanisms than recursion-only.
In addition to recursion-based learning, evolving plasticity (Soltoggio et al., 2008; 2018; Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021) has been proposed to reproduce the natural evolution and plasticity in simulation, as shown in Figure 1. Implementing plasticity is not straightforward; unlike gradient-based learning methods, plastic rules are not universal but have to be optimized beforehand, which is not possible without a further outer-loop optimizer over the inner-loop learning. Evolutionary algorithms (Zhang et al., 2011; Salimans et al., 2017a) are typically applied in the outer loop to search for meta-parameters shaping the learning rules, which can be regarded as information carried by genomes during evolution. Those optimized plasticity rules are then applied in the inner loop to further tune NN’s parameter for better adaptions to the environment. Another line of works tries to bring gradient-based learning algorithms to plasticity rule optimization (Miconi et al., 2018; 2019). It is found that evolution can be more efficient in cases of long-horizon in reinforcement learning (Salimans et al., 2017b; Stanley, 2019).
Our Works. Inspired by the previous works (Cabessa & Siegelmann, 2014; Miconi et al., 2018; 2019) that improve recursive neural networks using plastic rules for capacity of learning, we propose
a novel meta-learning framework namely Evolutionary Plastic Recurrent Neural Networks (EPRNN) for task generalization. Specifically, this work makes contributions as follows.
• We study the potential of learning plasticity and recursion rules through the natural evolution in task generation. We show that recursion and plasticity-based rules can surpass gradientbased methods as inner-loop learners.
• We present investigations and analyses on the learned rules and parameters, showing that the learning framework discovers plasticity rules that effectively update the connection weights according to the learning tasks. The differences between the transformation of hidden states and parameters are also shown, verifying the efficacy of combining recursion with plasticity.
The most relevant works to our study are (Miconi et al., 2018; 2019; Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021). Compared to (Miconi et al., 2018) that leverage gradient oracles to efficiently search plastic rules, the proposed EPRNN could work well in gradient-free settings. Compared to (Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021) that use evolutionary strategies to learn plastic rules, EPRNN also incorporates an RNN-based inner loop for recursion-based learning. The work (Miconi et al., 2019) also uses recursion (i.e., RNN) and differentiable plasticity in nested loops to train self-modifying neural networks, EPRNN replaces the outer loop with evolutionary strategies to generalize tasks with non-differentiable objectives. Though EPRNN is not as competitive as gradient-based methods, which can optimize advanced neural networks with large datasets, our work still demonstrates the potential of using plasticity and recursion for meta-learning through natural evolution.

heading: 2 RELATED WORKS
text: Meta-Learning. Meta-learning aims at building learning machines that gain experience using taskspecific data over the distribution of tasks. Inspired by human and animal brains that are born with both embedded skills and the capability of acquiring new skills, meta-learning implements two nested learning loops: The outer learning loops optimize the meta-parameters that typically involves initial parameters (Finn et al., 2017; Song et al., 2019), learning rules (Zoph & Le, 2017; Najarro & Risi, 2020; Pedersen & Risi, 2021), model structures (Soltoggio et al., 2008; Li & Malik, 2016) and all of three (Real et al., 2020) over distribution of tasks; The inner learning loops adapt the model to specific tasks by utilizing those meta-parameters. According to different inner-loop optimizers, we roughly classify the methods into model-based and parameter-updating methods. The model-based methods do not update the parameters in the inner-loop, where only hidden states is updated (e.g., recursion); The parameter-updating methods re-modify the connection weights in the inner-loop (e.g., gradient descent (MAML), plasticity). From this point of view, our method can be classified into both groups. A brief review of the typical meta-learning paradigms is presented in Table 1.
Plasticity-based Learning. The proposal of the learning mechanism of BNNs is initially raised by Hebb’s rule (Hebb, 1949), the most prominent part of which is “neurons fire together wire together”. It is further polished by Spike Time-Dependent Plasticity (STDP) (Gerstner et al., 1993) indicating that the signal of learning is dependent on the temporal patterns of the presynaptic spike and postsynaptic spike. Learning could also appear in inhibitory connections, also known as anti-Hebbian
(Barlow, 1989). Also, relationships between STDP and memory are investigated (Linares-Barranco & Serrano-Gotarredona, 2009). Since many of those rules are related to spiking neural networks (Ghosh-Dastidar & Adeli, 2009), to apply them to ANNs, simplified rules are proposed (Soltoggio et al., 2008) instead: given the pre-synaptic neuron state X and post-synaptic neuron state Y , the connections between X and Y are updated by
δW = m[A ·XY +B ·X + C · Y +D], (1) m is the output from neuron modulators that adjust the learning rates of plasticity. Most of the existing rules are sub-classes of Equation 1. For instance, some works neglect the neural modulator m (Najarro & Risi, 2020; Miconi et al., 2018), others have set B, C, and D to 0 (Miconi et al., 2018; 2019). The learned rule A,B,C,D will inevitably be dependent on initial parameter of W , however, learning plastic rules that is not dependent on the initial parameters was also investigated (Najarro & Risi, 2020; Yaman et al., 2021).

heading: 3 ALGORITHMS
text: Problem Settings. We consider an agent (learner) that is dependent on meta-parameter θ. It has the capability of adapting itself to a distribution of tasks Tj ∈ T by interacting with the environment Tj through observation it and action at. In K-shot learning, the agent is allowed to first observe samples of length K (this stage can be referred as meta-training-training, see Beaulieu et al. (2020)), then its fitness is calculated in meta-training-testing rollouts. In Generalized Supervised Learning tasks (GSL), the observations typically include features (xt) and labels (yt) in meta-training-training stage (it = {xt, yt}), and the labels are left out for predicting in the meta-training-testing stage (Santoro et al., 2016; Garnelo et al., 2018). In Generalized Reinforcement Learning tasks (GRL), the observations typically include states (st), actions (at−1), and feedbacks (rt−1) (it = {st, at−1, rt−1}, sometimes rt−1 can not be observed) (Mishra et al., 2018). The goal of meta-training is to optimize θ such that the agent achieves higher fitness in meta-training-testing. In meta-testing, similarly, the learned parameters are given meta-testing-training and meta-testing-testing in order, the performances in meta-testing-testing are evaluated.
Plastic Recurrent Neural Networks (PRNN). Given a sequence of observations i1, ..., it, ..., we first consider an recurrent neural network (RNN) that propagates forward and yields sequence of outputs at following:
ht+1 = σ(Wt · ht +Wi · it + b), (2) at = f(ht+1) (3)
where ht is the hidden states at step t. In PRNN, we kept Wi stationary, but we set Wt to be plastic, so that we add a subscript t to mark different Wt at different steps. Regarding ht as pre-synaptic neuron states, and ht+1 as post-synaptic neuron states, by applying Equation 1, we update Wt with:
Wt+1 =Wt + δWt (4)
δWt =WA (ĥt+1 · hTt ) +WB (mt · hTt ) +WC (ĥt+1 · 1T) +WD ·mt (5) ĥt+1 = mt ht+1, (6)
where we use and · to represent “element-wise multiplication” and “matrix multiplication” respectively. h and 1 are column vectors. WA,WB ,WC ,WD are collection of plastic rules of A,B,C,D from Equation 1, which has the same shape as Wt. mt is the neural modulators that adjusts the learning rates of plasticity. We calculate mt by applying a neuron modulating layer denoted with:
mt = σ(W (m) h · ht +W (m) i · it + b (m)). (7)
A sketch of PRNN is presented in Figure 2. The main difference between PRNN and naive RNN is that PRNN updates both the hidden states and the connection weights during the forward pass.
Evolving PRNN. Given task Tj ∈ T , by continuously applying Equation 2 to 7 over meta-trainingtraining and meta-training-testing, the fitness is eventually dependent on the initial parameters, learning rules, and the sampled task T , which is denoted as:
Fit(θ, T ) = Fitness(iK+1, aK+1, iK+2, aK+2, ..., ) (8)
Wi,W0,WA,WB ,WC ,WD,W (m) h ,W (m) i , b, b (m) ∈ θ (9)
Following Evolution Strategies (ES) (Salimans et al., 2017a), in kth outer-loop iteration, we sample different tasks from T , and meta-parameters θk,i (i ∈ [1, n]) from the neighbourhoods of θk. We evaluate the fitness of sampled meta-parameters, and update the meta-parameters by applying:
θk+1 = θk + α 1
n n∑ i=1 Fit(θk,i, Tk)(θk,i − θk) (10)
Why Recurrent Neural Networks? As stated in Equation 1, plasticity in feed-forward-only NNs allows NNs to gain experiences from single-frame observation only. In cases of non-sequential GSL, the plasticity has chances to tune the connection weights to the specific task by relying on observing one single frame of data (it = {xt, yt}), since its information of the feature and the supervision is complete. However, in general cases, learning can be effective without summarizing sequences of observations. For instances, a human driver getting used to a new car through continuously interacting and observing. Moreover, in GRL, there are time lag between the observation of states and feed-backs. Recursion helps to summarize historical observations to give the correct update for the connection weights.
Although, compared with naive RNN, there are obviously bunches of more sophisticated neural structure such as GRU and LSTM, we believe it is more desirable to start from simplest recurrent structure to study the potential of combining recursion and plasticity.

heading: 4 EXPERIMENTS

heading: 4.1 TASKS FOR EVALUATION
text: In generalized tasks, we have each task Tj dependent on some configuration parameters that are hidden from the agent. Below we introduce two groups of generalized tasks that we experiment on.
Sequence Predicting (Generalized Supervised Learning tasks). We randomly generate sequences of vectors y(t) = (y1(t), ..., yl(t)), where yi(t) = Aisin( 2πtni + φi), t = 1, 2, 3, ... and Ai ∼ U(1, 3), ni ∼ U(10, 100), φi ∼ U(0, 2π), and U(a, b) represents the uniform distribution between a and b. A, n and φ are hidden from the agent. The front part of the sequence y(1),y(2), ...,y(K) is exposed to the agent, and the left part y(K + 1),y(K + 2), ...,y(N) is to be predicted. The fitness is the opposite of mean square error (MSE) between the predicted sequence and the ground truth. We test the methods for comparison in four groups of tasks including (l = 1,K = 10, N = 20), (l = 1,K = 25, N = 50), (l = 3,K = 10, N = 20), and (l = 3,K = 25, N = 50) (see Figure 3(a)(b)).
Wheeled Robot Navigating (Generalized Reinforcement Learning tasks). The agent is to navigate a two-wheeled robot to a randomly generated goal in 2-D space g = (gx, gy). We assume that there is a signal transmitter on the goal and a receiver on the robot. The robot observes the signal intensity decided by At = A0 − k · log(dt/d0) + (inspired by the attenuation for electromagnetic wave, see Friis (1946)), where dt is the current distance between the robot and the goal, ∼ N (0, σ) is the white noise in the observation. gx, gy, A0, k are environment related configurations that is hidden from the agent. For each task, we sample configurations by gx, gy ∼ U(−0.5, 0.5),A0 ∼ U(0.5, 2.0), and k ∼ U(0.1, 0.5). The action is the rotation speed of its two wheels that controls the orientation and velocity of the robot. The reward at each step is rt = −dt, an episode terminates when the robot approaches the goal or steps reaches the maximum of 100. We also hide its own position and orientation from the agent, such that the agent relies on recording its own action the signal strength At to locate itself. We investigate three types of navigating circumstances with different level of noises in the observed signal: Low Noise (σ = 0.01), Median Noise (σ = 0.05), and High Noise (σ = 0.2) (see Figure 3(c)).

heading: 4.2 EXPERIMENT SETTINGS
text: We add the following methods into comparison, the methods share exactly the same outer loop and differ in the inner loop.
• ES-MAML (Song et al., 2019) : We use four gradient descent steps in the inner loop, the learning rate of each step is treated as meta-parameters which is to be optimized by the outer loop. As MAML can not utilize instant observation in zero-shot case, we show results of both ES-MAML (zero-shot) and ES-MAML (one-rollout) in Wheeled Robot Navigating. Except for ES-MAML, the other methods are measured with zero-shot meta-testing score only in Wheeled Robot Navigating.
• ES-RNN: Vanilla RNN as the inner loop learner. • ES-LSTM: Long Short Term Memory (Hochreiter & Schmidhuber, 1997) as the inner loop
learner.
• EPMLP (Soltoggio et al., 2018): Multi-Layer Perceptrons (MLP) with plasticity rules implemented.
• EPMLP (Random) (Najarro & Risi, 2020): The main difference with EPMLP is randomly setting the parameters of the plastic layers at the beginning of each inner loop instead of using fixed trainable initial parameters.
• EPRNN (w/o m): Removing neuron modulator (mt) from the plasticity rule in PRNN.
We add additional non-plastic fully connected layers before and after the plastic layers to increase the representation capability of the models. For fairness, we kept those layers identical for all compared methods. For every 100 outer loops in meta training, we add an extra meta testing epoch, evaluating the average fitness of current meta-parameters on testing tasks. Each run includes 15000 outer
loops and 150 meta-testing epochs. Each result is concluded from independent 3 runs. Our code1 relies on PARL2for parallelization. We leave the detailed illustration of the model structures and hyper-parameters in the Appendices.

heading: 4.3 RESULTS
text: We present the experiment results in Figure 4 (In Table 2, and Table 3, we also list the summarized performance by averaging the Top-3 meta-testing scores in the latest 10 meta-testing epochs of each run over 3 independent runs). Generally, we can conclude that PRNN performs substantially better when compared with naive RNN. In some cases, it even produces better results compared with LSTM, despite the simpler model architecture. It is also interesting to notice that the gap between RNN and PRNN are smaller in shorter sequences or low-noise environments, but larger in more challenging tasks with longer sequence or higher noise (In Wheeled Robot Navigating tasks, higher noise pushes the agent to maintain a longer memory in order to filter the noise and figure out the way to goal). This phenomenon reaffirms the lack of long-term memories in RNN, and shows that PRNN significantly improves this drawback. Comparing EPRNN (w/o m) with RNN and EPRNN clearly demonstrates that simple ABCD rule (without the neural modulator) may also work to some extent, but introducing the neuron modulator mt can further benefit the learner.
Among plasticity based methods, we show that recursion is more advantageous than MLP in evaluated tasks. It is also worth noticing that EPMLP and EPMLP (Random) perform steadily beyond the gradient-based learner (ES-MAML). In ES-MAML, the gradient can only be calculated after an episode is completed, while EPMLP is able to perform sequential learning even though no feedback is available during its life time. This demonstrates the possibility of surpassing human-designed gradient-based learning rules with automatically learned unsupervised rules. Moreover, comparison between EPMLP and EPMLP (Random) validate the proposal of Najarro & Risi (2020), implying the possibility of discovering global plastic learning rules instead of rules coupled with the initial
1https://github.com/WorldEditors/EvolvingPRNN 2https://github.com/PaddlePaddle/PARL
parameters. Yet we see optimization of the initial parameters is still advantageous, which can also be validated by evidences in nature that the newborn lives already have certain embedded skills (e.g., Newborn human babies have reflexes of suckling and grasping; Foals can stand shortly after being born).

heading: 4.4 ANALYSIS AND DISCUSSION
text: To investigate whether plasticity rules update the connection weights as expected, we test the trained model with different tasks and record the updating trajectories of plastic connection weights Wt and hidden states ht. We run t-SNE visualization to map those tensors (Wts and hts) to 2-D space and show their trajectories in Figure 5. The connection weights Wt typically start from the same position and gradually move in different directions depending on the task configurations. The final weights effectively capture environmental configuration that was hidden from the agent. Particularly, in Figure 5(d) for Wheeled Robot Navigating tasks, we can see that Wt captures only the signal transmission patterns (A0, k), but neglects the position of the goal (gx, gy). We guess that A0, k are important stationary patterns that helps the agent to interpret the observed signal strength (At), while the absolute position of goal is less important as its relative position to the robot is changing continuously. This demonstrates that plasticity has performed meaningful updates on the weights of the connections depending on the tasks. On the other hand, the hidden states (Figure 5(c)(d)(e)(h)) are noisier and less distinguishable among different tasks. The small repeated circles in the trajectory of hidden states might correspond to short-term patterns such as the periodicity of Sequence Predicting task and the circular movements in Wheeled Robot Navigation tasks. (see Appendices A.2)
Note that we did not compare EPRNN with many gradient-based solutions such as differentiable plasticity (Miconi et al., 2018; 2019), since the classic Hebbian plasticity has been observed and confirmed from neurons for more than half century (Hebb, 1949) — there is not evidences that the nature do differentiation or backpropagation (Lillicrap et al., 2020) to update plastic rules. The purpose of our work is to reveal the potential of natural evolution of plasticity, and we don’t hope to claim that EPRNN could outperform gradient-based solutions on machine learning tasks.

heading: 5 CONCLUSIONS
text: In this paper we present EPRNN a nature-inspired learning framework composed of Evolution Strategies, Plastic rules, and Recursion. Experiment results show that plasticity can be effectively forged with recursion to enhance the learning capability. The proposed framework can achieve equivalent or even better performances compared with more sophisticated neural structures, by applying the simplest recurrent neural structures. Moreover, we also show that under proper meta parameters, plasticity has a chance to surpass gradient descent methods.
We believe the learning framework of Figure 1 can be extended to more sophisticated plastic rules and model structures, uncovering better learners in the future. Also, it would be more interesting if such framework can be validated in more complex environments such as natural language processing tasks and vision-related tasks. Finally, we are looking forward that this work can shed light to new paradigm of building intelligent machines and inspire more efforts in this line of research.

heading: A APPENDICES
text: A.1 MODEL ARCHITECTURE AND TRAINING DETAILS
To maintain fairness for comparison, the model architectures of the compared methods are similar, where the differences lie in only one of their layers (Figure 6). We use 3 hidden layers for Sequence Predicting tasks and 4 hidden layers for Wheeled Robot Navigating tasks. For ES-MAML, we replace the PRNN layer with fully connected layer; For ES-RNN and ES-LSTM, we replace it with RNN and LSTM respectively; For EPMLP, we replace it with plastic fully connected layer, where the plasticity rule is stated by Equation 1, and the neural modulator m is calculated by an additional dense layer with sigmoid activation. The hidden sizes of all the hidden layers are 64 (for LSTM it is 64 hidden states and 64 cell states).
For Sequence Predicting tasks, the input observation and the output action has the dimension of l. In the meta-training-training and meta-testing-train stages, we use the ground truth yt−1 as input observation; In the the meta-training-test and meta-testing-test stages, its previous action (at−1) are taken as inputs. For Wheeled Robot Navigating tasks, the output action is the control command of its two wheels (a length 2 vector), the input observation is the concatenation of its previous action and the current observed signal intensity (At).
The training process is accelerated by utilizing the paralleling mechanism of PARL. We employ 400 workers (400 Intel(R) Xeon(R) CPU E5-2650) running inner-loops for 400 off-springs in each generation of the evolution, and additional 1 CPU to perform evolutionary update (shown in Figure 7(b)). It takes 3 to 12 hours for each run depending on the length of the inner loop and model architectures. Following the previous work (Salimans et al., 2017a), we rank normalize the fitness among 400 workers. The learning rate α is set to 0.2. The mutation is performed by adding independent Gaussian noises to each parameter. During the meta-training, we sample B different tasks and 400 meta-parameters. Each worker run B meta-training-training and B meta-trainingtesting given the assigned meta-parameter, then the fitness is averaged. We set B = 16 for Sequence
Predicting tasks, and B = 4 for Wheeled Robot Navigating tasks. For each meta-testing epoch, we evaluated the current meta parameters in newly sampled 1600 tasks.
A.2 SUPPLEMENTARY EXPERIMENTAL RESULTS AND ANALYSES
We show more t-SNE visualization of the connection weights and hidden states in Figure 8 which can be compared with Figure 5. The transformation of hidden states in ES-RNN (Figure 8(a)(b)(c) and (e)) are even less distinguishable among different tasks compared with that of EPRNN (Figure 5(d)(e)(f) and (h)), showing naive RNNs fail to capture task configurations. The weights of EPMLP can also capture some hidden configurations (Figure 8(d)) but are less accurate compared with that of EPRNN (Figure 5(g)). For instance, the gold and grey line share the same hidden configuration of A = 0.5 and k = 0.5. The connection weights effectively captures this information and the two lines overlap with each other in EPRNN, but it does not happen in EPMLP. Similar phenomenon can be observed for the red and the green lines (A = 0.5 and k = 0.1).
We also plot the trajectories of the robot in Wheeled Robot Navigation tasks. All the agents struggle to find the goal by continuously shifting their directions, which is more frequent in cases of higher observation noise. A reasonable guess is that in higher noise cases, the agents are less confident in their estimations. They have to more frequently adjust their directions to acquire diverse observations for more effective denoising.

Review summaries:
In this work, the authors use evolutionary strategies to train recurrent neural networks with Hebbian plasticity rules. They test the system on two tasks, sequence prediction and a simple RL tasks that involve robot navigation. The approach is compared against previous work that uses plasticity but without recurrent connections and other approaches such as LSTMs. For the problems presented in this paper, the proposed approach outperforms most methods used in the comparison.