Do What Nature Did To Us: Evolving Plastic Recurrent Neural Networks For Generalized Tasks

While artificial neural networks ( ANNs ) have been widely adopted in machine learning , the gaps between ANNs and biological neural networks ( BNNs ) are receiving increasing concern . In this paper , we propose a framework named as Evolutionary Plastic Recurrent Neural Networks ( EPRNN ) . Inspired by BNN , EPRNN composes Evolution Strategies , Plasticity Rules , and Recursion-based Learning in one meta-learning framework for generalization to different tasks . More specifically , EPRNN incorporates nested loops for meta-learning — an outer loop searches for optimal initial parameters of the neural network and learning rules ; an inner loop adapts to specific tasks . In the inner loop of EPRNN , we effectively attain both long-term and short-term memory by forging plasticity with recursion-based learning mechanisms , both of which are believed to be responsible for memristance in BNNs . The inner-loop setting closely simulates BNNs , which neither use gradient-based optimization nor require the exact forms of learning objectives . To evaluate the performance of EPRNN , we carry out extensive experiments in two groups of tasks : Sequence Predicting , and Wheeled Robot Navigating . The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning-based approaches in the tasks . The experiment results suggest the potential of EPRNN to generalize to a variety of tasks and encourage more efforts in plasticity and recursion-based learning mechanisms . 1 INTRODUCTION . ANNs have achieved great success in handling machine learning tasks . Despite being initially inspired by Biological Neural Networks ( BNNs ) , there are apparent gaps between ANNs and BNNs . Mainstream ANNs use gradient-based optimizers to minimize learning objectives . Shreds of evidence show that BNNs learn through plasticity ( Gerstner et al. , 1993 ) without explicit learning objectives , among which Hebb ’ s rule ( Hebb , 1949 ) is most well known . Though gradient descent methods are the most efficient optimizers for ANNs , their side effects are also noticed , including the problems of catastrophic forgetting , over-consumption of data , and the requirement for manual efforts in designing objective functions . Those challenges are becoming an essential impedance to the further development of machine intelligence . Recent studies show the learning mechanisms of BNNs , such as plasticity ( Soltoggio et al. , 2008 ; Najarro & Risi , 2020 ) and model-based learning ( Santoro et al. , 2016 ; Mishra et al. , 2018 ) , under appropriate meta-parameter optimization , can be effective alternative for task generalization in ANNs . Unlike gradient-based methods , these mechanisms simulate the learning behaviors of BNNs and don ’ t require any explicit-form learning objectives . More recently , authors in ( Miconi et al. , 2019 ) proposed a plastic recurrent neural network for lifelong learning of ANNs , where implements Hebbian plasticity with differentiable objectives and gradient-based optimization . Though the above studies have investigated learning of ANNs using the two mechanisms derived from BNNs with gradient-based methods ( Miconi et al. , 2019 ) optionally , in this work , we aim at further verify the path of discovering those rules evolutionarily , simulating that of BNNs . Backgrounds . Though learning in BNNs has not been fully understood , some of the learning mechanisms and rules , such as plasticity ( Gerstner et al. , 1993 ) and recursion ( Pollen , 2003 ) , have been observed in brains and adopted by ANNs . Typically , Model-based learning employs recurrent neural networks ( RNN ) , LSTM ( Hochreiter & Schmidhuber , 1997 ) , and self-attention ( Mishra et al. , 2018 ; Chen et al. , 2021 ) layers as learners . Learning is based on memories within the feed-forward pass . The information is updated in the hidden states instead of the parameters . Model-based learners are found to be sample efficient in generalized supervised tasks ( Santoro et al. , 2016 ) , zero-shot generalization in language ( Brown et al. , 2020 ) , and reinforcement learning ( Mishra et al. , 2018 ; Chen et al. , 2021 ) when compared with various type of gradient descent methods . So far , among model-based learners , though self-attention-based learners such as Transformers have state-of-the-art performance , the O ( T 2 ) ( where T is the sequence length ) makes them only available to relatively short sequences . On the other hand , recurrent learners such as RNN and LSTM have O ( T ) inference costs but suffer from poor asymptotic performances . That is , when sequences are getting longer , performances no longer improve or even deteriorate . It is partly due to the limitation of the memory spaces . For instance , an recurrent neural network of hidden size n has a memory of O ( n ) . In contrast , its parameters scale with O ( n2 ) , making parameter-updating more powerful as learning mechanisms than recursion-only . In addition to recursion-based learning , evolving plasticity ( Soltoggio et al. , 2008 ; 2018 ; Lindsey & Litwin-Kumar , 2020 ; Yaman et al. , 2021 ) has been proposed to reproduce the natural evolution and plasticity in simulation , as shown in Figure 1 . Implementing plasticity is not straightforward ; unlike gradient-based learning methods , plastic rules are not universal but have to be optimized beforehand , which is not possible without a further outer-loop optimizer over the inner-loop learning . Evolutionary algorithms ( Zhang et al. , 2011 ; Salimans et al. , 2017a ) are typically applied in the outer loop to search for meta-parameters shaping the learning rules , which can be regarded as information carried by genomes during evolution . Those optimized plasticity rules are then applied in the inner loop to further tune NN ’ s parameter for better adaptions to the environment . Another line of works tries to bring gradient-based learning algorithms to plasticity rule optimization ( Miconi et al. , 2018 ; 2019 ) . It is found that evolution can be more efficient in cases of long-horizon in reinforcement learning ( Salimans et al. , 2017b ; Stanley , 2019 ) . Our Works . Inspired by the previous works ( Cabessa & Siegelmann , 2014 ; Miconi et al. , 2018 ; 2019 ) that improve recursive neural networks using plastic rules for capacity of learning , we propose a novel meta-learning framework namely Evolutionary Plastic Recurrent Neural Networks ( EPRNN ) for task generalization . Specifically , this work makes contributions as follows . • We study the potential of learning plasticity and recursion rules through the natural evolution in task generation . We show that recursion and plasticity-based rules can surpass gradientbased methods as inner-loop learners . • We present investigations and analyses on the learned rules and parameters , showing that the learning framework discovers plasticity rules that effectively update the connection weights according to the learning tasks . The differences between the transformation of hidden states and parameters are also shown , verifying the efficacy of combining recursion with plasticity . The most relevant works to our study are ( Miconi et al. , 2018 ; 2019 ; Lindsey & Litwin-Kumar , 2020 ; Yaman et al. , 2021 ) . Compared to ( Miconi et al. , 2018 ) that leverage gradient oracles to efficiently search plastic rules , the proposed EPRNN could work well in gradient-free settings . Compared to ( Lindsey & Litwin-Kumar , 2020 ; Yaman et al. , 2021 ) that use evolutionary strategies to learn plastic rules , EPRNN also incorporates an RNN-based inner loop for recursion-based learning . The work ( Miconi et al. , 2019 ) also uses recursion ( i.e. , RNN ) and differentiable plasticity in nested loops to train self-modifying neural networks , EPRNN replaces the outer loop with evolutionary strategies to generalize tasks with non-differentiable objectives . Though EPRNN is not as competitive as gradient-based methods , which can optimize advanced neural networks with large datasets , our work still demonstrates the potential of using plasticity and recursion for meta-learning through natural evolution . 2 RELATED WORKS . Meta-Learning . Meta-learning aims at building learning machines that gain experience using taskspecific data over the distribution of tasks . Inspired by human and animal brains that are born with both embedded skills and the capability of acquiring new skills , meta-learning implements two nested learning loops : The outer learning loops optimize the meta-parameters that typically involves initial parameters ( Finn et al. , 2017 ; Song et al. , 2019 ) , learning rules ( Zoph & Le , 2017 ; Najarro & Risi , 2020 ; Pedersen & Risi , 2021 ) , model structures ( Soltoggio et al. , 2008 ; Li & Malik , 2016 ) and all of three ( Real et al. , 2020 ) over distribution of tasks ; The inner learning loops adapt the model to specific tasks by utilizing those meta-parameters . According to different inner-loop optimizers , we roughly classify the methods into model-based and parameter-updating methods . The model-based methods do not update the parameters in the inner-loop , where only hidden states is updated ( e.g. , recursion ) ; The parameter-updating methods re-modify the connection weights in the inner-loop ( e.g. , gradient descent ( MAML ) , plasticity ) . From this point of view , our method can be classified into both groups . A brief review of the typical meta-learning paradigms is presented in Table 1 . Plasticity-based Learning . The proposal of the learning mechanism of BNNs is initially raised by Hebb ’ s rule ( Hebb , 1949 ) , the most prominent part of which is “ neurons fire together wire together ” . It is further polished by Spike Time-Dependent Plasticity ( STDP ) ( Gerstner et al. , 1993 ) indicating that the signal of learning is dependent on the temporal patterns of the presynaptic spike and postsynaptic spike . Learning could also appear in inhibitory connections , also known as anti-Hebbian ( Barlow , 1989 ) . Also , relationships between STDP and memory are investigated ( Linares-Barranco & Serrano-Gotarredona , 2009 ) . Since many of those rules are related to spiking neural networks ( Ghosh-Dastidar & Adeli , 2009 ) , to apply them to ANNs , simplified rules are proposed ( Soltoggio et al. , 2008 ) instead : given the pre-synaptic neuron state X and post-synaptic neuron state Y , the connections between X and Y are updated by δW = m [ A ·XY +B ·X + C · Y +D ] , ( 1 ) m is the output from neuron modulators that adjust the learning rates of plasticity . Most of the existing rules are sub-classes of Equation 1 . For instance , some works neglect the neural modulator m ( Najarro & Risi , 2020 ; Miconi et al. , 2018 ) , others have set B , C , and D to 0 ( Miconi et al. , 2018 ; 2019 ) . The learned rule A , B , C , D will inevitably be dependent on initial parameter of W , however , learning plastic rules that is not dependent on the initial parameters was also investigated ( Najarro & Risi , 2020 ; Yaman et al. , 2021 ) . 3 ALGORITHMS . Problem Settings . We consider an agent ( learner ) that is dependent on meta-parameter θ . It has the capability of adapting itself to a distribution of tasks Tj ∈ T by interacting with the environment Tj through observation it and action at . In K-shot learning , the agent is allowed to first observe samples of length K ( this stage can be referred as meta-training-training , see Beaulieu et al . ( 2020 ) ) , then its fitness is calculated in meta-training-testing rollouts . In Generalized Supervised Learning tasks ( GSL ) , the observations typically include features ( xt ) and labels ( yt ) in meta-training-training stage ( it = { xt , yt } ) , and the labels are left out for predicting in the meta-training-testing stage ( Santoro et al. , 2016 ; Garnelo et al. , 2018 ) . In Generalized Reinforcement Learning tasks ( GRL ) , the observations typically include states ( st ) , actions ( at−1 ) , and feedbacks ( rt−1 ) ( it = { st , at−1 , rt−1 } , sometimes rt−1 can not be observed ) ( Mishra et al. , 2018 ) . The goal of meta-training is to optimize θ such that the agent achieves higher fitness in meta-training-testing . In meta-testing , similarly , the learned parameters are given meta-testing-training and meta-testing-testing in order , the performances in meta-testing-testing are evaluated . Plastic Recurrent Neural Networks ( PRNN ) . Given a sequence of observations i1 , ... , it , ... , we first consider an recurrent neural network ( RNN ) that propagates forward and yields sequence of outputs at following : ht+1 = σ ( Wt · ht +Wi · it + b ) , ( 2 ) at = f ( ht+1 ) ( 3 ) where ht is the hidden states at step t. In PRNN , we kept Wi stationary , but we set Wt to be plastic , so that we add a subscript t to mark different Wt at different steps . Regarding ht as pre-synaptic neuron states , and ht+1 as post-synaptic neuron states , by applying Equation 1 , we update Wt with : Wt+1 =Wt + δWt ( 4 ) δWt =WA ( ĥt+1 · hTt ) +WB ( mt · hTt ) +WC ( ĥt+1 · 1T ) +WD ·mt ( 5 ) ĥt+1 = mt ht+1 , ( 6 ) where we use and · to represent “ element-wise multiplication ” and “ matrix multiplication ” respectively . h and 1 are column vectors . WA , WB , WC , WD are collection of plastic rules of A , B , C , D from Equation 1 , which has the same shape as Wt . mt is the neural modulators that adjusts the learning rates of plasticity . We calculate mt by applying a neuron modulating layer denoted with : mt = σ ( W ( m ) h · ht +W ( m ) i · it + b ( m ) ) . ( 7 ) A sketch of PRNN is presented in Figure 2 . The main difference between PRNN and naive RNN is that PRNN updates both the hidden states and the connection weights during the forward pass . Evolving PRNN . Given task Tj ∈ T , by continuously applying Equation 2 to 7 over meta-trainingtraining and meta-training-testing , the fitness is eventually dependent on the initial parameters , learning rules , and the sampled task T , which is denoted as : Fit ( θ , T ) = Fitness ( iK+1 , aK+1 , iK+2 , aK+2 , ... , ) ( 8 ) Wi , W0 , WA , WB , WC , WD , W ( m ) h , W ( m ) i , b , b ( m ) ∈ θ ( 9 ) Following Evolution Strategies ( ES ) ( Salimans et al. , 2017a ) , in kth outer-loop iteration , we sample different tasks from T , and meta-parameters θk , i ( i ∈ [ 1 , n ] ) from the neighbourhoods of θk . We evaluate the fitness of sampled meta-parameters , and update the meta-parameters by applying : θk+1 = θk + α 1 n n∑ i=1 Fit ( θk , i , Tk ) ( θk , i − θk ) ( 10 ) Why Recurrent Neural Networks ? As stated in Equation 1 , plasticity in feed-forward-only NNs allows NNs to gain experiences from single-frame observation only . In cases of non-sequential GSL , the plasticity has chances to tune the connection weights to the specific task by relying on observing one single frame of data ( it = { xt , yt } ) , since its information of the feature and the supervision is complete . However , in general cases , learning can be effective without summarizing sequences of observations . For instances , a human driver getting used to a new car through continuously interacting and observing . Moreover , in GRL , there are time lag between the observation of states and feed-backs . Recursion helps to summarize historical observations to give the correct update for the connection weights . Although , compared with naive RNN , there are obviously bunches of more sophisticated neural structure such as GRU and LSTM , we believe it is more desirable to start from simplest recurrent structure to study the potential of combining recursion and plasticity .

Summary: In this work, the authors use evolutionary strategies to train recurrent neural networks with Hebbian plasticity rules. They test the system on two tasks, sequence prediction and a simple RL tasks that involve robot navigation. The approach is compared against previous work that uses plasticity but without recurrent connections and other approaches such as LSTMs. For the problems presented in this paper, the proposed approach outperforms most methods used in the comparison.