Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learning
Authors: Fan Wang
Hao Tian
Haoyi Xiong
Hua Wu
Jie Fu
Yang Cao
Yu Kang
Haifeng Wang
Venue: Accepted by TMLR ()

Abstract: Artificial neural networks (ANNs) are typically confined to accomplishing pre-defined tasks by learning a set of static parameters. In contrast, biological neural networks (BNNs) can adapt to various new tasks by continually updating the neural connections based on the inputs, which is aligned with the paradigm of learning effective learning rules in addition to static parameters, \textit{e.g.}, meta-learning. Among various biologically inspired learning rules, Hebbian plasticity updates the neural network weights using local signals without the guide of an explicit target function, thus enabling an agent to learn automatically without human efforts. However, typical plastic ANNs using a large amount of meta-parameters violate the nature of the genomics bottleneck and potentially deteriorate the generalization capacity. This work proposes a new learning paradigm decomposing those connection-dependent plasticity rules into neuron-dependent rules thus accommodating $\Theta(n^2)$ learnable parameters with only $\Theta(n)$ meta-parameters. We also thoroughly study the effect of different neural modulation on plasticity. Our algorithms are tested in challenging random 2D maze environments, where the agents have to use their past experiences to shape the neural connections and improve their performances for the future. The results of our experiment validate the following: 1. Plasticity can be adopted to continually update a randomly initialized RNN to surpass pre-trained, more sophisticated recurrent models, especially when coming to long-term memorization. 2. Following the genomics bottleneck, the proposed decomposed plasticity can be comparable to or even more effective than canonical plasticity rules in some instances.

Published in Transactions on Machine Learning Research (09/2022)
Evolving Decomposed Plasticity Rules for Information-
Bottlenecked Meta-Learning
Fan Wangl.2
wang.fan@baadu.ccmm com tionhoo@aaidu, com zionghaoyi@baiduu com ww he_lhaaauiid.. com fujie Tujie@baai.oo oc..c..cccccm forrest@ustc. edu.ccn
Hao Tian'
Haoyi Xiong
Hua Wu'
Jie Fu²
Yang Cao²
Yu Kang²
kangdugu@ustc... edu.cn
Haifeng Wang!
wonghaiffnng Qbaidaaangguuauu..omooou com
BBidd Inc. Universitt of Science and Technology of China 3 Beijjing Academy of Artificial Intelligence
Reviewed on OpenReview: https: openreview. net/ forum? id= 6ghK2ttPon
Abstract
Artificial neural networks (ANNs) are typically confined to accomplishing pre-defined tasks by learning set of static parameters. In contrast, biological neural networks (BNNs) can adapt to various new tasks by continually updating the neural connections based on the inputs, which aligned with the paradigm of learning effective learning rules addition to static parameters, e.g,, meta-learning. Among various biologically inspired learning rules, Hebbian plasticity updates the neural network weights using local signals without the guide of an explicit target function, thus enabling an agent learn automatically without human efforts. However, typical plastic ANNs using large amount of meta- parameters violate the nature of the genomics bottleneck and potentially deteriorate the generalization capacity. This work proposes a new learning paradigm decomposing those connection-dependee plasticity rules into neuron-dependent rules thus accommodating O(n²) learnable parameters with only O(n) meta- partamtrrsssers. We also thoroughly study the effect of different neural modulation on plasticity. Our algorithms are tested challenging random 2D maze environments, where the agents have to use their past experiences shape the neural connections and improve their performances for the future. The results of our experiment validate the following: 1. Plasticity can be adopted to continually update randomly initialized RNN to surpass pre-trained, more sophisticated recurrent models, especially when coming to long-term memorization. 2. Following the genomics bottleneck, the proposed decomposed plasticity can be comparable to or even more effective than canonical plasticity rules some instances.
1 Introduction
Artificial Neural Networks (ANNs) with vast number of parameters have achieved great success in various tasks (LeCun et al. 2015). Despite their capability of accomplishing pre-defined lasks, the generalizability to various new tasks at low costs is much questioned. Biological Neural Networks (BNNs) acquire new skills continually within their lifetime through neuronal plasticity (Hebb) 1949). learning mechanism that shapes the neural connections based on local signals (pre-synaptic and post-synaptic neuronal states) only. In contrast, typical ANNs trained once and for all and hardly be applied to unseen tasks.

---

Published in Transactions on Machine Learning Research (09/2022)
Generation k-1
Genotype 10
Phenotype 10
a.k.a Meta-Parameters
Statie & Adaptive Components
DOG
Doo
DOG
Generation k
Aduptive Statie
Dou
DOO
2o0
Learning
Generation k+1
D
Doo
DAD
DDu
Individual Lifetime (Inner Loop)
Fitness influencing
Natural Evolution (Outer Loop)
offspring generation
A simple illustration of the emergence of BNNs compared with meta-learning: 'he genotype evolved based on its fitness, which may be the result individual development in its life cycle. The phenotype is initially dependent on the genotype but. updated its life cycle. We may more specifically divide into the static components and adaptive components. The static components will not change within its life cycle, including the learning functionalities, initial neural architectures, and static neural connections. The adaptive components are continually updated with learning, including the neuron's internal states and plastic neural connections. The natural evolution and individual life cycle are similar to the nested learning loops of meta-learning, which include the outer loop and the inner loop. The genotype is similar to meta-parameters, and the phenolype corresponds to the parameters and hidden states in ANNs. Inspired by BNNs, the genotype that has a low capacity for information (e.g., 10 base pairs in the DNAs of human beings) decides the learning mechanisms and the initialization of the phenotype. The phenotype has a massive capacity memorize information (e.g. up to 1012 synapses in the human brain).
Figure I:
More recently, the emergence of BNNs has been used inspire meta-learning (oohh & Le 2017 Finn al. 2017 Instead of training static neural network once and for all, meta-learning searches for learning rules, initialization settings, and model architectures that could be generalizable to various tasks. We therefore make analogy between meta-learning and BNNs (Figure 1]: The natural evolution and lifetime learning of BNNs correspond to the nested learning loops (outer loop and inner loop) in meta-learning: The genotype corresponds to the meta-parameters shaping the innate ability and the learning rules; The phenotype corresponds to the neural connections and hidden states that could be updated within the inner loop.
Although the full scope of the inner-loop learning mechanism of humans is not yet well known, investigations in this area have been used to interpret even design ANN-based learning algorithms Niv 2009 Averbeck & Costa 2017) for specific tasks. Those specific learning algorithms (including supervised learning, unsupervised learning, and reinforcement learning) can be applied directly to the inner loop in meta-learning {Finn et al. 2017). However, the majority of the previous works over-rely on the expert's efforts in objective function design, data cleaning, optimizer selection, etc., thus reducing the potential of automatically generalizing across tasks. Another class of meta-learning does not explicitly seek to understand the learning mechanism; Instead, they implicitly embed different learning algorithms in black-box mechanisms such as recursion (Duan et al. 2016) and Hebbian plasticity (Soltoggio al.) 2008 Najarro & Risi 2020).. Results have shown that these automated "black-bor box learning mechanisms be more sample efficient and less noise-sensitive than human-designed gradient-based learners. Unfortunately, they suffer from other defects: Model-based Iearners can be less effective when the life cycles are longer, Hebbian plasticity-basee learners require many more meta-parameters than the neural connections, raising considerable challenger in meta-training.
Considering the analogy between meta-lcarning and the emergence of BNNs, most previous works fail to meet significant hypothesis widely assumed in research on BNNs: Genomic Bottleneck (Zador 2019 Pedersen & Risi 2021 Koulakov al. 2021). Compared to existing meta-learning algorithms that intensively rely
2

---

Published in Transactions on Machine Learning Research (09/2022)
pre-training many meta-parameterss (genotype) Yosinski et al. 2014 Finn et al. 2017 with less adaptive
components (in the phenotype), BNNs actually acquire more information within the life cycle than those inherited from genotype (see Figure D. In our opinion, the genomics bottleneck preserves relatively high learning potential while keeping the evolutionary process light. Previous rescarch further finds that reducing meta-parameterss also has a positive correlation with the generalizability of ANNs (Risi & Stanley 2010
Pederson & Risi 2021).
Motivated by the aforementioned considerations, we propose meta-learning framework with fewer meta- parameters and more adaptive components. The proposal is remarked with the following aspects:
We revisit the canonical Hebbian plasticity rules (Hebb 1949 Soltoggio et al. 2008 that employ to 4 meeta-parameters for each neural connection and propose decomposed plasitcity. Instead of assigning unique plasticity rules for each neural connection, assume that the plasticity rules depend on the pre-synaplic neuron and post-synaplic neuron separately. As a result, we introduce decomposition of the plasticity rules reducing the meta parameesss from O(n²) to O(n) (n is the number of neurons). For better generality and fewer meta-parameterss, we further follow Najarro & Risi (2020) to force the plasticity rules to update neural connections from scratch, ie., from random initialization, instead of searching for proper initialization of the connection weights.
Following Miconi et al. 2018 2019); we combine Hebbian plasticity with recursion based learners in a mneaaleerring framework. But, in order to satisfy the genomic bottleneck, we go step further by validating that plasticity rules can update the entire recursive neural network from scratch, including both the recurrent neural connections and the input neural connections. We also propose method for visualizing plasticity-based learners, showing that plasticity is potentially more capable of long-term memorization than recurrence-based learners.
Inspired by neural modulation in biological systems (Burrell & Sahley 2003), we investigate the details of neural modulation in plastic ANNs. We validate that even a single neural modulator can be crucial to plasticity. We also validate that the signals from which the modulator neurons integrate can significantly affect performance.
2001 Birmingham & Tauck
We select the tasks of 2D random maze navigation to validate our proposal, where the maze architectures, the agent origins, and the goals are randomly generated. The agents can only observe their surrounding locations and have no prior knowledge of the maze and the destination. Compared with the other benchmarks, is able to generate endless must preserve both short-term and long-term memory to localize themselves while exploiting shorter paths. Following the genomics bottleneck, we found that the decomposed plasticity yields comparable or even better performance than canonical plasticity rules while requiring fewer outer-loop learning steps. We also validate that plasticity can be a better long-term memorization mechanism than recurrence. For instance, our plastic RNNs surpass Meta-LSTM with over 20K parameessss very challenging lasks by using only 1.3K meta- peta-parameters.
distinct tasks, thus effectively testing the agents' generalizability. The agents
2
Related Works
2.1
Deep Meta-Learning
In meta-learning, an agent gains experience in adapting to a distribution of tasks with nested learning loops:
The outer learning loop optimizes the meta-parameters that may involve initializing settings Finn et al.)
2017 Song et al. 2019), learning rules (Li & Malik, 2016 Oh et al.] 2020 Najarro & Risi 2020 Pedersen &
Risi 2021), and model architectures (Zoph & Le 2017 Liu et al. 2018 Real ct al. 2019):
;
The inner lcarning
loops adapl the model to specific lasks by utilizing the meta-past partappprameers... Based on the type of inner-loop
learners, these methods can be roughly classified into gradient- gradient-based (Finn et al. 2017 Song et al. 2019
model-based (Santoro et al. 2016 Duan et al.) 20166] Mishra et al., 2018 Wang al. 2018 ) and metric-basel
Koch et al. 2015) methods (Huisman et al.) 2021). In addition, the Plasficity-based (Soltoggio et al. 2008
2018 Najarro & Risi 2020) methods update the connection weights of neural networks in the inner loop, but
3

---

Published in Transactions on Machine Learning Research (09/2022)
not through gradients. A key advantage of plasticity and model-based learning is the capability of learning withoul human-designed objeclives and optimizers. Our work combines both model-based and plasticity-based meta-learning.
2.2 Model-based Meta-Learning
Models with memorics (including recurrence and self-attention) are capable of adapting to various tasks by continually updating their memory through forwarding (Hochreiter & Schmidhuber 1997). Those models are found to be effective in automatically discovering supervised learning rules (Santoro et al. 2016), even complex reinforcement learning rules (Duan et al. 2016 Mishra et al., 2018). Similar capabilities are found in largo-scalc language models (Brown et al.) 2020). Model-basod learners own potential of unifying all different learning paradigms (supervised learning, unsupervised learning, reinforcement learning) within one paradigm. Still, the limitations of these learners become evident when life cycles get long. A reasonable guess is that the limited memory space restricted the learning potential since the adaptive components are typically much sparser than the static components (for the recurrent models, the adaptive components are the hidden states, which is in the order of (n)); The static components are the neural connections, which is 6(n²), with n being the number of hidden units). contrast, learning paradigms that update the neural connections have higher learning potential and better asymptotic performances.
2.3 Plastic Artificial Neural Networks
The synaplic plasticily of BNNs is found to depend on the pre-synaptic and post-synaplic neuronal states, which is initially raised by Hebb's rule (Hebb) 1949), known as "neurons that fire together wire together". Hebb's rule allows neural connections to be updated using only local signals, including the pre-synaptic and post-synaptic neuronal states. For ANNs, those rules are found ineffective without proper modulation and meta-parametens. For instance, in the aABCD plasticity rule (Soltoggio et al. 2008). given the pre-synaptic neuron state I, and post-synaptic neuron state yt, the neural connection weight We is updated by
11+1 = we + m,[A Feth + B xr + C- yc + D),
(1)
where A, C,D are meta- perameersss depending connections, and m, is the modulatory signal. In biological systems, one type of the most important modulatory neurons is dopamine neurons, which could integrate signals from the diverse areas of the nervous system (Watabe-U
et al. 2012) and affect the plasticity of certain neurons (Burrell & Sahley 2001 Birmingham & Tauck 2003). The effect of neural modulation on plastic ANNs is validated by (Soltoggio et al. (2008). Other works uses simpler feedback signal
[Frémaux & nerst 2016), or trainable constants (Pedersen & Risi 2021 as the modulation. The roactvvv
characteristic of dopamine neurons (Brzosko et al. 2015) also inspires the retroactive neuromodulated plasticity
(Miconi al. 2019). denoted by
W++1 = Wt + m,6++1s
c++1 = (1 (1- n)e + nragle
(2)
The plastic neural layers can be either in a feedforward layer Najarro & Risi 2020) or part of the recurrent
layer (Miconi et al. 2018 2019).
A challenge for plastic ANNs is the requirement for extensive meta-parameters. For instance, connections
with nh input neurons and ny output neurons require more than 4ngmy metaparameters (A, B, C, D), which is even more than the neural connections updated. Rules with fewer meta- parameeessssees such as retroactive neuromodulated plasticity have only been validated in cases where the neural connections are properly initialized (Miconi et al. 2018 2019
2.4 Implementing Genomics Bottleneck
Largo-scale deep neural networks typically lack robustness and generalizability (Goodfellow et al. 2014). A potential way to address this challenge to adapt large-scale neural network with relatively simple rules, following the genomics bottleneck in biology. Previous works utilizing genomics bottleneck include encoding
4

---

Published in Transactions on Machine Learning Research (09/2022)
forward, backward rules, and neural connections with number of tied smaller-scale genomics networks
[Koulakov et al. 2021), reducing plasticity rules (Pedersen & Risi 2021 encoding extensive neural network
parameters with pattern producing networks (or hyper--etworrs)) (tanley et al. 2009 Clune et al. 2009 Ha
ct al. 2016). and even representing plasticity rules with hyper-networks (Risi & Stanley 2010 2012). Among
those works, Evolving&&Merging (Pedersen & Risi, 202T) is more related to our decomposed plasticily. Based on similar motivations of reducing the learning rules, Evogving&Meerging
plasticity rules of different neural connections based on the similarity and re-evolve the tied rules. However, compared to the decomposed plasticity, it is less biologically plausible, more computationally expensive, and harder to scale up.
3 Algorithms
Problem Settings. We suppose agent has its behaviors dependent on both static components (including learning functionalities, and static neural connections) and adaptive components (including plastic neural connections and neuronal states, denoted by 0), which is initially decided by a group of meta-parsmettrrs (genotype, denoted by Φ, see Figure 2 Notice that in our cases, the adaptive components 0 are always started from scratch, and the meta- pettapparmeterss & only decide the static components; In other cases, the initialization of the adaptive components might also depend on Φ (Miconi et al. 2018 2019 In meta training or outer-loop learning, the parameeress & are optimized in set of training tasks T, E Tire and then used for initialization. In meta testing, & is evaluated a set of validation testing tasks Thalid Tist- For each step in meta-training and meta-testing, the individual life cycle of the agent (i.e., the inner loop) refers to the process through which it interacts with environments through observations é, and actions ag, where 0 is continually updated and change its behaviors. Specifically, in this paper, we mainly consider meta-reinforcemen learning problems, where the observation i, combines current observed states (or), previous-step action (a--1), and previous-step feedback (r--1) (Duan et al. 2016 Mishra et al. 2018 Wang al.) 2018) (In supervised learning is combines the features 2, and the previous-stey label Vt-1 (Santoro al. (0166)- The inner loop typically has two phases (Beaulieu and learns from the observations; It is latterly
al. 2020 : The agent first
tentatively explores the environment in meca--rrinningtrrann evaluated in the mela--raining--esliiie phase. In mela-lesling, similarly, the learned meta-parameterss are given wewttteetnggtiiinnn and seeaa-testing-tsstie in order. A life cycle marks the summarized length of an agent's 's inner-loop training and testing phases an agent.
Decomposed Plasticity. Considering a plastic layer with pre-synaplic (input) neurons states Rx and post-synaptic (output) neurons states y Ry,, we can rewrite Equation in the matrix form of
AW v)
W2 (10x) (y Wp),
(3)
where we use @ and & to represent "element-wise multiplication" and "outer product", respectively. Here AW is the updates for the noural connections, and WA. WB, We. WD E R" are the meta-parametton deciding the learning rules. In decomposed plasticity, we introduce a neuron-dependent decomposition of those peramears,,, e.g., WA = AAy @ V thus Equation g can be changed to
AW =m m [(VAy Cy) (VAr θx) + vBy @ (VBr Ox)
+ +Veo Oy) @ Vaz + @ &pll
(4)
where Va,x RT Vx.g Ryy, The decomposed plasticity rule contains all 4(n, + ny) parameters. For large ny and ny. it orders of magnitude smaller than the scale of the neural connections (nz x ny))
5

---

Published in Transactions on Machine Learning Research (09/2022)
Genotype (Mela-Paramettrrs +)
Decide
Learning Functionality
Static Neural Connections
Update
V/,A PhAy ... LA
W Wo, b,...
Static Components
Update 2
Update 2
Plastic Neural Connections
Update
Recursive Hidden States
W w LE
ht
Phenotype
Adaptive Components (0)
Agent Behavior
Figure 2: An overview of the modulated plastic RNNs. The static components include the learning rules and static neural connections decided by genotype, which will not change within the imner loop. The adaptive components include the plastic neural connections and hidden neuronal states continually shaped by learning rules and other components.
Modulated Plastic RNN. Given sequence of observations i mi byy plastic RNN updates the hidden states he with the following equation:
h441 = ttana (aal(W het
hy + w. + b),
(5)
f(Wohut1)
(6)
W(P)
A,4+1 = w h,t + AW(mi,,, h4t1)
(7)
W
= w(F + AW(ma, it h141).
(8)
6,1+1
We use superscript (p) to represent plastic neural connections. Unlike previous works of plastic RNN or plastic LSTM that implement plasticity only Wil)) we apply decomposed plasticity for both w(2) h,t and
w!()). This further reduces our meta perta-arammterss..
We consider two types of modulation: The pro-synaptic dopamine neuron generates the modulation by a non-plastic layer processing the pre-synaptic neuronal states and sensory inputs; The post-synaplic dopamine neuron integrates the signal from the post-synaptic neuronal states, follows:
Pre-synaptic Dopamine Neuron (PreDN): mi,s, m4,1 oWWmlee he))
(9)
Post-synaptic Dopamine Neuron (PostDN): ma,s my, oWmmet
(10)
The proposed plasticity can be implemented in both recurrent NNs and forward-only NNs. An overview of plastic RNN is shown in Figure 2 Parameters that decide learning functionality (plasticity rules) and static neural connections are regarded meta-parameterss (Φ). The variables contained by plastic neural connections and recursive hidden states are regarded adaptive components in the phenotype (0). The adaption of the model achieved through updating the hidden states (Update 1. or Equation 5) and updating the plastic neural connections (Update (2), or Equation
Figure 2 can fit various model-based and plasticity-based
learning algorithms. Based on this setting, the genomic bottleneck indicates |0] > (g)-
Outer-Loop Evolution. Given task T, E T, by continually exccuting the inner loop including meta-training- training and meta-traine aruininggtestinn we acquire the fitness of the genotype imeta-parametent 6 at the end of its life cycle, denoted as Fil(s, Tj). The genotype can be updated using evolution sttegiess (ES) approach,

---

Published in Transactions on Machine Learning Research (09/2022)
e.g. Rechenberg
1973
Salimans et al. (017))
Φ k+ = Φe + a-
Fit (Tt - -60)
(11)
The superscript k and i represent the kth generation and the ith individual in that generation. The subscript T marks the duration of an individual life cycle. The population gi+l. is sampled around d++1 with the covariance matrix = L"I. For high-dimensional parameessss selecting proper hyperparameters (e.g. the covariance matrix C*) is nontrivial. Improper selection could end up in inefficient optimization and local optimums. To address this challenge, CMA-ES greatly improves optimization efficiency and robustness by automatically adapting the covariance matrix using the evolution path (Hansen & Ostermeier 2001
However, it comes at the price of increasing the per-generation computational complexity from θ O to
@(())) P), which is infeasible for large-scale ANNs. Alternatively, we use seq- Bog-CMA-ES (Ros & Hansen 2008
where the covariance matrix degenerates to O (l)) by preserving the diagonal elements of C' only, which is affordable and empirically more efficient compared to ES.
The proposed model can also be optimized with a gradient- based optimizer following Miconi et al. (2018). In cases of supervised training, ES is typically less efficient than gradient-based optimization. However, for meta-RL with sparse rewards, ES could be more efficient than gradient-based optimizers (Salimans al.) 2017). Moreover, considering the models obeying the genomics bottleneck (10) > (Øl) and long life cycles (T >> 1), ESStype optimizers could be potentially more economical choice in both CPU/GPU memory and computation consumption.
The decomposed plasticity is inspired by neuronal differentiation (Morrison, 2001 in biological systems. Instead of assuming each neural connection has unique learning rules, it might. be more biologically plausible to propose that the learning rules are related to pre-synaptic and post-synaptic neurons separately. Although there other ways to make the learning rules more compact, e.g., hyper-networks (Risi & Stanley 2012). and Fvolving&&Merging (Pedersen & Risi 2021), the decomposed plasticity is relatively straightforward and casier to implement by tensor operations. The effect of neuromodulations on long-term memory and learning in biological systems is well supported by experiments (Schultz) 1997 Shohamy & Adcock 2010). Typical neural modulated plastic ANNs assign a unique modulator for a neural connection or a neuron (Soltoggio et al., 2008 Miconi et al.) 2019), which is inconsistent with the fact that dopamine neurons are much fewer than the other neurons in biological systems (German al. 1983) Our experiments show that assigning only single dopamine neuron for entire plastic layer can yield satisfying performance, saving many meta-patt petta-parmmters.
Biological Plausibility.
Summary. We formalize the inner-loop learning and meta-training process in Algorithm I and Algorithm 2 The framework g are also applicable to the other model-based and plasticity-basel meta-learning. Note that in meta-RL, a life cycle includes multiple episodes. The agent must gain experience from the earlier episodes tontta-training-tee -rainin-t-ntgngnnnngg))) to perform well in later episodes mmeta-training-tsstee the variable w, to tune the importance of each episode to assess fitness. An explanation of these training settings can be found in the Appendix A.2
Therefore, we use
4 Experiments
4.1 Experiment Settings
We validate the proposed method in Meta.azeeDD (Wang 2021); ; an open-source maze simulator that can generate maze architectures, start positions, and goals at random. The observation i, is composed of three parts: the 3 x 3 grids observed (ot), the previous-step action (a,-1), and the previous-step reward (r.-1). The maze structures, their positions, and the goals are hidden from the agents. Our settings have a 15-dimension input and a 5-dimension output in all. The output action includes 4 dimensions deciding the probability of taking a step in its four directions (east, west, south, north) and one additional dimension deciding whether it will take softmax sampling policy or an argmax policy. On top of the plastic layers, we add a non non-plastic
source code available at https::/ bttpps//gSthhb... http:////gtttbb..ca//etttiiiiiie http:///w......aaaaaaat//////aaa

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm Inner-Loop Learning
1: Input 6, T.
2: for T' E T do
3: Reset to scratch.
4: for = 0, 2,, ... until the end of a life cycle do for =0 2,...... until the end of an episode do
Observe or, is (ot, 02-1, Update 0 using & and Equation Execute de, receive rt-
acquire output az
8:
RS -. Fit(o,T) -2 w2 R..
10:
11: Return Fit(OGene: T) Erer T).
Algorithm Meta- Training and
1: Pre-sample Tvalid and Test-
2: Randomly sample initial genotypes ou i= I,....9 i,...
3: for Generations 0, 1, until convergence do Randomly sample training tasks Tua.
for 9....... do Acquire average fitness Fil(d"., Tera) by calling Algorithm
7: Apply Soq-CMA-ES to acquire the next generation centroid @++1 and population 6h-1,
8: Acquire Fil Tvatia) by Algorithm record o² acquiring the best fitness.
Return 6², Fit(Ø", Tia).
output layer that processes the hidden units to 5-dimensional output. We found that the performance was lower the output layer was plastic, and since it contains relatively few parameters, keeping it static does not violate the genomics bottleneck. The agents acquire the reward of 1.0 by reaching the goal and -0.01 in other cases. Each episode terminates when reaching the goal, or at the maximum of 200 steps. life cycle has totally episodes.
(a) 9 x
(b) 15 15
(c) 21 x
Figure Cases of mazes of different scales including the observed states (o.) and the god view. The red squares mark the current positions of the agents; The green squares mark the goals.
For mota-training, each generation includes g 360 genotypes evaluated on |Tral 12 tasks. The genotypes are distributed to 360 CPUs execute the inner loops. The variance of the noises in Seq-CMA-ES is initially set to be 0.01. Every 100 generations add validating phase by evaluating the current genotype in [Tratial = 1024 (validating tasks). By reducing g |Tiral we observed obvious drop in performances. Scaling those settings will stabilize the training but lead to increase of time and computation costs. Meta training goes for at least. 15,000 generations, among which we pick those with the highest validating scores for meta-testing.
The testing tasks include mazes (Figure (a)), 15 x 15 mazes (Figure (b)), and 21 21 mazes (Figure (c)) sampled in advance. There are |Taal 2048 tasks for each level of mazes. We run meta-training of the

---

Published in Transactions on Machine Learning Research (09/2022)
compared method twice, from each of which we pick those with the highest fitness in validating tasks. We report the meta-testing results on the two groups of selected meta-parameters. The convergence curves of meta-training is left to Appendix A.3.22
We inchude the following methods for comparison:
DNN: Evolving the parameter of a forward-only NN with two hidden fully connected layers (both with hidden size of 64) and one output layer. Two different settings are applicd: In DNN, we only use the current state as input; In Mela-DNN, we concatenate the state and the previous-step action and feedback as input.
Meta-RNN: Employing RNN to encode the observation sequence, the parameters RNN are treated as meta-paree pota-paramettes. We evaluate the hidden sizes of (MNta-RNN-XS), 16 (Meta-RNN-S), and 64 (Meta-RNN).
Meta-LSTM:
Employing LSTM to encode the observation sequence, the parameters of LSTM are treated meta-part petaa-arrametes. We evaluate the hidden sizes of 8 (Meta-LSTM-XSS) 16 (Meta-LSTM-S)) and 64 (Meta-LSTM).
Applying the aABCD plasticity rule (Equation 3 to the PRNN. We also evaluate the hidden sizes of 8 (PRNN-XS), 16, (PRNN-S), and 64 (PRNN).
PRNN:
DecPDNN: Applying the decomposed plasticity to the first two layers of Meta-DNN.
DccPRNN: Applying the decomposed plasticity to PRNN (Equation 4 We evaluate the hidden sizes of 32 (DecPRNN-S) and 64 (DecPRNN).
Applying the retroactive neuromodulated plasticity (Equation 2 to PRNN, but only to the recursive connections (W(?), the input connections s,t are not included. Following Backpropamine (Miconi et al. 2019), the initial parameters of the connection weights are not from scratch, but decided by meta- eettaparameters..
Retroactive PRNN
Retroactive(Randorn) PRNN: Start the plastic neural connections from scratch in Miconi et al. (2019).
Evelving&&Merging: Implementing evolving and merging Yaman et al. 2021 PRNN, where we start training with the aABCD rules and reduce those rules using K-Means clustering and re-train the tied rules. But unlike the original proposal that evolves and merges multiple times, merge and re-evolve for only one time, reducing the 20224 rules to 1144 rules, the same as the scale of meta-parameter in DecPRNN.
Notice that the plastic neural networks may be further combined with different types of neural modulation, includ- ing non-modulation, PreDN (Equation D and PostDN
DccPENNNP DePAANNRRaDN)
Reboactrre
(Equation 10p. We compare different modulations in DecPRNN. For all the other plasticity-based methods, we apply PostDN, which is proven to state-of tteatrr To emphasize the impact of different meta- portapparmeteess and adaptive components, we show the number of meta- parameters in genotypes and the number of the variables in adaptive components of all the compared methods in Figure
Adaplive Components
DePRENN S(NKIDK)
Information
PRNN-S
MctsLSTMMLL
PRNN-ASEPPODNI
Bn MST
Mca-INN
Mctt-RNN-
Mcb-15TM55
Meis-RMN-a
Meto-Purameters
Figure 4: Over-viewing the number variables in static and adaptive components of various meth- ods.
4.2
Experiment Results
4.2.1 Best-Rollout Performances
We first show the best-rollout performance of the compared methods, which calculated by picking the rollout with the highest average rewards among their 8-rollout life cycles (Figure 5). Besides the rewards,
9

---

Published in Transactions on Machine Learning Research (09/2022)
also show the failure rate, which marks the ratio of mazes among Tist where the agent fails to reach the goal within 200 steps. The best-rollout performances are regarded as an indicator of the agents" learning potential. We evaluated all the compared methods in 9 x 9 and 15 x 15 mazes and only the most representative and competitive mothods in 21 x 21. Here we omit all the results of DNN and Mcta-DNN (sce Appendix A.3.1 as showing those bars might cover up the other differences.
Plasticity-based vs. Model-based Learners. Generally, we show that plasticity- learners perform better than model-based learners (Meta-RNN, Meta-LSTM), especially for more complex cases of larger mazes. For instance, in 9 x 9 mazes, the best model-based learner (Meta-LSTM) is slightly better than the best plasticity-based learner (DecPRNN(PostDN)) However, they underperform plasticity-based learners in 15 x 15 mazes and 21 x 21 mazes. Among model-based learners, Meta-LSTMMs are better than Meta-RNNs, which is also more obvious in complex cases. For both Meta-LSTMs and -NNN, a hidden size below 16 yields a clear decline the performance, while a larger scale generally results in better performances. Since larger mazes mcan a longer life cycle (sce Appendix (.3.3),, our results show that plasticity- learners are more powerful in long-term memorization. Notice that model-based learners generally have smaller-scale adaptive components (see Figure 4 or Table 1 but larger-scale meta- paraameessssesss compared with plasticity-based learners. We argue that the scale of adaptive components also plays an important role in memorization.
Plasticity Rules Comparison. We validate that retroactive PRNN with PostDN can improve the Meta- RNNs, which is consistent with Miconi et al. (2019). However, when the connection weights are randomly initialized, retroactive PRNN performs much poorer than the other plasticity rules. It is not surprising as the retroactive plasticity have the fewest learning rules. In contrast, the aABCD rule (PRNN), Evolving& Evolving&Merggnng Merging, and decomposed plasticity rule can do reasonably well. For standard hidden size (64), decomposed plasticity can do the best among all plasticity rules, even though theoretically, the upper bound of aABCD rule should be higher. It might be related to the meta-trainin training process since a larger meta meta-parameter space raises higher challenge for ES. Another possible cause can be larger overfitting specific lask distributtins. The Soooving&&erginnn can not do as well as decomposed plasticily in 9 and 15x x 15 mazes. Considering has the same scale of meta-parameters as decomposed plasticity, and it requires at least two stages of training (meta-train PRNN in the first stage, then switch to merged rules in the second stage), the Evolving&:Merging seems less attractive than decomposed plasticity. Another surprising fact. is that PRNN-S with only 16 hidden units can do very well in 9 x 9 and 15 x 15 mazes and reasonably well in 21> x 21 mazes. It has slightly larger-scale meta- mara-eerssesers compared with DecPRNN (2119 vs. 1347). The scale of its adaptive components is much smaller than PRNN or DecPRNN (512 vs. 5120) but still much larger than model-based learners. It seems that the reduction of meta- mera-parameters plays a crucial role here However, as we tested DecPRNN-S (10)= = 1536, |ő| = =707), and even smaller PRNN-XS (10) 192, |Ó| 809), the performances is significantly degraded. Based on these findings and considering retroactive PRNN's performance and model-based learners' performance, we may conclude that relatively smaller-scale meta-parameters and larger-scale adaptive variables are helpful but with boundaries.
requires larger |Tiral to avoid
Effect of Neural Modulations. The results also show that PostDN > PreDN > non-modulation for DecPRNN. The difference between modulated/ modulatde DecPRNN is nontrivial, showing that employing even very few neural modulators can be crucial. The success of PostDN over PreDN suggests that the modulation signal is better to "backpropagate" than to integrate sensory inputs. Inspired by the analogy between rewards and neural modulations (Schultz) 1997), this seems to support the functionality of intrinsic reward and intrinsic motivation.
4.2.2 Inner-Loop Visualization
We plot the per-rollout rewards and failure rates within the agent's life cycles of several competitive methods in Figure 6. The performances of model-based and plasticity-based learners deviate over time: The model-based learners typically start at a good level but slop improving at 3rd rollouts and suffer from a relatively low ceiling. In contrast, the plasticity-based learner starts from scratch but keeps improving through its life cycle to eventually higher level. Also, we observe the different levels of improvement in different mazes. For simpler 9 x mazes, most methods stop improving after their 3rd rollouts. For complex 21 x 21 mazes, we can see signs of improvement even at the last. rollouts regarding the best. plasticity-based learners.
10

---

Published in Transactions on Machine Learning Research (09/2022)
Mares)
SteRhy
Mon-RN
LSTM
Aleo Lomee
Enstringk
Io HePRNNEPwOR
(a) x best-rollout rewards
Best Rollor Failure Rato (Ir Mazes)
Hareaetis
MeteLoraks
Mote Hote
MrostsyMtt
Par Part-araadtiy
Fiehiight Everiopi-MMrgla Merging
DeTRANPaiNg
Bee Pestar
Bic Pexttiyy
Mide
(b) 9 9 best-rollout failure rates
Bes Rellut Rese Moss)
Retroaetvv
Meieet 20
Mose Aod
Hinetousd
Meded Aotes
PROVe Kuroanng
Eieitiggt Merging
Bee Nerattnn
Mede
(c) 13 x 15 best-rollout rewards
Bed Rellou Failur: Rax (15115 Mazcc)
Retrentive
No
Me
Mena La
PROS KyAwiNy
Fowhttadd Merglor
HerPeee
de Siveer
ReePae Syaatth))
15 best-rollout failure rates
(21s21
(21x21
Met-dayM
MoseRhy
Passaarwah
Meted
(e) 21 x 21 best-rollout rewards
(f) 21 21 best-rollout failure rates
Figure 5: Best-rollout performances (including the rewards and failure rates) compared methods. Methods of the same genre with different scales or different modulations are marked with the same color, while different. genres of methods marked with different colors.
11

---

Published in Transactions on Machine Learning Research (09/2022)
-22
-44-
-14-
-04
leto LTM
-34
STV
-04
bechuaehoroon
(a) 9 X 9 rewards
(b) 15 x 15 rewards
(c) 21 x 21 rewards
574
Mets 2904
K1%
216
47C
15%-
PROS PPODE
40%
PRA PENSSRE
37%
305
376
345
3/5
37k
206
(d) 9 9 failure rate
(e) 15 x 15 failure rate
(1) 21 x 21 failure rate
Figure 6: Per-rollout performances of selected methods for 8-rollouts life cycles.
Visualizing Adaptive Components. We present the development of the adaptive components within the agents' life cycles, including hidden states (he, and the plastic neural connection weights (W/P)). We first sample 3 different tasks. For each task, we rum the life cycle of different methods twice, which yields 6 trajectories of adaptive variables for each method. Since those tensors are in relatively high dimensions, run t-SNE visualization (Van der Maaten & Hinton 2008} to map them to a 2-D space and show their temporal traces in Figure 7 In Figure 7(a), we also show the correspondence of the traces to the actual trajectories in the maze. Comparing Figure| ()) and (b), we see that the plastic neural connection weights behave differently from hidden states: The connection weights smoothly migrate toward certain directions across the agent's life, the traces of which are correlated with the hidden task configuration (Figure 7 (.,c.d,e.f)): In contrast, the hidden states vibrate fast without clear sign of long-term orientation (Figure 7b, f)). It could possibly be used to explain why plasticity-based learners do better than model-based learners in long-horize horizon tasks and why combining recursion and plasticity can yield better performances: PRNN and DecPRNN can keep the long-range information in connection weights and leave the short-term information to hidden states, while recursion-only learnors must keep all those memorics in the hidden states, where they might interfere with each other. In Figure 7(a) we also mark the possible "sweet spots" of the plastic neural connections for the three tasks, where the agents find the optimal solutions and the connection weights are at convergence. In other casos, especially non-modulated DecPRNN (Figure (c)) and recursion-free DecPDNN(PostDDN) (Figure d)), those "sweet spots" can be hardly observed, suggesting that their learning processes might be noisier.
To quantitatively investigate the learning process of plastic neural connections and hidden states, we introduce two measures related to short-term and long-term behaviors, respectively. To measure the short-term vibration of the lcarning process of variable (which can be cither connection weights w:(P) or hidden states he), we introduce aws(xr)) = E[ - LrI,, where & is the moving average of x with a sliding window [L m, L + m] (here we use m = 7, window size of 15), |- denotes the L2-norm. To measure the long-term migration of Fe, we use (x,)
E[ - Rol]. We show awb and dmig against the steps of the agent's life cycle in Figure B(a) and (b) from the statistics of testing on the full 2048 15 15 mazes. Since 90% of the life cycles are within 1000 steps (see Appendix (A3.3), we show only the statistics within 1000 steps. We can see more
12

---

Published in Transactions on Machine Learning Research (09/2022)
qpar 64-
(a) W.T De
(b) hu DecPRNN(PostDN]
(c) W..7 DecPRNN
M
28-
9-
-4--
-90
-390
-200
-100
200
306
460
-15
-50
164
-19
-75
300
(d) (PPP) DecPDNN
(e) W-(P) PRNN(PostDN)
(1) W.P)
Exolving&
h.t
h.
(g) he- Meta-LSTM
(h) ct (cell states) Meta-LSTM
(i) Meta-RNN
Figure 7: t-SNE visualization of the transformation of the connection weights (W(?) and hidden states (hr) in varions methods. Each figure represents the traces of particular components in a specific method. Each method is demonstrated
different tasks twice, yielding traces, as shown in (a), where each color (brown,
yellow, black) corresponds a unique task. The red * in each trace marks the end of rollout. There are *s each trace corresponding rollouts each life cycle.
clearly that the plastic neural connections migrate much more in the long run (dmig) compared with the hidden states, which are nearly flat within the life cycle. In contrast, the hidden states have more significant. short-term vibrations (a,s) than the plastic connection weights. Moreover, DecPDNN has a larger-scale oea (W(")) and dmig(W((")) in the later of the life cycle, which might be attributed to lack of hidden states to contain short-term memories. The non-modulated DecPRNN also has a larger scale in both a awa (W(P)) and dimig (W((), validating that the neural modulation plays crucial role in regulating the learning of neural connections.
Analyses on Explorations. It is well known that reinforcement learning depends on exploration and exploitation simultancously. We are then interested in investigating whether the inner loop has learned to balance exploration and exploitation. We visualize the exploration of the agents by using the coverage
13

---

Published in Transactions on Machine Learning Research (09/2022)
30
Moo 30.0
STRA
Neto (STM
Di
DacP Alf
385
(a)
(b)
Figure 8: Quantitative evaluation of the long-term and short-term change of the adaptive variables evaluated in 15 x mazes. (a) dmig (W(()) or (hey(hs) measuring the long-term migration. (b) oww((, or owa measuring the short-term vibration.
6%
245
400
PMB-3TM
116
41%
37%
ribent rolbur mileet
35%
99819 mlee)
mier role33
mibe)
miku mibe
wilee miaes miou
(a) x per-rollout Exploration
(b) 15 per-rollout Exploration
(c) 21 21 per-rollout exploration
V/4
115
700
ESh
576
610
Mr 20
286
309
1o0ms 100611
19863 moot2 robitt 19005 robis ml0)
25h
9/819 slor) mlbe2 19022 miooc
relk1 webe mile
1eb29 mioe) 5802 mBE3 midc sieas miees slse)
9 accumulated exploration
(c) 15 accumulated exploration
(1) 21 x 21 accumulated exploration
Figure 9: The coverage rate of the visited locations showing the exploration of different agents in various mazes.
rate of the mazes, which the unique locations the agent visited divided by all the reachable locations that maze. We plot both the per-rollout coverage rate, and the accumulated coverage rate (by counting the uniquely visited locations since the beginning of its life cycle) of meta-testing in Figure There are several interesting points worth mentioning. First, learn to explore more at the beginning cesta-testing-trannii stage) and gradually reduce the exploration. Second, some plasticity-based methods, especially those obeying the genomics bottleneck (Evolving&Ment and DeePRNN(PostDN) are more deterministic their first rollouts but then explore more the 2nd and 3rd rollout. It also consistent with Figure a()) where we show the traces of neural connections DecPRNN(PostDN) for different tasks highly overlapped the very beginning. A tentative explanation this strange behavior is that the agent is experiencing some warm-up stage" before actually learning about.
that all model-based and plasticity-based learners
14

---

Published in Transactions on Machine Learning Research (09/2022)
the maze since its initial neural connections can not support any complex behavior. Third, we find that by summing up the per-rollout explorations, the plasticity-based learners do not have the largest exploration every rollout but have the highest accumulated exploration. It could possibly mean that plasticity-based learners are more efficient in designing cross-cpisode exploration strategics.
5 Limitations and Future Prospects
Currently, typical large-scale deep models work with mostly static components and few adaptive components. They have been powerful in pre-defined tasks but suffered from high customization costs and the inability to generalize to various sconarios with little human effort. In this paper, we suggest designing models with relatively larger-scale adaptive components with inspiration from plasticity and the genomics bottleneck. Those models are expected to be not necessarily capable of everything initially, but capable of learning accomplish very complex tasks by interacting with the environments and learning automatically.
There are several limitations to this work. First, although our environments have probably been the most challenging 2D maze tasks that have been addressed until now, it is still far too simple compared with many tasks in reality. It also needs to be validated in image-related studies. We believe that building diverse and close-to-reality simulators is essential for the future development of AI. Although a lot effort is devoted to simulators where agents can learn general locomotion skills Dosovitskiy et al., 2017 Yu et al. 2020), natural language undorstanding (Hermann al. 2017 Yu et al.) 2018 Sedevalier-Boiveee et al.; 2018): natural language generation (Havrylov & Titov 2017). and even the const.ruction of artifacts (Grbic el al. 2021
there is still a long way to go, given the huge gap between simulation and reality. Second, we believe that our outer-loop optimizer (the Seq-CMA-ES) is good choice at the current stage, but it still severely limits the model design. Plasticity rules will be more prospective if combined with more sophisticated models, but the meta-training turns out to be the bottleneck. Meanwhile, the genomics bottleneck can alleviate this problem and thus less burden on meta-training, but it is not without limits.
since it means fewer mhea-pparaette To build valuable AIs for reality might still cost millions of meta-paramettrss calculated on inner loops of millions of steps and maintaining hundreds of millions of adaptive variables. Third, we have mainly tested our settings in meta-learning framework. An essential setting will be enabling single agent to continually adapt to nonstationary tasks during its lifetime without forgetting those old ones (Beaulieu et al. 2020 This should be direction for future investigations.
References
Bruno B Averbeck and Vincent D Costa. Motivational neural circuits underlying reinforcement learning. Nature Neuroscience, 2054):505-51, 2017.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In ECAI 2020, pp. 992-1001. IOS Press, 2020.
John T Birmingham and David L Tauck. Neuromodulation in invertebrate sensory systems: from biophysics to behavior. Journal of erperimental biology, 206(30):3541 (544)33341-3546 -3546, 2003.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020.
arXiv preprint 2rXiv:2005.1410
Zuzanna Brzosko, Wolfram Schultz, and Ole Paulsen. Retroactive modulation of spike timing dependent plasticity by dopamine. elife, 4:e09685, 2015.
Brian D Burrell and Christie L. Sahley. Learning in simple systems. Current opinion in neurobiology, 11(6): 757 764, 2001.
Maxime Chevalier- Bhoislier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. Xixio preprint ar 110.00812. 2018.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Jeff Clune, Benjamin E Beckmann, Charles Ofria, and Robert T Pennock. Evolving coordinated quadruped gaits with the hyperneat generative encoding. In 2009 iEFE congress on ecolulionary compulation, pp- 2764 2771. IEEE, 2009.
Alexcy Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 16. PMLR, 2017.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning. arXiv preprinl arXiv:1611.00 011166111.779, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic mela-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1135. PMLR, 2017.
Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike- muike-timin-ddeeedder plasticity, and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
DC German, DS Schlusselberg, and DJ Woodward. Throe-dimensional computer reconstruction of midbrain dopaminergic neuronal populations: from mouse to man. Journal of neural transmission, 57(4):243 254, 1983.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint orXiv:1412.6777 2014.
Djordje Grbic, Rasmus Berg Palm, Elias Najarro, Claire Glanois, and Sebastian Risi. Evocraft: A new challenge for open-endedness. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar); pp. 320-340. Springer, 2021.
David Ha, Andrew Dai, and Quoe V Lc. Hypernetworks. arXiv preprint arXiv:1609009106, 2016.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, ((2):159-13) 195, 2001.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Advances in neural information processing systems, 30, 2017.
Donald Olding Hebb. The organization of behavior; a neuropsycholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiu:1706. 06551, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)):1335- 1997.
Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning, Artificial Intelligence Review, (466):44334444 2021.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, pp. 0. Lille, 2015.
Alexei Koulakov, Sergey Shuvaev, and Authony Zador. Encoding innate ability through a genomic bottleneck. bioRriu, 2021.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 (7553))): 444, 2015.
Ke Li and Jitendra Malik. Learning to optimize. arxiv preprinl XXii:1666.0088: 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Diffcrentiable architecture search. In International
Conference Learning Representations, 2018.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Thomas Miconi, Kenneth Stanley, and Jeff Chune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, pp- 3559-336 PMLR, 2018.
Thomas Miconi, Aditya Rawal, Jeff Chune, and Kenneth O Stanley, Backpropamine: training self melf-mooiiying neural networks with differentiable neuromodulated plasticity. In International Conference on Learning Representations, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
Sean Morrison. Neuronal differentiation: proneural genes inhibit gliogenesis. Current Biology, 11(9): R349-R351, 2001.
Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks. arXiv preprint Sxfi::007.026880 2020.
Yacl Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3))1399-14 154, 2009.
Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070 2020.
Joachim Winther Pedersen and Sebastian Risi. Evolving and merging hebbian learning rules: increasing gen- eralization by decreasing the number of rules. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 892-900, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the anai conference artificial intelligence, volume 33, pp. 4780-4789, 2019.
Ingo Rechenberg. Evolutionsstrategie: Optimicrung technischer systeme nach prinzipien der biologischen evolution, frommann holzboo... stuttgart Germany, 1973.
Sebastian Risi and Kenneth O Stanley. Indirectly encoding neural plasticity pattern of local rules. In International Conference on Simulation of Adaptive Behavior, pp. 533-543. Springer, 2010.
Sebastian Risi and Kenneth O Stanley, A unified approach to evolving plasticity and neural geometry, In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2012.
Raymond Ros and Nikolaus Hansen. A simple modification in cma-es achieving linear time and space complexity. In International conference on parallel problem solving from nature, pp. 296-305. Springer, 2008.
Tim Salimans, Jonathan Ho, Xi Chen, and Tlya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. CoRR, abs/1703.0386, 2017. URL http:p//aaaiii..rr//////////..
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-angmented neural networks. In International conference on machine learning, pp. 1842-1850. PMLR, 2016.
Wolfram Schultz. Dopamine neurons and their role in reward mechanisms. Current opinion in neurobiology, 7(2):191-197) 197, 1997.
Daphna Shohamy and R Alison Adcock. Dopamine and adaptive memory. Trends in cognitive sciences, 14 (10):464 472, 2010.
Andrea Soltoggio, John Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Florcano. Evolutionary advantages of neuromodulated plasticity in dynamic, soward-based scenarios. In Proceedings of the 11th international conference artificial life (Alife XT), pp. 189-576. MIT Press, 2008.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Andrea Soltoggio, Kenneth O Stanley, and Sebastian Risi. Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. Neural Networks,, 108:48 67, 2018.
Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao Tang. Fs-maml: Simple hessian-free meta learning. In International Conference Learning Representations, 2019.
Kenneth O Stanley, David B D' D'Ambosso,, and Jason Gauci. A hypercube- based encoding for evolving largo-scale neural networks. Artificial life, 15(2):185-212, 2009.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Fan Wang. MetaMaze: Efficient 3D Navigation Simulator Benchmarking Meta-learning. https://github oatp://auu.aadd////aaeee//eaatte
2021.
Jane Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature neuroscience, 21(6):860-868, 2018.
Mitsuko Watabe-Uchin Uchida, Lisa Zhu, Sachie K Ogawa, Archana Vamanrao, and Naoshige Uchida. Whole-brain mapping of direct inputs to midbrain dopamine neurons. Newron, 74(5):858 873, 2012.
Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, Matt Coler, George Fletcher, and Mykola Pechenizkiy. Evolving plasticily for autonomous learning under changing environmental conditions. Evolutionary computation, 29(3):391-414, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in a 2d world. In International Conference Learning Representations, 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):1-7, 2019.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
18

---

Published in Transactions on Machine Learning Research (09/2022)
A Appendix
A.1 Details of The Model Structures
All the methods compared have a static output layer mapping the hidden units to 5-dimension actions. The difference lies the rest part that maps the inputs to the hidden units. We list the meta-parameterrs and adaptive components of all the methods compared in Table
Table 1: Model Structures of Compared Methods
Hidden Units
Adaptive Variables ((0)
Meta Parameters (lol)
DNN
64+664
5,125
Meta-DNN
64+64 64
5,509
Meta-RNNNXS
8
237
Meta-RNN-S
16
597
Meta-RNN
64
64
5,445
Meta-RNN-J
128
128
19,077
Meta-LSTM-XS
8
16
813
Mcta-LSTM-S
16
32
2,133
Meta-LSTM
64
128
20,805
Meta-LSTM-L
128
256
74,373
Evolving&&Merging
64
5,120
1,347
Retroactive
64
4,160
5577
Retroactive (Randomm)
64
4,160
1481
PRNN-XS(POstDDN)
8
192
809
PRNN-S(PostDN)
16
512
2,119
PRNN(PostDNN
64
5,120
20,743
DecPDNN(PostDN))
64
5,056
1,411
DecPRNN
5,120
1,217
DOPPRNN
64
5,120
1,379
DeePRNN-S(PostDDN
32
1,536
707
(e(PRNN(POsDDN)
64
5,120
1,347
A.2 Meta- Meta-Trainin Training Settings
To calculate the fitness an agent regarding its life cycle of rollouts, we apply Fil(d, 7) = =,, mm R.g with w, be
0 x<2 2
W12 =
(-80(--6-1) else
0.5
000 Sw--ar----Srrrr
2.8
450
Meta-training training
Meta-training sestting-tsttigggg
Figure 10: "stiff" and "smooth" training and testing phases in meta-learning.
Here 8 denotes the episodes in the life cycle. This formula is slightly different from canonical meta- supervised-learniig settings, where w, 0 for mete-trainnng--rrainnng (or support set) and wa 1 for
19

---

Published in Transactions on Machine Learning Research (09/2022)
mntta-traiinngtesttnng (or query set). The canonical training and testing phase settings guide the meta- training to improve the performance on the query set only. Although this "stiff" setting can be extended to meta-RL Finu et al. (2017), it is actually more difficult to decide the boundary between meta-training- traiiiing-training and meta-training- testing in meta-RL since we want the agent to gradually improve its performance over its iif. Thus, we propose this "smooth" selting for the training and lesting phases in the inner loop (Figure 10 In our cases, we found it more effective than "stiff" settings. For the outer-loop optimizer -Mn-)), we used an initial step size of 0.01, and the covariance C I for all the compared methods.
A.3 Additional Results
A.3.1
Performances of DNN and Meta-DNN
Table 2:
Best-Rollout performance (rewards) of DNN and Meta-DNN, along the performances of random
policy and oracles.
Maze 9 x 9
Maze 15 x 15
Maze 21 x 21
DNN
-0.860 + 0.038
-1.603 + 0.025
Meta-DNN
0.480 = 0.070
-0.506 = 0.018
Random 11.308 + 0.031
-1.934 + 0.010
-1.976 + 0.005
Oracle
0.908
0.820
0.781
A.3.2 Convergences of Meta- Trainira
-01
1969
-22
Fou Ae11
-35
-1.
2506
50K
100KC 3259 15900 17560 2000
2004
4003
1000
10000
1290 3400
3989
20099
50000
1000
50000
(a) 9 x 9
(b) 15 15
(c) 21 21
-15
-10
Foxic'M
Poxic'M
-15
2506
56k
750
100K
3250
15000
17560
280
2003
4000
0009
0000
100K
1200
34269
1000
1000
2000
33000
48000
(d) 9 x
(e) 15 x
(f) 21 x 21
We present the average fitness score evaluated on validating tasks (Teaki) against the evolved generations of all the compared methods in Figure g For clarity, we group modcl-based and plasticity-based learn- ers into 6 groups and show each group independently to avoid confusion. For comparison, we also add DecPRNN(PostDN) in each group. Notice that we follow curriculum learning process for the meta-training.
20

---

Published in Transactions on Machine Learning Research (09/2022)
man 247
-335
24-
-445
30C
403
6XC
100 1800 1200
3489 1900
990
1500
3900
12500
20009
37598
2900
2500
3000
750
1000
130K
2500
2000
(g) 9 X
9
(h) 15 15
(1) 21
04
825
130
04
-12
-150
-42
18
-190
-126
12999
paressare
per premsore
9
(k) 15 15
03-
19
fezzaciie
-30
perestaree
geseraion
(m) 15 15
-05
-13
Decee1
-14
-12
-10
Ded SAMPPOOS
(n) 9
(o)
Figure 9: Plotting the mean and variance of the fitness score in validating tasks.
We train the learners on x 9 mazes first and apply a warm start in 15 x 15, and 21 x 21 sequentially to reduce the meta-training cost. Also, is not fair to compare DecPRNN(PostDN) with Evolving&MMeggnn directly since we must start from well-trained PRNN(PostDNN model Evolving&&Merging Also, we occasionally reset the covariance avoid the local optimum. Generally, we found that DedPRNN(PostDN) is not only of the bests in performances but also converges fastly thanks to fewer meta-parameters. Model-based
21

---

Published in Transactions on Machine Learning Research (09/2022)
learners such as Meta-LSTM generally harder to reach convergence. We sometimes models longer than the other models in case the models have not converged.
+
EA
(a)
(b)
Figure 10: Example trajectories of the DecPRNN (PostDN) agents in each of the rollouts in different mazes (9 x 9, 15 x 15, and 21 x 21). The red squares mark the start position, the green square marks the goal, and the red lines denote the agents' trajectories.
A.3.3 Lengths of the Life Cycles
The life cycle the agents lasts for eight episodes. Each episode has at most 200 steps. The agent takes 1600 steps most, but it shortened as the agent performs better. The steps of life cycle vary for different learners and different meta-training stages. In Figure [1 we show the changes in the average life cycle length for DRcP DPostDN) learners versus evolved generations, well the distribution of the life cycle length.
22

---

Published in Transactions on Machine Learning Research (09/2022)
120
Dirrbaaie lss cyde langh (DeftiXNNhaadi))
23:21
300
130
200
300
400
300
600
500
500
1230
3153
1330
1300
3400
1500
1900
Sizps
(a)
(b)
Figure 11: Lengths of the life cycles of DecPRNN(PostDDN) agent in Tealid- (a) Changes in the average length in meta-training, (b) Distribution of the lengths at convergence.
A.3.4 Case Demonstration
We randomly sample several mazes and show the actual trajectory of the agents of each rollout in Figure I0 We demonstrate cases where the agents find or do not find the global optimum. This could give intuitive interpretation of inner-loop learning. We can also observe the behaviors of exploilation to maintain high performance in the current rollout and exploration to reveal better routes for the following rollouts. For instance, the agent tends to explore new directions in case its previous rollout is not successful enough and takes the shortcut discovered in the previous rollout.
2Visualization DecPRNN(PostDN) can be found httppp///wwayyyuuuaccccoooeeeetr
23


Reviews:
[{"review_id": "AXYJ6RTfR2", "reviewer": "Reviewer_hAdL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The authors show that simpler models (i.e. those with many fewer parameters) do reasonably well, if not better, on the Maze task studied here.\n\nWeaknesses:\n\n* I had a really hard time understanding the motivation behind this paper and the related works section (see below).\n* The Maze task is not particularly challenging relative to some of the tasks studied in previous papers (e.g. Miconi et al 2019 apply this idea to Penn Tree Bank, language modeling task). Adding additional tasks and experiments would improve the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the effects of imposing low-rank structure on differentiable Hebbian plasticity learning rules. These ideas are tested in a meta-learning / reinforcement learning setting. The method (DecPRNN) seems to perform reasonably well on a simple Maze navigation task, although other methods seem comparable (maybe slightly worse) to my eye. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The \"Related Works\" section should be re-written to make the contributions of this paper more clear. Please remove statements to the effect of \"...plasticity and model-based learning... closely simulates the learning of BNNs\" (see section 2.1). This assertion lacks justification as currently written. The authors should keep in mind that our current understanding of learning in BNNs is very limited.\n\nI only understood the methods and motivation for this project after reading Miconi et al (2019, ICLR) in detail. For example, equation 2 in the submitted manuscript is not clear because the time index, t, is not incorporated into these expressions. I had to refer to Miconi to understand.\n\nI found the discussion of \"genomics bottleneck\" to be confusing and unnecessary to motivate the paper. Similarly, I found the $\\theta_\\text{Gene}$ notation an unhelpful distraction / confusion. It seems like the ideas could be presented in a more straightforward manner.  If I understand correctly, the paper is suggesting a simple extension to prior methods that used differentiable Hebbian learning &mdash; the idea is to impose a low-rank structure on the meta-learning rule to reduce the number of trainable parameters to O(n + m) rather than O(n * m). The current way the paper is written obfuscates this because the intuition behind existing models is not explained, they are just listed in a citation dump.\n\nSeveral sections and ideas seem to be superfluous. For example, sections 4.3 and 4.4 are very short and it is not clear what I should take away from them. Overall, I would like the see the paper be revised to key in on what the main takeaways are. It was honestly difficult for me to read and understand.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qPKORuQQzq", "reviewer": "Reviewer_Z376", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of the paper:\n * Clarity and readibility of the paper are on a good level\n * The proposed \"decomposed plasticity\" is well-motivated and a simple change that can be easily reimplemented\n * The experimental section contains a comparison to a relatively large number of baselines\n * Experiments contain a diverse set of settings in MetaMaze2D, including different maze sizes, different life cycles, different spaciousness/crowdedness, and OOD meta-testing\n * Illustrations of trajectories in Figure 5 are helpful to understand the meta-learned learning process of the agent over its life time\n\nWeaknesses:\n * The proposed \"decomposed plasticity\" is a very special solution for reducing the number of meta-parameters. Alternatives such as sparse neuron connectivity or other low-rank matrix decompositions are not explored but could help to establish a more complete picture of \"Information-Bottlenecked Meta-Learning\"\n * Validation is limited to Reinforcement Learning in the discrete MetaMaze2D domain. Further validation on other problems and domains would be required to establish that the findings of the paper hold more broadly.\n * Some of the claims of the paper are too speculative and not backed by evidence (see Requested Changes)\n * As discussed by the authors in Section 5.2. \"Limitations\", further scaling up in terms of life cycle length, hidden size, depth of layers, and maze scale is not demonstrated and the findings of the paper remain restricted to rather small-scale settings.\n * Table 1 is not very accessible since it contains a lot of information. Accessibility could be improved by breaking it into several smaller tables or figures. Moreover, standard error of mean would be a better quantity for providing error bars than variance.\n * Similar to the point above, the text block containing the discussion of Table 1 is not accessible. It should be structured by text formatting or subparagraphs.\n * The fitness function of the long life cycle is non-trivial [w_z = 0.8^(7-z)] - is such manual engineering of the fitness function necessary and if yes, why?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is situated in the field of meta-learning (Hebbian-like) plasticity rules for neural networks. A core challenge for meta-learning is the that number of (meta-)parameters scales quadratically with the number of neurons in the network (because there are 3-4 meta-parameters per connection and dense neuron connectivity is assumed). This is neither biological plausible (due to the genomic bottleneck for meta-parameters) nor suitable for data/compute-efficient meta-learning. To address this, the paper proposes a \"neuron-dependent decomposition of those meta-parameters\" (essentially a rank-1 matrix factorization), such that meta-parameters are learned per neuron and not per connection. This reduces the meta-parameters to be proportional to the number of neurons (and not to the square of the number of neurons). The approach is validated MetaMaze2D benchmark and compared to various baselines.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nIn certain parts, the paper makes claims that I consider too speculative and not connected to the scope of the paper and the empirical findings:\n - \"Interestingly, human beings also experience similar cognitive decline when getting old; a reasonable guess is the explosive growth of human life cycle length in recent years goes far beyond our average life cycle length in evolutionary history.\" (page 9-10, this statement would need to be connected to research in biology)\n - \"We are looking forward to simulating inner loops as long as the human life cycle, genomes (NGene) and memories (NMem) of human scale, which has the potential of revealing human-like general intelligence.\" (page 11, please avoid speculations about human-like general intelligence in a context where experiments have been done in a Maze world)\n\nI would kindly ask the the authors to remove these claims (or rephrase them such that they are backed by evidence in the paper or by references).\n\nSource code is not accessible for the reviewers - if feasible, I would ask the authors to make it accessible in a anonymous form for us.", "broader_impact_concerns": "I don't see additional broader impact concerns besides the (potentially) increased carbon footprint caused by the additional computational requirements for meta-learning that are discussed in Section 5.2.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Ge81nkmr8", "reviewer": "Reviewer_FY1B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation makes sense: Due to their high number of parameters, existing plasticity methods are lacking in biological plausibility; the proposed method directly tackles this issue.\n- The proposed method is an elegant attempt to addressing this issue head-on.\n- Nice point that standard recurrent and self-attention models have inherent scalability issues as memory size increases.\n- Nice point that information in memory should be larger than information in functionality\n- Experimentally compares to many recent plasticity methods\n- Compelling visualizations (although in the Appendix).\n- The fact that the agent starts with random initialization in its lifetime is intriguing.\n- Experimental results support the hypothesis that plastic networks that satisfy the genomic bottleneck and start from scratch do worse than comparisons at the beginning of the agent’s lifetime, but can do better by the end.\n\nWeaknesses:\n- The paper is fairly difficult to read. This is partly due to use of uncommon terms and terms that are not clearly defined early on in the paper. I have listed some possible ways to clarify in Requested Changes.\n- The experimental random mazes environment is not clearly motivated.\n- The experimental evaluation metric is not easily interpretable, so it is difficult to understand what exactly is being learned.\n- Lack of comparison on existing meta-learning benchmarks, e.g., ones used by earlier plasticity approaches.\n- It is not clear that all methods have converged.\n- Important visualizations are in Appendix, not main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method of learning plasticity rules in which the parameters for plasticity are learned at the neuron level instead of at the connection level. These plasticity rules are then used to update the agent’s NN weights from scratch (random initialization) during its lifetime. This method reduces the number of parameters to optimize from \\Theta(n^2) to \\Theta(n), where n is the number neurons. This decomposition of rules from connections to neurons should be of general interest to the plasticity community.\n\nThe motivation is that, in biology, the information (genome) used to initialize the organism is orders-of-magnitude less than the information (experiences/memories) accumulated over the organism’s lifetime; whereas, in SOTA plasticity-based (and other) meta-learning models in ML, the number of initially-learned parameters is at least as many as the number of parameters updated during the agent’s lifetime.\n\nThe proposed system is a direct, elegant method for addressing this mismatch: the “genome” is \\Theta(n), and the “memories” are \\Theta(n^2), so it satisfies the property of “genomics bottleneck”. In experiments in a random maze environment, the paper shows that the proposed method can perform as well as methods that do not satisfy the bottleneck, and may do even better when generalizing to OOD tasks. Visualizations show that the plasticity operates in a smoother way than prior methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Changes critical for acceptance\n\n- Abstract: Reduce generalization claim, e.g., “…and deteriorate the generalization capacity” -> “and potentially deteriorate the generalization capacity”\n\n- Introduction: \n\t- Clarify meaning of “innate ability/capability” or rewrite this point in a different way. It is a confusing way to start the paper because “innate capability” in the first sentence refers to trained ANNs, but later in the paragraph it refers to untrained BNNs.\n\t- More generally, the genome metaphor should be made immediately clear in the introduction, since the whole paper depends on it. E.g., currently, the fact that meta-parameters can be thought of as “genomes”, and NN weights as “memories”, is not made explicit until half-way through Section 3.\n\t- “(Miconi et al., 2019b;a)” -> “(Miconi et al., 2018;2019)”. The 2019 paper is duplicated in the references. The first one should be removed.\n\n- Outer-Loop Evolution:\n\t- Describe differences between seq-CMA-ES and standard ES to make more clear why seq-CMA-ES is used. This is critical information for future researchers.\n\n- Experiment Settings:\n\t- Make clear why $w_z = 0.8^{7-z}$ for $z > 1$ in the Long Life Cycle. Otherwise, it looks like this was set simply to favor the proposed method, which does best towards the end of the life-cycle.\n\n- The maze domain must be more clearly motivated. E.g., answer the questions: What are other common domains for plasticity methods? Why is this maze domain chosen instead of those? Is a high level of memory needed for maze-solving for some reason?\n\n- A more interpretable metric should be added, e.g., to Table 1. I suggest using % mazes solved. Without this, since this is a new benchmark, it is difficult for the reader to interpret the significance of the performance differences.\n\n- Make clear what the error bars mean in Table 1 and in the Figures. It appears these are computed across mazes, not across independent trials of each algorithm. If there is indeed only one run of ES per method, I highly suggest increasing the number of runs since the only evaluation is in a toy domain with a relatively small number of weights. Even 3 runs per method should be sufficient to see if ES training is reliable, i.e., doesn’t depend too much on the initial centroid.\n\n- Learning curves, like the ones in Figure 7 (in the Appendix) should be shown for all methods. Otherwise, the reader does not know whether the differences in meta-test performance come from convergence in meta-training or fundamental differences in the final models, as claimed. To match the story in the paper, we would expect to see all the curves eventually flatten-out in these figures. \n\n- Figure 6 (or some version of it), along with some of the insights from A.1 should be moved to the main paper. It is the most interesting analysis in the paper, showing that the proposed method yields smooth learning behavior over the agent’s lifetime that is qualitatively different than other methods. Without this, the reader may think the difference is due to the reduction in parameters, which may be inherently preferred in a toy domain like mazes to avoid overfitting. So, Figure 6 should be moved to the main paper and/or additional comparisons to PRNN and Meta-LSTM should be performed with number of neurons chosen so that N_gene approximately equals N_gene of DecPRNN. Doing both would strengthen the paper further.\n\n- 4.2.2\n\t- Make clear why the “selected models” were selected. Why not show all of the models?\n\t- The point “that generalizability to OOD environments is more related to the genomics bottleneck than Hebbian plasticity” appears too strong without experiments showing that simply reducing the number of hidden neurons in PRNN would not lead to similar performance as DecPRNN. Either these experiments should be performed, or this claim should be reduced, e.g., “is” -> “could be”.\n\n- 4.3\n\t- Make clear why the experiments with long life cycles are so limited compared to those with short. E.g., was it just due to computational cost? Long life cycles are more interesting to the reader, so they should know any pitfalls that prevented further experiments with long life cycles. Because DecPRNN seems to have more advantage in longer life cycles, the paper would likely have a bigger impact if these experiments were more complete.\n\n\n\n### Further changes to strengthen the work\n\n- Algorithms:\n\t- Problem Settings:\n\t\t- This paragraph defining the standard meta-learning setup is difficult to read through. Consider reorganizing, e.g., defining “meta-training” before “meta-training-training”. This paragraph might also be clarified with a figure.\n\t- Decomposed plasticity:\n\t\t- “It is orders…” -> “For large $n_x$ and $n_y$, it is orders…”\n\t\t- Make clear why this decomposition method is chosen instead of some other $O(n)$ method, e.g., learning an $O(n)$-sized function of the $v$’s instead of simply taking their outer-product. Is it just because outer-product is the simplest approach that works? Is it somehow sufficient for capturing complex learning rules? Are there any theoretical claims that can be made here?\n\t- Outer-loop evolution:\n\t\t- Make clear why ES is preferred here instead of gradient descent as in Miconi, et al.\n\n- Experiment Settings:\n\t- Say why the final layer is non-plastic, i.e., is there some pitfall if they are plastic?\n\n- For completeness, it could be useful to mention existing approaches that use hyper networks to generate both network weights and plasticity rules, since they may satisfy the genomics bottleneck condition, though they may be challenging to directly compare to because of the varying number of parameters depending on evolutionary dynamics (Risi & Stanley: “Indirectly encoding neural plasticity as a pattern of local rules” 2010; “A unified approach to evolving plasticity and neural geometry”, 2012). \n\n- The curves for DecPRNN(PostDN) in Figure 6 are so simple that one might be able to characterize qualitatively how the weights are changing over time. A solid explanation for this extremely-simple-looking behavior would make the paper stronger.\n\n- For Figure 6, make clear whether or not this visualization method has been previously used for visualizing meta-learning methods.\n- Evolve&Merge should be added to Figure 6 since it is the most similar to DecPRNN in terms of n_gene and n_mem.\n\n- The paper should explain why some comparisons take so long to converge with ES.\n\n- Running the experiments in an established benchmarks could be the single biggest way to strengthen the paper.\n\n\n\n### Additional clarity suggestions\n\n- abstract: \n\t- “with verbose meta-parameters” -> “methods that have many more meta-parameters.”\n\t- $O(n^2)$ -> $\\Theta(n^2)$\n- introduction:\n\t- “BNNs Genomic Bottleneck” -> “BNNs: Genomic Bottleneck”\n\t- “such like human babes” -> “as in the case of humans” or remove completely\n\t- “few-shots” -> “few-shot”\n\t- Cite prior work for LBI\n\t- “neuron type” -> “neuron” (2x)\n\t- “from scratch” -> “from scratch, i.e., from random initialization,”\n- figure 1:\n\t- “genomes that” -> “genomes, which”\n\t- “environment, which is” -> “environment, and are”\n\t- “massive information” -> “massive amounts of information”\n- Modulated Plastic RNN:\n\t- Mention/cite previous work that uses modulated plastic RNNs\n\t- “input neurons respectively” -> “input neurons” \n\t- “actually use seq-CMA-ES” -> “use the ES variant seq-CMA-ES”\n- Figure 2:\n\t- “crowed” -> “crowded”\n- Standard Meta Testing:\n\t- Make clear how many hidden neurons are used in your approach.\n\t- “equal number of genomes” -> “approximately equal genome size”\n", "broader_impact_concerns": "I could not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on a few contributions related to meta-learning agents, primarily the decomposed plasticity approach introduced by the authors to reduce the number of meta-parameters required for supporting meta-learning.  The authors also evaluate a form of structured artificial neuromodulation.  

The reviewers largely understood the relevance of the problem.  For example, reviewer FY1B noted “Due to their high number of parameters, existing plasticity methods are lacking in biological plausibility; the proposed method directly tackles this issue.”  Reviewer Z376 noted “proposed ‘decomposed plasticity’ is well-motivated and a simple change that can be easily reimplemented”.  Some reviewers (FY1B and hAdL) found parts of the original presentation unclear.  Reviewer hAdL engaged with the authors concerning some of the terminology and notation, which the authors adjusted.  

Based on the initial feedback from reviewers, the authors undertook substantial revisions.  Note that this paper lagged relative to the target timeline for TMLR submissions, primarily due to relatively substantial revisions being performed as part of the author-reviewer discussion phase.  The authors updated the paper, making the notation and formalism more self-contained, better motivating the choices in the work, and refining the claims made in light of the literature and the contributions made.  The authors also adjusted the contents of the main paper, and now include material in the main text that had previously been in the appendix.  
 
After revisions, reviewers were largely satisfied with the updates. Reviewers FY1B and Z376 issued a few additional more minor pieces of feedback and the authors implemented additional changes. All reviewers ultimately either recommend acceptance or lean towards acceptance.

One shortcoming raised by all reviewers has to do with the exclusive reliance on the maze task.  Essentially, reviewers felt that inclusion of other tasks may increase confidence in the generality of the findings and that the maze is relatively simple.  The authors respond to this by further motivating the use of the maze, noting that the maze has partial observations, substantial procedural diversity, and a long life cycle.  They also note that the maze is compatible with limitations on computational resources. While this is helpful to clarify what about the maze made it suitable, I don’t think the features of the maze task identified by the authors are exclusive to it, so in principle other tasks with these features could also be considered.  However, application of the proposed method to additional tasks is somewhat onerous, and I think the evaluation is sufficient for a paper of this scope.  

Very minor points:
- “Phenoype” is misspelled in figure 2.
- The observation in this work is a partial state feature (along with previous action and reward). The partial state feature is notated with s_t (“the 3 × 3 grids observed (s_t)” ) which could easily be mistaken for the whole state, so this is probably a suboptimal choice of notation.  

Decision: Accept (poster)