Evolving Neural Update Rules for Sequence Learning
Authors: Karol Gregor
Peter Conway Humphreys
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Update Rules
Evolution

Abstract: We consider the problem of searching, end to end, for effective weight and activation update rules governing online learning of a recurrent network on problems of character sequence memorisation and prediction. We experiment with a number of functional forms and find that the performance depends on them significantly. We find update rules that allow us to scale to a much larger number of recurrent units and much longer sequence lengths than has been achieved with this approach previously. We also find that natural evolution strategies significantly outperforms meta-gradients on this problem, aligning with previous studies suggesting that such evolutionary strategies are more robust than gradient back-propagation over sequences with thousands(s) of steps.

Under review as a conference paper at ICLR 2022
EVOLVING NEURAL UPDATE RULES FOR SEQUENCE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of searching, end to end, for effective weight and activa- tion update rules governing online learning of recurrent network on problems of character sequence memorization and prediction. We experiment with a number of functional forms and find that the performance depends on them significantly. We find update rules that allow us to scale to much larger number of recurrent units and much longer sequence lengths than has been achieved with this approach previously. We also find that natural evolution strategies significantly outperforms meta-gradients on this problem, aligning with previous studies suggesting that such evolutionary strategies are more robust than gradient back-propagation over sequences with thousands(s) of steps.
INTRODUCTION
The field of neural sequence processing has become dominated by neural networks trained by back- propagation, with the best models being transformers currently, and LSTM recurrent networks in the not recent past. These networks can be used for range of problems such as label prediction, modelling and reinforcement learning. The basic computational nature of these networks can be characterized follows: One can design an arbitrary (differentiable) forward computation that updates part of the network (activations) and then one executes a specific algorithm to do the updates of another set of parameters (weights using paok-propagation and variants of stochastic gradient descent). This has an advantage of freedom to flexibly design a forward computation. However, the hand-coded update algorithm is fixed, and may not be the most effective means of achieving given training objective. There has been interest, as we review below, in directly finding the full computation (both activations and weights updates). In this paper, we aim push this approach further by finding update rules that work better and that scale, by comparing different functional forms and methods for optimizing the search for these rules.
We pursue what we call end-to-end search for update rules. A good way to explain this is to consider the analogous process in nature. Roughly speaking the brain of an organism is produced from the information of the parents, "run" for the lifetime of the organism, and selected based how it, as well its descendants perform (for example keep making descendants). Over time, this process developed neural networks that can learn. adapt quickly and that possess whichever other abilities were needed for the success of the organisms. Similarly we consider a space of neural networks and search for updates to their parameters, the activations and the weights in our case, that work well problems we are interested in. This can allow networks to discover whatever updates are needed to solve these problems.
We consider parameterization of neural networks that is somewhat similar to classic artificial re- current neural networks, with state consisting of activations h; and weights wi time There are, however, two primary differences. The first is that the entire computation (including learning or whatever computation the network does) operates online, that is, the network receives input x, and updates according some function
hy Wt. Pt (x fy, w;-1)
where fo is the update function parameterized by syper-parameters 0 that are fixed for the lifetime of the network. The p. is the output- in our case the probability over the next character in the sequence (we will discuss the problems we solve later). The second difference is that the activation h4 of

---

Under review as a conference paper at ICLR 2022
given neuron i as well as the weight Wayi between neurons j and are (potentially) small vectors, as in [Bertens& & Lee, 2019; Gregor 2020).
Biological neurons
inspiration for artificial neural networks, and so is the case here. The for- mer are complex objects, and representing the cell body state as well as the synapse state by vectors (rather then scalars) should allow for closer representation of the neuron's computation. Biolog- ical neurons furthermore provide a proof of principle that online neural computation is capable of powerful learning.
There are several additional motivations for pursuing this approach. Learning general update rules end to end might allow networks to use weights in more interesting or powerful fashion than that prescribed by propagaopgg Finding online rules might allow for casier hardware implemen- tation, especially in neural hardware architectures (Modha 2017 Davies et al. 2018). Developing the ability to search for update rules might help to discover how brains compute, for example if we can implement what we know about the computational structure and search for what we don't t. Finally in artificial life, entities propagate into the future often not based on an objective but simply they find means of making descendants (and those descendants make more descendants and so on) (Ray) [19T]] Aguilar et al., 20144 SSoros & Stanley 2014] Gregor & Besse, (021). Being able to evolve learning algorithms without objective might provide a path to intelligent artificial life.
1.1 RELATION TO PREVIOUS WORKS.
Evolving network weights
There are many works that directly evolve network weights, for example (Wierstra et al. 2014 Salimans et al. 2017). In (Stanley & Miikkulainen, 2002) they parameterize the weights by compo- sitional pattern producing networks (CPPN) and in (Risi & Stanley 2010) they consider an adaptive version that updates the weights based on pre- and post-synaptic activations. While in one sense this is similar to our approach in that the weights are updated, it is also very different with very dif- ferent parameterization (CPPN's s and no hidden network units), and problems (evolving the system to solve a T-maze task), and without focus on pure end-to-end learning the parameters are also evolved for the task.
End-to-ende learning and meta-learning
A range of related works explore end-to-end learning and meta-learning, but don't strictly operate in the local update rules setting. Furthermore, many of these works learn the full weights of the network instead of a smaller number of update rule hyper-parameters. In (Schmidhuber 1992) they consider end-to-end setting, parametrizing updates of (fast) network, by another (slow) network, that outputs vectors used to compute Hebbian updates to parameters. In AutoML zero (Real et al., 2020) they evolve sequence of operations such as multiplication and non-linearities that implement a machine learning algorithm (that includes learning).
A series of works such as (Ravi & Larochelle 2016 Metz et al., 2019; Wichrowska et al. 2017;
Bello et al. 2017 Li & Malik] 2017; et al. 2017) focus on learning generalisations of gradient- descent optimisers that utilise sack-propagater gradient information to update weights. Although these works optimise the parameters of the update rule over multiple update steps, the tasks them selves are typically feed-forward in nature, such as image classification.
some works (Ravi & Larochelle 2016), they explicitly consider few-shot learning, which is related to online learning. although task relevant information preserved across tasks in the full network weights.
In (Miconi 2016 Miconi et al. 2018) they consider recurrent neural networks, but introduce per- weight state variables that are updated in a Hebbian fashion with parameters of the updates trained by pack-propagations along with standard set of weights. This is capable of learning fast weight adaptations.
Alternative network parametrizations
In (Ha et al. 2016) they consider a recurrent network with weights parameterized by a smaller set of hyper- sreertteights... This is trained by standard propparopagttion over short periods. Because the num- ber of heper- is still relatively large, the system is still relying on back- took-propagation learning to encode the structure of the current data-sequence, rather than learning algorithm primarily.
2

---

Under review as a conference paper at ICLR 2022
Learning local update rules end-to-end
There are a number of works more closely related to our objective of end-to-end learning of lo- cal update rules (Bengio et al. 1992, Orchard & Wang 2016 Gu et al., 2019: Munkhdalai et al. 2019 Bertens & Lee 2019; Gregor 2020; Kirsch Schmidhuber, 2020). For example [engii linear combination of terms derived from local scalar activations and previous weights and searches for these parameters using genetic algo- rithms, stochastic gradient descent or simulated annealing applied to toy problems. In Orchard & Wang (2016) a population of agents is evolved for the task of foraging in 2d world. However both the initial weights of the controller network as well as the synaptic update network are evolved, in principle putting good policies in initial weights (but showing that updates improve performance). Bertens Lee (2019): Ciregor (2020) introduce the idea of using vectors to represent both the acti- vation and weight states, and updates using LSTM's on T-maze tasks in the former and MLP's on sequence memorization respectively in latter.
al. (1992) parameterizes scalar weight updates
A key drawback in all these works is that the networks used are tiny, of usually less than ten neurons. In Gregor (2020), they attempt to use larger hidden layers, but find that the network does all its computation in the input layer. In Kirsch & Schmidhuber (2020) they consider more general network parameterizations, using LSTM- -STMMbasee local updates and scaling to a larger number of neurons. However they only consider problems such as MNIST classification that do not require algorithm to learn from long-range time dependencies. That is, because the same classes often appear close to one another in random sequence, there is signal over just a few steps (longer range dependencies might be needed for good performance, however the reported one is low).
The main contribution of this paper is find a parameterization of update rules and method of search that scales to large number of neurons (we tested up to a thousand hidden units) and weights (million) and that can learn recurrent network training over long time spans (thousands).
2
TASKS
We train the networks on two tasks both defined on a sequence of characters a text. We use the Pg19 data-set (Rae al. 2019) containing large collections of books our text source. In the first task we consider sequences of length N taken at random points from the text. The N is of the order of ten thousand in our experiments, but in principle it can be arbitrarily long or should ideally use the entire data-set, this is just the scale we have managed to achieve so far. At each iteration, we initialize neural network with random weights and zero activations, run it through this sequence online as in eq ( and measure the total log-likelihood:
Σ log p(a.(ho-1)
We are looking for set of byper-parameters parameterising the update rule f that minimizes the loss L. The resulting 0 should encode activation and weight dynamics that learn model of the sequence while running through it online. In the second task we play such randomly sampled sequence twice and measure the likelihood on the second part, testing network's 's ability to memorize.
OPTIMIZATION
There are two algorithms we compare for optimizing The first algorithm is (mneta--radieen where we repeat the following process. We sample mini-batch of randomly selected sequences from the text. We run the network forward and back pack-propagate, both through the whole batched sequence, and take stochastic gradient optimizer step we use Adam (Kingma & Ba 2014) in the experiments.
The second technique we employ is natural evolution strategies (NES) (Wierstra et al.) 2014; Sal- imans et al. 2017). Instead of optimising only the parameters 0, this method instead maintains search distribution over the parameters. We use separable NES (SNES), in which this distribution is diagonal normal distribution parameterised by the parameter means and variances. At each itera- tion. a population of update rules (population of f's) is sampled from this distribution as well as a single sequence from the dataset. The networks are run forward obtain fitnesses, which are then
3

---

Under review as a conference paper at ICLR 2022
used to estimate improved values of the parameter means and variances. In contrast to (Salimans et al. 2017), we find that updating the variances at each iteration is beneficial. Further details for our implementation given in App. A.3
4 MOTIVATION BEHIND THE TWO TASKS.
The standard way to compare models is on their likelihood on withheld test set. However, this has its issues: For example, how much overlap (either direct or approximate) there with the train set, how much out of distribution the data set, but compare models on how well they can compress it. For models, the compression essentially equals the likelihood of the data under the model, plus the cost to encode the model, or better, the code defining the model training and its decompression mechanism (Bellard 2019] 2021). This is exemplified by the Hutter Prize (Hut) challenge to produce as small file as possible, which when run, outputs a one billion character version of Wikipedia.
want our models generalize. A different view is to only have
Here, let consider model that goes over the text (say over the letters), making next step pre- dictions. If a model encounters new piece of text, such as a word "computer", it needs to store some information about it. This because the next time this word, or something related to it ap- pears, perhaps many thousands of steps later, a good processing of the original word would allow the model make good predictions. Standard likelihood training has a handcrafted means of stor- ing the text: increase the likelihood of the current piece of text (replaying it many times, (Bellard) 2019)). This working relies on a generally reasonable axsumption that statistics of data in the future will be similar to the current ones. In our approach we don't 't make such an assumption the system evolves how it should process the current piece of information in order to do well in the future. This could in principle evolve both fast storage of the current piece of text, as well as process similar to increasing the likelihood under a model over many time steps, in the same network. To do this, the search process needs to be able to find relationships that span thousands even possibly millions of steps. Thus, rather than hoping to find such signal by back-propes proppprrpgggiing over such periods, we can expect evolution work better: For example, if local update rules are modified by chance to store information, this modification can have immediate effect over long time scales. To find strong relationships, we should ideally run over long text such as Hutter prize text. While we have not achieved such scale in yet, we show, for the first time, that the language model learning algorithm can be evolved end to end, even on a shorter pieces of text (order 10k).
In addition to prediction/compprssionn, we would also like to have a task that has an adjustable diffi- culty level and clear performance measure; fast to experiment with for easier difficulties, and chal- lenging for standard recurrent networks at harder difficulties. For this purpose we study sequence memorization: Presenting a sequence twice and measuring the likelihood on the second part. We define success at memorizing when the network likelihood is significantly smaller in magnitude (closer to zero) than pure prediction performance on just the text.
NETWORK DETAILS
In this section we describe in detail the operation of the neural networks that we use.
We use a three layer neural network in our experiments (though the formulation allows for arbitrary number of layers): The input layer h', the hidden layer h² and the output layer h3. For any given network, each layer is of the shape n' x na where is the number of neurons in layer I and n. is the size of the activation state of a given neuron, usually six our experiments (one in classic RNNs).
Layers are connected by weights. The weights going from layer / to layer k are denoted by w and form a tensor of shape nt' x n x The where Th is the size of the state of a given weight. classic RNN this is one in the forward pass, and it also ends up being one for the best performing networks, but we also experiment with more complex weight updates requiring a larger size.
The computation at a given time step is shown in the Algorithm| I Here explain the steps of this algorithm. The input x is a one hot vector representing the input text character at the current time step, and is of size n. (equals the number of possible characters, 259 in experiments). It is fed to the 0-th index of the layer I tensor: [t. 0J (so there are 259 neurons in the first layer, each with 6 dimensional (activations) state, and therefore the h- has shape 259 6).
4

---

Under review as a conference paper at ICLR 2022
Algorithm Network update at given time step
Require: Input
Require: Activations h² h of layers/ I,..., shapes n' x Be
Normally: L = 3 with h² the input layer, h2 the recurrent hidden layer, h the output layer Require: Weights w for each pair of layers that are connected
Normally: (1, 2). (2, 1), (2, 3), (3, 2), (2, 2)
Require: Activations update AU
Require: Weights update WU h [=(,0)
Neuron layer
Synapse
for = 1...., I...,,Ld do
a a= [b'
for ..... L do
h
8
if 1, k connected then
a.append wh h [; 0)) o.append w Mt h2t 1))
PMLP
end if end for h' AU((a) for ....... do
Neuron i in layer
From other neurons layer
T
if connected then Wik = Www A1,, /)
h
+
From neurons a different layer
PMLP
end if end for
+
p nerm(relu(h41: -1) 0.001)
end for
Figure 1: The network update at given time step, Eq The steps of the algorithm are explained in the text, The diagram shows the computation for the functional forms with the best performance, except WT, is omitted for clarity. The h is six dimensional vector (the six red squares) representing the state of neuron (it's activation part). Blue: Communication between neural activations, consist- ing of matrix multiplications (multiplying w and h[] in the figure and then adding with "+") that get appended to a vector a (all the red squares within the neuron). Red: the neuron activations update (AU), containing hyper-weights 0 inside of the product MLP (PMLP), Eq that we evolve. The is the delay that returns previous value. Appending results from different layers (the two red squares on the left side of the neuron) and passing through the PMLP, allows for different layers to affect given neuron differently. for example excitation from one layer and gating from another. Green: Hebbian like weight update (WU) taking product of component 2 of pre-synaptic and component of post-synaptic neuron. We can also consider PMLP and LSTM for synaptic updates (also con- however, we found that the evolved performance worse as well
taining evolved as being computationally much more costly.
Next we iterate over the layers. At each layer, we first initialize a list of tensors a = [h'] to include the activations h of the layer. We next check which layers are connected to the current layer. These are the layer above and the layer below (but could be other connectivity), if present, as well as recurrent connection. We don't 't use recurrent connections in the input and output layers- only in the hidden layer(s). For given connected pair of layers, we do two matrix multiplications (denoted by :): with (that "goes" from layer & to layer 1) multiplies the 0-th component h [ 0] of tensor h² and the transpose of wz (that "goes" from layer to layer k) multiplies the 1-st component ( 1] of A The results of the multiplications are added to the list a. After the k-th loop, the a will contain multiplications from all the layers that the current layer is connected to, well the current value of its state h². Given this collection, we perform an activations update (AU), which produces the new activation state h' This is wose update (without communication between neurons) and will be described in detail below.
Next, still at given layer I, we go over all the layers i and update the weights of the connections " that go from those layers to this one using a weight update WU described in detail below. Again, this is independent update for each weight.
5

---

Under review as a conference paper at ICLR 2022
Having updated the activations and weight at this time step make prediction of the next charac- ter. We take the last component of the last layer [[]] and compute the probability directly from it. We experimented with two versions. The first version is more standard treating the neural vector as logits and computing the probability = so toftmar(h" (1 I))- In the second version, we treat the neural vector un-normalized probabilities and computing the probabilities by normalizing as shown in the algorithm.
Next, let us describe the activations update AU. The activations update can in principle be any function, usually parameterized by some parameters Oa. The function that we found to work well is a one hidden layer MLP with a x tanh non-linearity, which we denote by PMLP (product-MLP):
(a = M3 (M1 - a) tanh a) + b1) + bs
(3)
where M's are matrices and b's are vectors and there is point-wise multiplication between the a and tanh.
We also experimented with the LSTM non-linearity, where we split h, into the hidden state and cell state of the LSTM, treat the rest of inputs and do standard LSTM update.
Let us now describe the weights update WU. While we experimented with various forms of the update, we found that simple Hebbian-like update with scalar weights for every connection worked the best:
WU J1, (DD) = [t N/T [}.2
(4)
where we have written the update for the weight going from neuron j in layer k to neuron i in layer 1. Importantly, note that we have used the index two component the source neuron and index three component of the target neuron. That is, different components of the activation vector of given neuron are used for different purposes: for matrix multiplication, I for matrix multiplication by transpose, 2 in weight update for the source (pre-synaptic) neuron, 3 in the weight update for the target (post-synaptic) neuron (the specific numbers don't ton't matter, only that they are different). This way the network can different components as needed. Again, we also experimented with more complex forms, making w a vector and using MLP and LSTM weight updates.
Here we explain the motivations behind various choices. We a vector representation for neural activations as this provides a richer update while barely increasing computational cost, as it is of order N=number of neurons, not N2=number of weights. We use product MLP because good update might require multiplications, but we don't 't know of which variables- - PMLP can select which variables are needed. can implement for example gating by one type of neuron (in one layer) and excitation by another type of neuron (in another layer), possibly matching closer the functionality of biological neurons. For the weights, we could have them between each pair of components of the activations vector, but that would significantly increase the cost currently we have the same number of weights as a standard recurrent network one per pair neurons (actually somewhat more because connections go both ways). We use Hebbian update rather than more general parameterization by a neural network (even though we experiment with it as well) because again the computational cost is low roughly equal to the classical tanh RNN with the same number of neurons (if we include baok-propagation for the latter). We use W to help with ability to imitate passing gradients. We use different components of the neuron activation for multiplication by W, for multiplication by 4 as well as for the weight updates at pre and pos-synaptic neurons in order to allow network to put whatever information it needs (e.g. delayed information or gradient like) for each operation.
RESULTS
We would like to investigate the following questions. Does the system use the hidden layer? the system able to train with a large hidden layer size (is it stable) and does increasing size improve performance? Can the system learn if the signal is separated by larger timescales? What functional forms for the activations update and weights update work well? How does meta-gradients compare to evolution as an optimizer for update rules? How does memorization performance compare to LSTM for different network sizes? Are we able to train end to end for the language modelling, how long sequences can we train on and what is the performance? And finally, does the evolved learning algorithm transfer to sequences with different statisties?

---

Under review as a conference paper at ICLR 2022
6.1 SCALING WITH HIDDEN SIZE AND SEQUENCE LENGTH
We start by investigating scaling of performance with hidden layer size on the memory task for the update rules defined in Algorithm[] with the activations update as defined in (3) and weight update in 4 and trained with natural evolution strategies.
2 shows the performance of the system for different sequence lengths and different hidden sizes. From these curves see: 1. The update rules are able to perform at large hidden sizes including up to 1000 units that we studied here; 2. The system performance improves with larger hidden size; 3. When the hidden size is greater or equal to about third of the sequence length, the network is able to essentially perfectly memorize the sequence even for sequences of 1000 steps. 4. As the number of units decreases further, the performance degrades smoothly, and the network able to form a compressed representation of the sequence,
0.0
act act_PMLP 1LLP hebb
act PMLP W, hebb
-05
LSTM w_hebb
act LST.STMMM W. hcbb
Filness
act PMLP w. PMLP
-15
act LSTM W. PMLP
-15
40
act BBL___PPMM PMLP
100
-2.0
act, PMLP w. LSTM
-2.0
300
PMLP
act_ISTM w. ISM
10a
10
10
-3.0
LSTM
Sequence length
2000
6000
10000
Figure 2: Left: Performance of the PMLP network trained with NES on the memorisation task for different sequence lengths and number of hidden units. The lines show the best performing seeds for each number hidden units, with dots indicating individual seeds' performances. Right: Comparison of different functional forms for activation and weights updates on memorization of sequence of length 100. The x-axis the evolution update number. We plot the best performing (solid) and fourth best (dashed) seed for each parameter combination. PMLP: One hidden layer neural network with a tanh nonlinearity. LSTM: standard LSTM. Hebb as described text (4). Note that the non-Hebb weight updates are significantly more expensive computationally.
6.2 COMPARING FUNCTIONAL FORMS
In this subsection study the performance of different functional forms for the activations and weights updates. For the activations update we compare: Network with a X tanh non-linearity (PMLP) (3). standard MLP (different non-linnarities) and LSTM. Note that this is LSTM con- trolling the updates, but the base network is not an LSTM. For the weights update we compare: Hebbian update (4) MLP with (anh non-linearity and LSTM. We find that the combination of PMLP with the Hebbian update works significantly better than the other combinations. Plotted in Figure 2 Right comparison for simple problem of memorizing sequences of length 100 with 160 units as harder problems the other combinations fail or are too slow). As can be seen, the cho- sen combination performs significantly better. Furthermore, using a neural network for the weight update (MLP or LSTM) instead of the Hebbian update significantly increases computation since this network is applied at every single weight (~ N2 times where is the number of neurons).
We experimented with different ways of using the MLP and LSTT for the weight updates. First we have a choice of concatenating the input and output activations (we use 3rd and 4th component as discussed above) versus inputting their outer product (the same product used in the Hebbian update). We sweep over combinations concatenating, outer product and both. In the PMLP case, we also investigated using the output of the network directly for the new weight, opposed adding the output to the old weight to get the new weight.
Taking the best combination (PMLP for activations and Hebbian for weights), replace PMLP by a regular MLP. All runs fail with relu non- linearity. Tanh non- non-linenn con-lincar solves the 100 steps memory task, but does not succeed task of length 300 completely, and larger proportion of runs fail.
7

---

Under review as a conference paper at ICLR 2022
6.3 OPTIMIZER
In the previous experiments found the meta- pattaaaaaeeeerss 0 of the PMLP network by NES. In this section we compare this to a meta-gradient optimizer, for which we simply back-propagate the objective through the entire sequence to calculate gradients of 0 and Adam optimizer to optimize for 0. Figure Left shows the results. We see that meta-gradients struggles with this problem and only able solve relatively short sequences.
-0.25
40
100
Fitress
-0.75
300
-4
-1.25
100
-5
300
1000
-1.75
2000
10a Sequence length
10a
Sequence length
Figure 3: Left: Performance meta-gradient optimisation the PMLP network on the memorisa- tion task for different sequence lengths. The lines show the best performing seeds for each number of hidden units, with dots indicating individual seeds' performances. The dashed lines reproduce the equivalent best performing seeds under NES optimisation shown in Fig 2 We ran all combinations of sequence length and hidden units that were possible given device memory constraints. Right: Performance of an LSTM network trained by meta-gradients show the best performing seeds for each number of hidden units, with dots indicating individual seeds' performances. In order to facilitate comparison with a PMLP network with the same number of hidden units (but a very different architecture), the dashed lines reproduce the best performing seeds from Fig 2
the memorisation task. The lines
6.4 COMPARISON AN LSTM NETWORK
Next, evaluate the performance of standard LSTM network on the memorisation task (Fig Right). The LSTM trained in the standard way on the task: weights are kept constant within the sequence and updated across sequences they trained to make activations memorize the sequence. Therefore they correspond to hyper-weights of the update rules network. The LSTM has much more of these than the update rules network and the course of training they will learn a language model. The activations of the LSTM correspond to the weights and activations of the update rules network (in that they change during the episode and are reset between episodes).
We find that meta-gradients are more effective than NES for training the LSTM network. likely because has many more parameters than the small number of syper-parameters of the PMLP update rules. We find that while LSTM can handle this task to an extent, it is unable to achieve good performance of sequence of length 1000, while the update rules network can solve this task near perfectly.
6.5 ONLINE LANGUAGE MODELLING
In this section we study how well can the system evolve, end to end, a learning algorithm for lan- guage modelling. Figure 4 Left we show the performance for different text lengths and network sizes. As expected we see that the performance increases with sequence length as there are more dependencies that can be captured and more time to learn. In addition, importantly, we also observe that the performance increases with network size for long enough sequence lengths (2k, 10k 100k) where there are enough dependencies to be captured to benefit from larger hidden size, showing that the update rules able utilize the network size (10k steps, 1 units hasn't 't converged yet).
8

---

Under review as a conference paper at ICLR 2022
leamning algorithi
0.0
40
-20
-3.5
memory direct (best)
1000
memory direct Imedian;
Riress
-25
Runes
memurry transfer (nuedien)
-1.5
predi ction direct (besti
direct Imedian)
transter (best)
-3.5
prodi ction transfer (median)
-3.0
3x0 10a 10 10
10'
step
Figure 4: Left: End-to-end language modelling. As expected the prediction performance increases as the length increases there more dependencies that can be captured in longer sequences. We also observe that for sequences of 2k and 10k, larger hidden sizes generally improve the perfor- mance (the Ik units 10k length has not converged. see Figure in the appendix). This shows that the algorithm is able to utilize network capacity for language modelling. For shorter sequences where there is not much structure be captured, large number of hidden units is likely not needed. Right: Transfer experiment. Comparison of performance on "Alice's Abenteuer im Wunderland" in Ger- man when the algorithm was evolved directly on this text (direct) and when evolved "Wizard of Oz" in English (transfer) on both memory and language modeling (prediction) task. We see that there is very little difference between the original and transfer, showing that the evolved (learning) algorithm transfers well.
6.6 TRANSFER
Here we test whether evolved update rules capture a more general learning algorithm that can be applied sequences of different statistics. For this purpose evolve the model on the English text of "Wizard of Oz" and then test on German text "Alice's Abenteuer im Wunderland". We compare the resulting performance that obtained by evolving the latter text directly. Figure 4 Right, shows the result for memorization (300 sequence length, 200 hidden units) and language modeling tasks (10000 sequence length, 300 hidden units). We see very little difference between the transfer and direct training, showing that the algorithm transfers well. (Note: In this version we converted to lower case 27 characters, the numerical values are somewhat different)
6.7 DEPENDENCE ON
We have also studied the dependence of performance on the basic parameters of the network, the results are the Appendix.
7 DISCUSSION AND CONCLUSIONS
We have succeeded at evolving online neural update rules, end to end on short-scale language mod- elling and somewhat difficult length sequence memorization. The main features were the following: 1. Using natural evolution strategies instead of back propaaaapag,,onn; 2. Using four different compo- nents of neuron's activation vector for different purposes; 3. Using a simple Hebbian update for weights obtained from components of the activations tensor; 4. Using
tanh non-linearity for neural update; Using matrix multiplication by both weights and weights transpose between layers; 6. Using direct normalizing non-lincarity the output instead of softmax.
There are several possible future directions. One is simply scaling, as most of the experiments were run a single GPU (per curve), while evolution is highly scalable. Achieving direct evolution on Hutter prize could be considered a next grand challenge for this approach. Another direction is to evolve functional forms well. Finally, discussed in the introduction, this work could have applications for building specialized neural hardware, modelling brain circuits and evolving artificial life.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hutter Prize. pttp:///pii.ehtttrrr wwatzereuttrrrrr.. net
Wendy Aguilar, Guillermo Santamaría-Bonfi. Tom Froese, and Carlos Gershenson. The past, present, and future of artificial life. Frontiers Robotics and AI, 1:8, 2014.
Fabrice Bellard. Lossless data compression with neural networks, 2019.
Fabrice Bellard. Nnep v2: Lossless data compression with transformer. 2021.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with rein- forcement learning. In International Conference on Machine Learning, pp. 459-468, PMLR, 2017.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synap- tic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2. Univ. Texas, 1992.
Paul Bertens and Seong-Win Whan Lee. Network of evolvable neural units: Evolving to learn synaptic level. arXiv preprint arXiv:1912.075899 2019.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82-99)) 2018.
Karol Gregor. Finding online neural update rules by learning to remember. arXiv preprint arXiv:2003.03/24, 2020.
Karol Gregor and Frederic Besse. Self-organizing intelligent matter: A blueprint for ai generating algorithm. arXiv preprint arXiv:2101.07627 2021.
Keren Gu, Sam Greydanus, Luke Metz, Niru Maheswaranathan, and Jascha Sohl-Dickstein. Meta- learning biologically plausible semi-supervised update rules. bioRxiv, 2019.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint wiXiv:160.009106 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 12 2014.
Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. arXiv preprint arXiv:2012. 149005 2020.
Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint (003v:/703.00444 2017.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning, pp. 2247-2255. PMLR, 2017.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556-4566 PMLR, 2019.
Thomas Miconi. Learning to learn with backpropagation of hebbian plasticity. arXiv preprint arXiv:1609.022286 2016.
Thomas Miconi, Jeff Clune, and Kenneth O Stanley. Differentiable plasticity: training plastic neural networks with backpropagation. arXiv preprint 18X.vv1804.02466 2018.
Dharmendra Modha. Introducing brain-inspired computer. Published online at http://www. research. ibm. com//rriicee/bruinnchhi shtml, 2017.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metaleamndd neural memory. Advances in Neural Information Processing Systems, pp. 13310-13321, 2019.
Jeff Orchard and Lin Wang. The evolution of generalized neural learning rule. In 2016 Interna- tional Joint Conference Neural Networks (IJNNN, pp.
IEFE, 2016.
10

---

Under review as a conference paper at ICLR 2022
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillierap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL
hnttps://arxili wtw./org//aaa wtt.///g//////ooooo http//bs//1999 05507
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Thomas Ray. Evolution and optimization digital organisms. Scientific excellence in supercom- puting: The IBM 1990 contest prize papers, 1991.
Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning, pp. 8007-8019. PMLR, 2020.
Sebastian Risi and Kenneth O Stanley. Indirectly encoding neural plasticity as pattern of local rules. In International Conference Simulation of Adaptive Behavior, pp. 533-543. Springer, 2010.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprini arXiv:1703.0380 2017.
Jürgen Schmidhuber: Learning to control fast-weight memories: An alternativve to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
L Soros and Kenneth Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In ALIFE 14: The Fourleenth International Conference on the Synthesis and Simulation of Living Systems, pp. 793-800. MIT Press, 2014.
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein, Learned optimizers that scale and gen- eralize. In International Conference Machine Learning, pp. 3751-3760. PMLR, 2017.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15())999--980, 2014.
A APPENDIX
A. DEPENDENCE ON HYPER-PARAMERR PARAMETERS
We look how the performance depend on size of the neuron's state and the size of the hidden layer in neuron's activation's update. We choose problem that is challenging but not too challenging so that large number of sweeps can be run memorization of sequence length of 300 with network of 100 neurons. The performance is shown in Figure We see that for 100 networks in the population, the best performance happens at around 10 state size and 14 hidden size while for larger population, where NES signal is smoother, the network can optimize better larger hidden sizes (of neuron's update, not the overall network size).
A.2 LANGUAGE MODEL TRAINING CURVES
Figure shows the training curves for sequence lengths and 10k.
A.3 NATURAL EVOLUTION STRATEGIES
Our implementation of separable natural evolution strategies (SNES) is summarised in Algorithm 1 (lightly modified from Wierstra et al. 2014)). ) At each iteration, we generate a population of nsamples perturbed samples of the update rules parameters 0, based on the current SNES search distribution. In our case, this is parameterised as a set of separable normal distributions with means and log- standard deviations log or. For each sample, fitness function as in Wierstra et al. (2014) Section 3.1 to convert this into a utility ug. The hyper- parameters used summarised in Table
measure the loss L(zx). We use the same rank- based

---

Under review as a conference paper at ICLR 2022
Dependence fitness on neuron's sizes
Dependence of fitness on neuron's sizes
$2e of neuron's MLP hidden layer
MUP layer
36
0.5
-05
30
30
-1.0
18
-1.5
18
14
2.0
-2.0
10
10
-25
-2.5
7
11 12 14 size neuron's stace
20
10 size of neuron's state
14 16
18
Figure 5: Dependence of fitness neuron's hidden size and neuron's (activation's) update hidden layer size. Left vs Right: 100 vs 300 networks in NES optimizer. Only some experiments suc- ceed finding the solution, those that show good performance. However, succeed with 300 networks. An improvement to finding rules algorithm would also include evolution over parallel running networks, selecting networks (update rules) that find good solutions.
Dependence nes optimization params
-025
0.55
0.2
-0.75
1.00
30
0.1
-1.25
0.04
-1.50
-1.75
0.02
-2.00
0.1
0.2
0.4
1.0
2.0
3.0
mean
Figure 6: Dependence on NES mean and std parameters. In the experiment, the best performing update rules were memorization of sequence length 300, with 100 hidden units. The point in time was taken early enough where difference between the performance can be seen well. We see that std=0.1 1 and mean=1.0 work most robustly.
Table Separable NES (SNES) hyper-parameters.
Haper-par
Value
Mp
1.0
0.1
Pn
0.0
To
0.1
Nsamples
sweep over {128, 256)
12

---

Under review as a conference paper at ICLR 2022
Seq. length: 2000
Seq. length: 10000
-2.2
-2.2
-2.4
Finess
2.1
mh=40
m==300
-2.6
-2.6
mv=00000
-3.0
3.0
80000
10000
20000
40030
60000
00000
100003
Evalution step
step
Figure 7: Language model training curves
Algorithm Separable NES (SNES) update step
Require: Input μt, log o., for k = 1... Nsamples do
Draw sample 8; (O,
0g8k
Evaluate the loss L(za)
end for
sort ((8,, zx)) with respect L(zx) and compute utilities we
compute natural gradients
V VoJ
Hamper k=
k=1
us Sk
ug
{s}
1)
M++1 f- pr + T2 . Vud
update parameters
log 0e+l log o: + Bo/ 2 V
13


Reviews:
[{"review_id": "zFkwWdqOsVu", "reviewer": "Reviewer_3DR8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper deals with an important topic: how can we train neural update rules that are more efficient than general update rules based on approaches such as gradient descent. However, currently, the contributions of the paper are not made clear enough and some crucial comparisons are missing.\n\nHow does the approach compare to related work on evolving Hebbian learning rules with an evolutionary strategy (e.g. Najarro&Risi 2020, \"Meta-Learning through Hebbian Plasticity in Random Networks\")? This approach seems very related to the described work. Additionally, other relevant work that should be compared against includes the mentioned work by (Kirsch & Schmidhuber, 2020) or the more complex model of Bertens and Lee \"Network of evolvable neural units can learn synaptic learning rules and spiking dynamics”. \n\nThe introduction describes the approach as a HyperNetwork, in which a trained function is responsible for the weight updates (also mentioning related work such as HyperNetworks/CPPN), but then only later in the paper is the actual Hebbian approach described. It would be good to see an actual comparison to a HyperNetwork style approach (or rewriting the paper to focus on a local-learning rule approach), even though it might be computational more expensive. \n \nAdditional comments:\n- “In (Stanley & Miikkulainen, 2002) they parameterize the weights by compo- sitional pattern producing networks (CPPN)” -> CPPN were not invented at this point. They evolved the weights and architecture of networks with NEAT”\n\n-\"The N is of the order of ten thousand in our experiments, but in principle it can be arbitrarily long or we should ideally use the entire data-set, this is just the scale we have managed to achieve so far” -> if it works in principle, what are the challenges in scaling to longer sequences?\n\n-\"The standard way to compare models is on their likelihood on a withheld test set.” -> This comparison should also be included", "summary_of_the_paper": "This paper presents an approach that trains local Hebbian learning rules to allow a neural network to perform reasonably well in two types of problems: sequence memorization and prediction. Two approaches to train these models are compared, which are based on meta-gradients and evolutionary strategies. The evolved model is able to perform sequence predictions of length 1000.  ", "summary_of_the_review": "The paper presents an approach based on local learning rules to deal with two tasks: sequence memorization and prediction. The results are promising but crucial comparisons to related approaches are missing, making it difficult to judge the particular contributions of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4PXyTXOIzJ", "reviewer": "Reviewer_2qrk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My first and primary concern with this paper is that most of the experimental results are obtained on the memorization task. The results seem to indicate that the architecture and the evolved meta-parameter configuration are adequate for long working memory problems. It is very hard for me to judge how impressive this feat is. Despite its great focus on this task, little motivation and discussion is provided to understand its importance. Moreover, training details are missing. How is $\\theta$ learned in this case? Are sequences presented twice during a training episode? The section on language modelling experiments is then disproportionally shorter. How does performance compare here against e.g. standard LSTMs?\n\nMy second concern is related to how the paper itself is presented. In general I enjoyed reading the paper but I think its clarity could be strongly improved:\n- The paper often mixes the presentation of the model and training algorithm with remarks about results that will later appear. I found this very distracting, in particular in section 5.\n- I missed a complete and concise set of equations that fully (and \"quickly\") defined the model. Some things are never properly defined. As an example, the delay $\\tau$ is mentioned in the caption of Figure 1 and in the diagram but it is never explicitly defined.\n- By far not enough detail is given when presenting variations of the model, for example involving LSTM weight or activation updates. The appendix could be used for these.\n- Fitness (the y-axis of the plots) is never defined, I assume it is simply the log-likelihood?\n- Details on how hyperparameters were chosen and which ranges were considered should be given for all variants (including e.g. standard LSTM models). Since it is difficult to optimize RNNs with gradient-based methods, how were these choices informed, in particular for standard models?\n- While a few occurrences is understandable, there are currently too many references to data that is not shown (\"we also experimented with ...\").\n\nRelated to issues above, there a few design choices that are not sufficiently well motivated or compared against controls/alternatives. To give examples:\n- How important is it to use the weight transpose in backward connections? It's not clear a priori that \"emulating gradients\" is needed or achieved by NES. And weight transposes are clearly biologically implausible.\n- Since so many variations of the model can be tried, why did the authors choose to study softmax vs. norm(relu(.))? Why is this question especially interesting?\n\nI liked the use of multi-dimensional activations that are then locally combined, as well as the use of different activation channels in the weight update. In fact, some additional biological motivation could be given here, and I think it would greatly strengthen the paper. For example, many neurons (including the ubiquituous pyramidal cell) are best modelled as multicompartmental units and synaptic plasticity does not only depend on the frequency of pre- and post-synaptic firing.\n\nThe text is slightly inaccurate at times:\n- \"[the total] log-likelihood\" -> \"negative log-likelihood\"\n- \"[The first algorithm is] (meta)-gradient\" -> \"(meta)-gradient descent\", or \"(meta)-gradient-based optimization\"?\n\nIf possible the paper should be proofread, as it contains some English mistakes.", "summary_of_the_paper": "The authors design and apply recurrent neural network models with \"fast weights\" to sequence compression problems. The \"slow\" parameters that determine how the \"fast\" weights change are determined by gradient descent or evolution strategies. Experimentally, the latter method appears to perform better for the architectures designed by the authors. The paper focuses on a long-sequence working memory problem and additionally presents some results on language modeling.", "summary_of_the_review": "Approching sequence learning problems with fast-weight architectures is interesting, but the experimental results are not yet convincing enough and the clarity of presentation should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Yb7hHGrGEl", "reviewer": "Reviewer_92Z4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the premise of the paper is interesting, the stage the paper is in, unfortunately, seems not ready for publication. After a major rework (notation and clarity of writing) and additional experiments, it might be acceptable.\n\n\n# Originality\n\nWhile the paper combines existing methods (LSTM, MLP, self-attention PMLP, Hebb rule, etc.), the combination seems novel to me.\n\n## Major\n1. Given the topic and purpose of the paper, I would find a citation of “Learning to Learn Using Gradient Descent“ (Hochreiter, 2001) necessary. Especially w.r.t. section 6.4.\n2. A citation of “Learning to learn by gradient descent by gradient descent” (Andrychowicz, 2016) would be appropriate w.r.t. section 1.1, paragraph “End-to-end learning and meta-learning”.\n\n## Minor\n2. The claim that “with the best models being transformers currently, and LSTM recurrent networks in the not so recent past.” is unsupported and, to my knowledge, wrong. LSTM outperform transformers on sequence learning tasks that involve integrating information over time, such as rain fall-off prediction [“Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks” (Kratzert, 2018)].\n\n\n# Quality\n\n## Major\n3. If I understand correctly, the experiments were only conducted on one task (next-character prediction for books), which seems insufficient to me. I advice the authors to look into either next-character prediction for biological data (DNA, amino acids) or tasks involving math, such as simple calculations that need to be performed. Both tasks allow for very long sequence lengths. With such experiments, the empirical significance could be improved drastically.\n4. When considering long sequences and comparing to LSTM networks, it is very important to clarify if a forget gate is used. A forget gate will re-introduce the vanishing gradients problem and prohibit learning on longer sequences, if not initialized with a large positive bias (so that the gate is forced “open” for many updates). I have not found information on this in the paper, so I assume a forget gate was used without consideration of vanishing gradients.\n\n## Minor\n5. “Being able to evolve learning algorithms without objective might provide a path to intelligent artificial life.” - but you do have an objective for your method, no? Also, how do you define “artificial life”?\n6. Section “Tasks”: “The N is of the order of ten thousand in our experiments, but in principle it can be arbitrarily long or we should ideally use the entire data-set, this is just the scale we have managed to achieve so far.” - that is a bold claim but I don’t see any justification for it?\n7. “This working relies on a generally reasonable assumption that statistics of data in the future\nwill be similar to the current ones. In our approach we don’t make such an assumption - the system\nevolves how it should process the current piece of information in order to do well in the future.” - but that still implies that the future data is similar to the current data. Otherwise how would learning to process the current data help in the future data?\n8. “[...] the search process needs to be able to find relationships that span thousands even possibly millions of steps. Thus, rather than hoping to find such signal by back-propagating over such periods, we can expect evolution to work better: For example, if local update rules are modified by chance to store information, this modification can have immediate effect over long time scales.” - I disagree – a sequence model with an attention mechanism can pinpoint a signal very well. I don’t see why evolution would be expected to work better here.\n9. Please consider plotting the means of your runs with a shaded area for the error bars or best/worst runs instead of only plotting the best and 4th best seed. (It might seem cherry-picked otherwise.)\n10. “The LSTM has much more of these [hyper-weights] than the update rules network[…]” - Please give some ranges here so that it is clear how many more parameters the LSTM has in your case.\n\n\n# Clarity\n\nThe paper seems very unclear to me. Unfortunately, I find the notation inconsistent and hard to read.\n\n## Major\n11. The mixture of Python syntax and math formulas is not very readable. I would suggest introducing variable names or suffices instead of indexing the vectors with h[:, 0], h[:, 1],…, h[:, -1] and giving the indices different meanings.\n12. Please define all abbreviations, even if they are common (MLP, LSTM).\n13. Please define all symbols. (sigma, uppercase W).\n\n## Minor\n14. “For a given connected pair of layers, we do two matrix multiplications (denoted by ·): w^{lk} (that “goes” from layer k to layer l) multiplies the 0-th component h^k[:, 0] of tensor hk and the transpose of w^{kl} (that ”goes” from layer l to layer k) multiplies the 1-st component h^k[:, 1] of hk .”  - I assume you mean that w^{lk} is multiplied by h^k[:, 0]? Or am I missing something here?\n15. Eq. (3) is self-attention or gating – please mention this when you introduce it, it would make understanding it much easier in my opinion.\n16. Are the matrices M are trainable? They are not really described.\n17. Eq. (3) and (4): I would recommend to also show the output of the equation, e.g. w_new = WU() = … .\n18. I would suggest to annotate the matrices and vectors with their sizes (e.g. $w\\in R ^{n \\times m}$).\n19. In the abstract I would suggest to write “recurrent neural network” instead of just “recurrent network” for clarity.\n20. n^l and n_l are the same because of a typo, I assume?\n21. Sometimes you write “sigma tanh” and sometimes “sigma x tanh”, is this on purpose?\n22. “While we experimented with various forms of the update, […]” - I would suggest to list the updates you have considered in your experiments here for clarity.\n23. “(10k steps, 1k units hasn’t converged yet).” - Does this mean it is still running for the rebuttal or just not finished and final?\n\n\n# Significance\n\nI cannot judge the significance of this work, mainly because of missing clarity on points 3. and 4. in “Quality”.", "summary_of_the_paper": "This paper aims at using evolution strategies instead of back-propagation for evolving the parameters of weight and activation updates for an online-learning sequence model, specifically a next-character prediction model. Each neuron activation is represented by a vector, where each vector element has a different purpose, such as matrix multiplication for forward and multiplication with a transposed matrix for recurrent connections. LSTM, MLP, MLP with self attention/gating (PMLP), and Hebb rule were considered for activation and weight updates. Experiments were conducted on a next-character prediction task on long sequences.", "summary_of_the_review": "While the premise of the paper is interesting, I can not recommend it for publication. The paper is very hard to read and suffers from missing clarity. The experimental evaluation is insufficient.\nI would suggest to the authors that they conduct a major rework of the paper (notation and clarity of writing), add additional experiments, and submit to a later conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an interesting problem, namely how to evolve effective weight and activation update rules for online learning of a recurrent network. The work focuses on two specific tasks: character sequence memorisation and prediction. Two approaches based on meta-gradients and evolutionary strategies are explored. Unfortunately the paper is missing some important related works. Moreover, presentation needs to be improved, as well as experimental assessment should be expanded both in terms of tasks and in terms of comparable models presented in the literature.
Decision: Reject