Meta Learning Backpropagation And Improving It
Authors: Louis Kirsch
Jürgen Schmidhuber
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
general-purpose meta-learning
learned learning rules
fast weights
distributed memory
backpropagation
gradient descent
modularity
self-organization

Abstract: Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VSML) unifies the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VSML where the weights of a neural network are replaced by tiny LSTMs allows for implementing the backpropagation LA solely by running in forward-mode. It can even meta learn new LAs that differ from online backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta learned LAs learn through fast association in a way that is qualitatively different from gradient descent.

Meta Learning Backpropagation And Improving It
Louis Kirsch', Jürgen Schmidhuber's 1.2
'The Swiss AI Lab IDSIA, University of Lugano (USI) & SUPSI, Lugano, Switzerland 2 Kiig Abdullah University Science and Technology (KAUST), Thuwal, Saudi Arabia {louis, juergen)Gidsia.co Midsiaddaaa.. ch
Abstract
Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VSML) unifies the above and demonstrates that simple seight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VSML where the weights of a neural network are replaced by tiny LSTMs allows for implementing the backpropagation LA solely by running in forward-mode modee. can even meta learn new LÅs that differ from online backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta learned LAs learn through fast association in way that is qualitatively different from gradient descent.
Introduction
The shift from standard machine learning to meta learning involves learning the learning algorithm (LA) itself, reducing the burden on the human designer to craft useful learning algorithms [43]. Recent meta learning has primarily focused on generalization from training tasks to similar test tasks, e.g., few-shot learning []], or from training environments to similar test environments [17]. This contrasts numan-engineeee LAs that generalize across wide range of datasets environments. Without such generalization, meta learned LAs can not entirely replace numan-engineered variants. Recent work demonstrated that meta learning can also successfully generate more general LAs that generalize across wide spectra of environments 220 BT, e.g., from toy environments to Mujoco and Atari. Unfortunately, however, many recent approaches still rely on a large number of human- -usan-designed and unmodifiable inner-loop components such as backpropagation.
Is it possible to implement modifiable versions of backpropagation or related algorithms as part of the end-to-end differentiable activation dynamics of neural net (NN), instead of inserting them separate fixed routines? Here we propose the Variable Shared Meta Learning (VSML) principle for this purpose. It introduces a novel way of using sparsity and weight-sharing in NNs for meta learning. We build on the arguably simplest neural meta learner, the meta recurrent neural network (Meta RNN) [6. 10 (6). by replacing the weights of a neural network with tiny LSTMs. The resulting system can be viewed as many RNNs passing messages each other, or as one big RNN with a sparse shared weight matrix, or as a system learning each neuron's functionality and its LA. VSML generalizes the principle behind end-to-end differentiable fast weight programmers [45 46 3
41
hyper networks [4], learned learning rules 4 [3 33), and hebbian-like
26 30). Our mechanism, VSMI., implement backpropagation solely in the forward-dynamice of an RNN. Consequently, enables meta-optimization of backprop- like algorithms. We envision future where novel methods of credit assignment can be meta learned while still generalizing across vastly different tasks. This may lead to improvements in sample efficiency, memory efficiency, continual learning, and others. As first step, our system meta
synaptic plasticity
44
46
25
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

learns online LAs from scratch that frequently learn faster than gradient descent and generalize to datasets outside of the meta training distribution (e.g., from MNIST Fashion MNIST). Our VSML RNN is the first neural meta learner without hard-coded backpropagation that shows such strong generalization.
2
Background
Deep learning-based meta learning that does not rely on fixed gradient descent in the inner loop has historically fallen into two categories, 1) Learnable weight update mechanisms that allow for changing the parameters of an NN to implement a learning rule (FWPs/ / LLRs), and 2) Learning algorithms implemented in black-box models such as recurrent neural networks (Meta RNNs).
a standard NN, the weights are updated by fixed LA. This framework be extended meta learning by defining an explicit architecture that allows for modifying these weights. This weight-update architecture augments a standard NN architecture. NNs that generate change the weights of another or the same hypernetworks 14, NNs
Fast weight programmers & Learned learning rules (FWPs LLRs)
NN are known as fast weight programmers (FWPs)
45
46
3
41,,
Often these architectures make use of local Hebbian-lik- update rules or outer- piverrproducts, and we summarize this category as FWPs/ / LLRs. In FWPs / LLRs the variables V5 that are subject learning are the weights of the network, whereas the meta-vane variables VM that implement the LA are defined by the weight-update architecture. Note that the dimensionality of V2 and VιΛ can be defined independently of each other and often VM are reused in coordinate- wise fashion for VZ resulting in [V >> [Varl, where is the number of elements.
with synaptic plasticity [25 26 30 or learned learning rules 13
B3].
Black-box learning in activations (Meta RNNs) It was shown that an RNN such as LSTM can learn to implement an LA [16] when the reward or error is given as an input 47). After meta training, the LA is encoded in the weights of this RNN and determines learning during meta testing. The activations serve the memory used for the LA solution. We refer to this Meta RNNs [6 10 [6]
(Also sometimes referred to as memory- based meta learning.). They are conceptually simpler than FWPs/ LLRs as no additional weighh-update rules with many degrees of freedom need to be defined. In Meta RNNs VL are the RNN activations and Var are the parameters for the RNN. Note that an RNN with neurons will yield IV1 O(N) and |Vrl O 46 This means that the LA is largely overparameterized whereas the available memory for learning is very small, making this approach prone to overfitting (0). a result, the RNN parameters often encode task-specific solutions instead of generic LAs. Meta learning simple and generalizing LA would benefit from
|V/| > VMrl Previous approaches have tried to mend this issue by adding architectural complexity
through additional memory mechanisms [53 29 40 27
3 Variable Shared Meta Learning (VSML)
In VSML we build on the simplicity of Meta RNNs while retaining Vz >> |VMl from FWPs / LLRs. We do this by reusing the same few parameters VM many times in an RNN (via variable sharing) and introducing sparsity in the connectivity. This yields several interpretations for VSML:
VSML as single Meta RNN with sparse shared weight matrix (Figure la). The most general description.
VSML as message passing between RNNs Figuure 1b). We choose a simple sharing and sparsity scheme for the weight matrix such that it corresponds to multiple RNNs with shared parameters that exchange information.
VSML as complex neurons with learned updates (Figure 1c.. When choosing a specific connectivity between RNNs, states activations Vy. of these RNNs can be interpreted the weights of a conventional NN, consequently blurring the distinction between a weight and activation.
Introducing variable sharing to Meta RNNs We begin by formalizing Meta RNNs which often use multiplicative gates such the LSTM 12 15 or its variant GRU (6). For notational simplicity,

---

FRN
IRNN
RAUM
Fh-M
Aute
RMN
W
(a) Viewed as a single RNN (structured weight matrix)
(b) One VSML RNN many sub-RNNs
(c) Viewed as an NN with complex neurons
Figure I: Different perspectives on VSML: (a) A single Meta RNN 16 where entries in the weight
matrix are shared or zero. (a) VSML consists of many sub-RNNs with shared parameters Var passing messages between each other. (c) VSML implements
NN with complex neurons (here neurons). VM determines the nature of weights, how these are used in the neural computation, and the LA by which those are updated. Each weight was represented by the multi-dimensional RNN state Sab R Neuron activations correspond to messages 7i passed between sub-RNNs.
we consider vanilla RNN. Let & E R.N be the hidden state of an RNN. The update for an element j E [I....,N} given by
Sj + [RNN(s); = o - Wi),
Σ
(1)
where a is non-linear activation function, W NNN, and the bias is omitted for simplicity. We also omit inputs by assuming a subset of & to be externally provided. Each application Equation reflects single time tick in the RNN.
We now introduce variable sharing (reusing W) into the RNN by duplicating the computation along two axes size A, B (here A B, which will later be relaxed) giving s E AAXBxX For E A},, .1....,.. B} we have
Sabj fRNN(Sab)s o E Sabi Wiz).
This can be viewed as multiple RNNs arranged on a 2-dimensional grid, with shared parameters that update independent states. Here, chose a particular arrangement (two axes) that will facilitate the interpretation C of RNNs weights.
VSML as message passing between RNNs
The computation so far describes A independent RNNs. We connect those by passing messages (interpretation BB)
Sab + (Sb. m
(3)
where the message n =
with b fl...... A= fm RN N
fed as an
additional input each RNN. This is related Graph Neural Networks 58). Summing over the axis A (elements a') corresponds to an RNN connectivity mimicking those of weights in an NN (to facilitate interpretation ( We emphasise that other schemes based on different kinds of message passing and graph connectivity are possible. For a simple fi defined may equivalently write
by the matrix C' E NxN we
Σ Sabi Wij + E = al E Sabi Wij +
sara d'
Sabj f-
This constitutes the minimal version of VSML with VM == (W, C) and visualized
Figure Ib
VSML as a Meta RNN with sparse shared weight matrix It is trivial to see that with A = 1 and B we obtain single RNN and [Equation 4 recovers the original Meta RNN In the general case, we can derive an equivalent formulation that corresponds to single standard RNN with single matrix W that has entries of zero and shared entries
Sabj f- o Σ
Sodi W Vodiabs), c,d.i
(5)

---

where the six axes can be flattened to obtain the two axes. For Equation 4 and Equation to be equivalent, W must satisfy (derivation in Appendix A
Cgp
if = a (d if f a d bbaaddddbb----a bAc=a. if d b otherwise.
Wiy
W ediabj
(6)
Ci + Way
0,
This corresponds to interpretation (A) with the weight matrix visualized in Figure la To distinguish between the single sparse shared RNN and the connected RNNs, now call the latter sub-RNNs.
VSML as complex neurons with learned updates The arrangement and connectivity of the sub- RNNs as described in the previous paragraphs corresponds to that weights in a standard NN. Thus, interpretation O. VSML can be viewed as defining complex neurons where each sub-RNN corresponds weight in standard NN visualized in Figure Ic. All these sub-RNNS share the same parameters but have distinct states. The current formulation corresponds to a single NN layer that is recurrently. We will generalize this other architectures in the next section. A corresponds to the dimensionality of the inputs and B to that of the outputs in that layer.
The role of weights in standard neural network now assigned to the states of RNNs. This allows these RNNs define both the neural forward computa- tion well as the learning algorithm that determines how the network is updated (where the mechanism shared across the network). In the case of backprop- agation, this would corre- spond to the forward and backward passes being im- plemented purely in the re- current dynamics. We will demonstrate the practical feasibility of this in [Sec- tion 3.2 The emergence of RNN states as weights quickly leads to confusing terminology when RNNs have 'meta weights'. Instead, we simply refer meta variables VM and learned variables Vz.
A m
LSTMs with state
8 1
with state d
GD
Layer
classes
Layer 1
Redrectional
Outputs
forward
forward dimmesiional cimensional
mag.
backwward
Enar outputs
The neural interpretation of VSML replaces all weights of standard NN with tiny LSTMs using shared parameters (resembling complex neurons). This allows these LSTMs to define both the neural forward computation as well the learning algorithm that determines how the network is updated. Information flows forward and back- ward in the network through multi-dimensionall messages 7 and fn, generalizing the dynamics of an NN trained using backpropagation.
Figure 2:
With this interpretation, VSML can be seen as generalization of learned learning rules and Hebbian-like differentiable mechanisms or fast weights more generally 44 46
where RNNs replace explicit weight updates. In standard NNs, weights and activations have multiplicative
25
26
interactions. For VSML RNNs to mimic such computation we require multiplicative interactions between parts of the state s. Fortunately, LSTMs already incorporate this through gating and can be directly used in place of RNNs.
Stacking VSML RNNs and feeding inputs
To get structure similar to one of the non-recurrent deep feed-forward architectures (FNNs), we stack multiple VSMI. RNNs where their states are untied and their parameters tied I This is visualized with two layers in Figure 2 where the states s(2) of the second column of sub-RNNs are distinct from the first column s(1), The param- eters (k) and ((k) describing layer sizes can then be varied for each layer k E (1,..., K} constrained by (k)) = B(k-1), The updated Equation 3 with distinct layers k is given by ab fRNN(S (k) mb where m,
(k++)) Σx. (s (k)) with b E (1,..,, (bk) A(++1).
'The resultant architecture whole is still recurrent. Note that even standard FNNs are recurrent if the LA (backpropagation) is taken into account.

---

To prevent information from flowing only for- ward in the network, we use an additional back- ward message
=
B
A
A
LISTTA
LSTM
ab t -RNNN(S (R)
(1) m (
LSTM
where ((-11) = En (s with a E
{1.... A = B(k-1)) (visualized in Fig- ure 3p. The backward transformation given by fm RRV. R
LSTM
LSTM
I
LSTT
IF
LSTM
Similarly, other neural architectures can be ex- plicitly constructed (e.g. convolutional NNs, Section B.2). Some architectures may be learned implicitly if positional information is fed into each sub-RNN (Appendix C). We then update all states s(k) in sequence I..... K to mimic sequential layer execution. We may also apply multiple RNN ticks for each layer h.
Figure 3: VSML with forward messages nt and backward messages in to form a two-layer NN with shared LSTM parameters but distinct states.
To provide the VSML RNN with data, each time we execute the operations of the first layer, a single new datum æ IR A(1) (c.g. one flattened image) is distributed across all sub-RNNs. In our present experiments, we match the axis A(1) to the input datum dimensionality such that each dimension (e.g., pixel) is fed to different RNNs. This corresponds to initializing the forward message
7 La (padding It with zeros if necessary). Similarly,
read the output ý E R B(K) from
Uo == mh al (K+1) Finally, we feed the error E B(K) at the output such that i1
{(K) := eg. See
Figure 2 for a visualization. Alternatiely, multiple input or output dimensions could be patched
together and fed into fewer sub-RNNs.
3.1 Meta learning general-purpose learning algorithms from scratch
Having formalized VSML, we can now use end-to-end meta learning create LAs from scratch in Algorithm I, We simply optimize the LSTM parameters Vas to minimize the sum of prediction
losses many time steps starting with random states Vz (k() }. We focus meta learning online LAs where one example is fed at time as done in Meta RNNs [16] [66 [0]. Meta training may be performed using end-to-end gradient descent or gradient-free optimization such as evolutionary
strategies [57 38 The latter is significantly more efficient on VSML compared to standard NNs due the small parameter space VM. Crucially, during meta testing, no explicit gradient descent used.
Algorithm 1 VSML: Meta Training
Require:
Dataset(s) D = { 31)
Vi f- initialize LSTM parameters
while meta loss has not converged do
Outer loop in parallel over datasets D
Vz (k) initialize LSTM states Va, 6,
for (z, y) ab
E (1.1.V),..... (rrr c D do 7 Ta Va
b> Inner loop over T examples
Initialize from input image > Iterating over K layers
for l E {l...., do
(k) t fRNN(S (k) (1))
Va,
Equation
mp (k+1) ; =Ew (8/b))
(k)
Vb
Create forward message
mo (k-1) = Ev ( Va.
> Create backward message
9. 7 (K+1) Va
Read output
e = V,I(D.1)
b Compute error at outputs using loss
1- (K)
Eb vb
Input errors
Vy + V/M - a V vir DT, L(g(t), v(t),, obtaining Vvy either by back-propagation through the inner loop
evolution strategies, using search distribution p(VM)
5

---

3.2 Learning to implement backpropagation in RNNs
An alternative to end-to-end meta learning is to first investigate whether the VSML RNN can implement backpropagation. Due to the algorithm''s 's ubiquitous use, it seems desirable to be able to meta learn backpropagation- like algorithms. Here investigate how VSML RNNs can learn to implement backpropagation purely in their recurrent dynamics. We do this by optimizing Vas (I) store a weight w and bias b as a subset of each state Sab. (2) compute to implement neural forward computation, and (3) update w and b according to the backpropagation algorithm [23]
y tanh(z))
We call this process learning algorithm cloning and it is visualized in Figure 4
Figure 4: To implement back- propagation we optimize the VSML RNN to use and update weights and biases b as part of the state Sab in each sub- RNN. Inputs are pre-synaptic x and error e. Outputs are post-synaptic ý and error é'.
We designate an element of each message ma m fm ($ (k)) fm (b)) as the input x, error e, and output ý and error é'. Similarly, we set w := Sab1 and 6 == 9ab2. We then optimize VM via gradient descent to regress ÿ. Aw. Ab, and è' toward their respective targets. We can either generate the training dataset D {zz, w. b, y, e, e')} randomly or run a 'shadow' NN on some supervised problem and fit the VSML RNN its activations and parameter updates. Multiple iterations in the VSML RNN would then correspond to evaluating the network and updating it via backpropagation. The activations from the forward pass necessary for credit assignment could be memorized as part of the state or be explicitly stored and fed back. For simplicity, we chose the latter to clone backpropagation. We continuously run the VSML RNN forward, alternately running the layers in order ..... K and in reverse order K K....,
4 Experiments
Leaminng distribution)
Learning on Fashion
First, we demonstrate the capabilities of the VSML RNN by showing that it can implement
Cloned BP (shallow)
Regu lar SGD (shallow) Cloned
Transng toa
neural forward computation and backpropaga-
Regu Ideap)
tion in its recurrent dynamics on the MNIST [21]
8.0
(59] dataset. Then, show how we can meta learn an LA from scratch on one set of datasets and then successfully ap- ply it to another (out of distribution). Such generalization is enabled by extensive variable sharing where we have very few meta variables [vll 2, 400 and many learned variables |V.. R 257, 200. We also investigate the robust- ness of the discovered LA. Finally, we introspect the meta leamed LA and compare it to gradient descent.
and Fashion MNIST'
0.5
30x
20k
30k
0030c
1.56
2.44
30kc
40c
Gradient stop
Cradient stop
Figure 5: The VSML RNN is optimized to im- plement backpropagation in its recurrent dynam- ics on MNIST, then tested both MNIST and Fashion MNIST where it performs learning purely by unrolling the LSTM. We test shallow and deep architectures (single hidden layer of 32 units). Standard deviations are over seeds.
Our implementation uses LSTMs and the message interpretation from Equation Hyperparameters, training details, and additional experiments can be found in the appendix.
4.1 VSML RNNs can implement backpropagation
As described in Section 3.2. we optimize the VSML RNN to implement backpropagation. We structure the sub-RNNs to mimic a feed-forward NN with either one hidden layer or no hidden layers. To obtain training targets, we instantiate a standard NN. the shadow network, and feed MNIST data. After cloning, we then run the LA encoded in the VSML RNN on the MNIST and Fashion MNIST dataset and observe that performs learning purely its recurrent dynamics, making explicit gradient calculations unnecessary. Figure 5) shows the learning curve on these two datasets. Notably,
2 Executing layers in reverse order is not strictly necessary as information always also flows backwards through br but makes LA cloning easier.

---

learning works both on MNIST (within distribution) and on Fashion MNIST (out of distribution). We observe that the loss is decently minimized, albeit regular gradient descent still outperforms our cloned backpropagation. This may be due to non-zero errors during learning algorithm cloning, in partieular when these errors accumulate in the deeper architecture. is also possible that the VSMI. states (" ('weights') deviate too far from ranges seen during cloning, in particular in the deep case when the loss starts increasing. We obtain 87% (deep) and 90% (shallow) test accuracy on MNIST and 76% (deep) and 80% (shallow) on Fashion MNIST (focusing successful cloning over performance).
4.2 Meta learning supervised learning from scratch
In the previous experiments, we have estab- lished that VSML is expressive enough to meta- optimize backpropagation-lii
Meta Testing (within distribution)
Testing Fashion (out distribution)
1.0
algorithms. In- stead of cloning an LA, now meta learn from
Cumaativee accurace
0.5
scratch as described in [Section 3.1. Here, we use single layer (K 1) from input to output dimension and run it for two ticks per image with N = 16 and N' N" = 8. First, the VSML RNN is meta trained end-to-end using evolutionary strategies (ES) [3 on MNIST to minimize the sum of cross-entropies over 500 data points starting from random state initializa- tions. As each image is unique and VM can not memorize the data, we are implicitly optimizing the VSML RNN to generalize to future inputs given all inputs it has seen so far. We do not pre-train VM with a human -ammanengieeeredd LA.
0.4
VSML Backprop
Backgrop Adam
0.0
2000
1500
1000 1500
examples
exmmplxxmpeseee seen
Figure 6: The VSML RNN can be directly meta trained on MNIST to minimize the sum of errors when classifying online starting from random state initialization. This allows for faster learning during meta testing compared to online gradient descent with Adam on the same dataset and even generalizes
different dataset (Fashion MNIST). In comparison, a standard Meta RNN [16] strongly overfits in the same setting. Standard deviations are over 128 seeds.
During meta testing on MNIST (Figure 6) we plot the cumulative accuracy on all previous in- puts on the axis & Ei= Cr after example T
with binary 6, indicating prediction correctness). For each example, the prediction when this example was fed to the RNN is used, thus measuring sample efficient learning. This evaluation protocol is similar to the one used in Meta RNNs [56. [0] We observe that learning is considerably faster compared to the baseline of online gradient descent (no mini batching, the learning rate appropriately tuned). One possibility that VSML simply overfits to the training distribution. We reject this possibility by meta testing the same unmodified RNN on different dataset, here Fashion MNIST. Learning still works well, meaning we have meta learned a fairly general LA (although performance at convergence still lacks behind little). This generalization is achieved without using any hardcoded gradients during meta testing purely by running the RNN forward. In comparison to VSML, a Meta RNN heavily overfits.
4.3 Robustness to varying inputs and outputs
Meta Testing MNIST (within distribution)
Meta Testing Fashion MNIST (out of distribution)
A defining property of VSML is that the same parameters Va can be used to learn on varying input and output sizes. Further, the architecture and thus the meta learned LA is invariant to the order of inputs and outputs. In this experiment, we investigate how robust we are to such changes. We meta train across MNIST with 3, 4, 6, and classes. Likewise, we train across rescaled versions with 14x 14, 28x28, and 32x32 pixels. We also ran- domly project all inputs using a linear transformation, with the transforma- tion fixed for all inner learning steps. In
Referrecc
(unseen))
40w48 inout (unsean)
21x21 Input junseen)
Random projection (unscen)
Shuflled impuis (unseen)
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Accuracy of 2k examples
Accurasy of first 24 exomoles
Figure 7: The meta learned learning algorithm is robust to permutations and size changes in the inputs and outputs. All six configurations have not been seen during training and perform comparable the unchanged reference. Standard deviations are over 32 seeds.
Figure 7 we meta test on 6 configurations that were not seen
7

---

Teeling
NUSTT
ST
Sucuatt NN
Traniin www.nnggaaaaaa.
Rancom
VSM.
Molanaa
HeatlanT
EVMtemory
dhullww
ST
2003
Leave out Shizuhh NN ST
Joave out Handom
Mata Teaking en Ku NINST
Reta Tasting on Random
Mala Testing on Sum Sign
Tectlon VNNT
DNUST
Hela OatANds
KucssttiMM
Bandom
Loave out HNST
Leaee out Facrive MENTT
EHNST
Jaave out Random All dacaseta
Accuracy first examples
Accuracy of Sral 25
Accuracy of frus 26
Figure 8: Online learning on various datasets. Cumulative accuracy in % after having seen 2k training examples evaluated after each prediction starting with random states (VSML, Meta RNN, HebbianFW, FWMemory) or random parameters (SGD). Standard deviations are over 32 meta test training runs. Meta testing is done on the official test set of each dataset. Meta training is on subsets of datasets excluding the Sum Sign dataset. Unseen tasks, most relevant from a general-purpose LA perspective, are opaque.
during meta training. Performance on all of these configurations is comparable to the unchanged reference from the previous section. In particular, the invariance to random projections suggests that we have meta learned a learning algorithm beyond transferring learned representations [compare
54 55
4.4 Varying datasets
To better understand how different meta training distributions and meta test datasets affect VSML RNNs and our baselines, we present several different combinations in Figure 8) The opaque bars represent tasks that were not seen during meta training and are thus most relevant for this analysis. This includes four additional datasets: (1) Kuzushiji MNIST [7] with 10 classes, (2) EMNIST [9] with 62 classes, (3) A randomly generated classification dataset (Random) with 20 data points that changes with each step in the outer loop, and (4) Sum Sign which generates random inputs and requires classifying the sign of the sum of all inputs. Meta training is done over 500 randomly drawn samples per outer iteration. Each algorithm is meta trained for 10k outer iterations. Inputs are randomly projected as in [Section 4.3 (for VSML; the baselines did not benefit from these augmentations). We again report the cumulative accuracy on all data seen since the beginning of meta test training. We compare to SGD with a single layer, matching the architecture of VSML, and a hidden layer, matching the number of weights to the size of V2 in VSML. We also have included Hebbian fast weight baseline (25] and an external (fast weight) memory approach 42).
We observe that VSML generalizes much better than Meta RNNs, Hebbian fast weights, and the external memory. These baselines overfit to the training environments. Notably, VSML even generalizes to the unseen tasks Random and Sum Sign which have no shared structure with the other datasets. In many cases VSML's performance is similar to SGD but a little more sample efficient in the beginning of training (learning curves in Appendix B). This suggests that our meta learned LAs are good at quickly associating new inputs with their labels. We further investigate this in the next Section 5]
8

---

5 Analysis
Crepesstdd digttt
Given that VSML seems to learn faster than online gra- dient descent in many cases
0:15
0.00
0:92
0.20
0.60
0.00
0.00
E:00
8:20
0:22
0:31
0.00
0:00
2:00
E.GH
D/03
0.20
0.00
6000
6.00
0:25
9:00
1:15
D/R3
DUR2
0.20
9.03
5.07
8:00
we would like to qualita- tively investigate how learn-
Det p0000 FM
0:59
RKI
RUDS
0.20
ing differs. We first meta train on the full MNIST dataset as before. During meta testing, we plot the output probabilities for each digit against the number of samples seen in Figure We highlight the ground truth input class O well as the predicted class O. In this case, our meta test dataset consists of MNIST digits with two examples of each type. The same digit is always repeated twice. This allows us to observe and vi- sualize the effect with only a few examples. We have done the same introspection with the full dataset in App pendix B
0.13
222
C:4
C-22
6:22
0.00
3:00
S.0G
D:02
0:28
DUSD
0.20
0.00
60
4.
R14
C:IT
0:33
S:P1
6:11
6:20
3.00
1000
Total HATTIR FAAn
SGD
Adhm
digits)
2.0
0:03
0:20
0:00
3:00
3000
0:00
D:03
0:20
0.28
D
0.20
2.00
0.00
10
0.00
0.00
0:20
D
0.20
D
0.CO
2.00
6.00
0:01
0:33
0:32
0.00
0.00
0.00
2:00
0.00
0:23
0
0.22
D4
6:25
0.33
GER
0:20
0.02
0
6/02
DF
0
0:53
0:00
0.50
0.50
8:93
0:93
0:20
DC
0
3.09
2
COL
COLL
0:22
0.02
0.00
0.00
3.00
2.00
5.00
8:00
0:20
0.28
0.02
0.53
2.11
6.10
9:00
Figure 9: Introspection of how output probabilities change after ob- serving input and the prediction error when meta testing on MNIST with two examples for each type. We highlight the ground truth class D well as the predicted class O The top plot shows VSML quickly associating the input images with the right label, almost always making the right prediction the second time with high confidence. The bottom plot shows the same dataset processed by SGD with Adam which fails to learn quickly.
We observe that in VSML almost all failed predictions are followed by the correct prediction with high cer- tainty. In contrast, SGD makes many incorrect predictions and fails to adapt correctly in only 20 steps. seems that SGD learns qualitatively different from VSML. The VSML RNN meta learns to quickly associate new inputs with their class whereas SGD fails to do so. We tried several different SGD lcarning rates and considered multiple steps the same input. In both cases, SGD does not behave similar to VSML, either lcarning much slower or forgetting previous examples. As evident from high accuracies in Figure 8. VSML does not only memorize inputs using this strategy of fast association, but the associations generalize to future unseen inputs.
6 Related Work
Memory based meta learning (Meta RNNs) Memory-based meta learning in RNNs 16 TO 56 is
simple neural meta learner (see SSection 2.. Unfortunately, the LA encoded in the RNN parameters is largely overparameterized which leads to overfitting. VSML demonstrates that weight sharing can address this issue resulting in more general-purpose LAs.
Learned Learning Rules/ / Fast Weights NNs that generate or change the weights of another or the same NN are known as fast weight programmers [44), hypernetworks [14), NNs with synaptic plasticity [25] or learned learning rules [4] (see /Section 2}. In VSML we do not require explicit
architectures for weight updates weights are emergent from RNN state updates. In addition to the
learning rule, we implicitly learn how the neural forward computation is defined. Concurrent to this work, fast weights have also been used to meta learn more general LAs [39].
Learned gradient- based optimizers Meta learning has been used to find optimizers that update the parameters of model by taking the loss and gradient with respect to these parameters as an input [34 2 22 24 In this work, we are interested in meta learning that does not rely on fixed gradient calculation in the inner loop.
9

---

Discrete program search An interesting alternative to distributed variable updates in VSML is meta learning via discrete program search 48 351. In this paradigm. a separate programming language needs to be defined that gives rise to neural computation when its instructions are combined. This led to the automated rediscovery of backpropagation 35 programming language is not required and purrrol- LAs can be discovered and encoded in variable-shared RNNs. Search over neural network parameters is usually easier compared to symbolic program search due to smoothness in the loss landscape.
VSML we demonstrate that a symbolic
Multi-agent systems In the reinforcement learning setting multiple agents can be modeled with shared parameters [30 32 [8]. also in the context of meta learning [36]. This is related to the variable sharing in VSML depending on how the agent-environnee
boundary is drawn. Unlike these works, we demonstrate the advantage of variable sharing in meta learning more general-purpose LAs and present a weight update interpretation.
Discussion and Limitations
The research community has perfected the art of leveraging backpropagation for learning for a long time. At the same time, there are open questions such as how to minimize memory requirements, effectively learn online and continually, learn sample efficiently, learn without separate backward phases, and others. VSML suggests that instead of building on top of backpropagation as a fixed routine, meta learning offers an alternative to discover general- purpose LAs. Nevertheless, this paper is only a proof of concept- until now we have only investigated small-scale problems and performance does yet quite match the mini-batched setting with large quantities of data. In particular, we observed premature convergence of the solution at meta test time which calls for further investigationss Scaling our system to harder problems and larger meta task distributions will be important future work.
The computational cost of the current VSML variant is also larger than the one of standard back- propagation. we run a sub-RNN for each weight in a standard NN with W weights, the cost is in O(W )²), where N is the state size of sub-RNN. If N is small enough, and our experiments suggest small N may be feasible, this may be an acceptable cost. However, VSML is not bound to the interpretation of sub-RNN one weight. Future work may relax this particular choice.
Meta optimization also prone to local minima. In particular, when the number of ticks between input and feedback increases (e.g. deeper architectures), credit assignment becomes harder. Early experiments suggest that diverse meta task distributions can help mitigate these issues. Additionally, learning horizons are limited when using backprop- based meta optimization. Using ES allowed for training across longer horizons and more stable optimization.
VSML can also be viewed as regularizing the NN weights that encode the LA through a representa- tional bottleneck. is conceivable that LA generalization as obtained by VSML can also be achieved through other regularization techniques. Unlike most regularizers, VSML also introduces substantial reuse of the same learning principle and permutation invariance through variable sharing.
Conclusion
We introduced Variable Shared Meta Learning (VSML), a simple principle of weight sharing and sparsity for meta learning powerful learning algorithms (LAs). Our implementation replaces the weights of neural network with tiny LSTMs that share parameters. We discuss connections to meta recurrent neural networks, fast weight generators (hyper networks), and learned learning rules.
Using learning algorithm cloning, VSMI RNNs can learn to implement the backpropagation algo- rithm and its parameter updates encoded implicitly in the recurrent dynamies. On MNIST it learns to predict well without any human-designed explicit computational graph for gradient calculation.
VSML can meta learn from scratch supervised LAs that do not explicitly rely on gradient computation and that generalize to unseen datasets. Introspection reveals that VSML LAs learn by fast association in way that qualitatively different from stochastic gradient descent. This leads gains in sample efficiency. Future work will focus on reinforcement learning settings, improvements of meta learning. larger task distributions, and learning over longer horizons.
10

---

Acknowledgements
We thank Sjoerd van Steenkiste, Imanol Schlag, Kazuki Irie, and the anonymous reviewers for their comments and feedback. This work was supported by the ERC Advanced Grant (no: 742870) and computational resources by the Swiss National Supercomputing Centre (CSCS, projects s978 and s1041). We also thank NVIDIA Corporation for donating several DGX machines as part of the Pioneers of Al Research Award, IBM for donating a Minsky machine, and weights & biases [3] for their great experiment tracking software and support.
References
[1] F. Alet, M. F. Schneider, T. Lozano-Perez, and L. P. Kaelbling. Meta-learning curiosity algorithms. In International Conference on Learning Representations, 2020.
[2] M. Andrychowicz. M. Denil, S. G. Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, B. Shilling- ford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3988-3996, 2016.
[3] J. Ba, G. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using Fast Weights to Attend to the Recent Past. In Advances in Neural Information Processing Systems, pages 433 4331-4339, 2016.
[4] Bengio, Y. Bengio, Cloutier, and Gecsei. On the optimization
synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pages 6-8, 1992.
[5] L. Biewald. Experiment Tracking with Weights and Biases, 2020. URL https: /twp:/www.......
com/
[6] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and
Y. Bengio. Learning Phrase Representations using RNN Sacoder-Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1734, 6 2014.
[7] T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep Learning for Classical Japanese Literature, 2018
[8] E. F. Codd. Cellular automata. Academic press, 2014.
[9] G. Cohen, Afshar, J. Tapson, and A. V. Schaik. EMNIST: Extending MNIST to handwritten letters. 2017 International Joint Conference on Neural Networks (LJCNN), 2017. doi: 10. 109/ jjenn.0017.7666217.
[10] Y. Duan, Schulman, X. Chen, P. L.. Bartlett, Sutskever, and P. Abbeel. RI.22 Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint arXiv: (6l:166170777 2016.
[11] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Von Volume 70, pages 1126-1135, 2017.
[12] A. Gers, Schmidhuber, and F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10)22451-2471 2000.
[13] K. Gregor. Finding online neural update rules by learning to remember. arXiv preprint arXiv:200.033/24, 2020.
[14] D. Ha, A. Dai, and Q V. Le. HyperNetworks. In International Conference on Learning Representations, 2016.
[15] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 1997.
[16] Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, 2001.
[17] R. Houthooft, R. Y. Chen, P. Isola, B. C. Stadie, F. Wolski, J. Ho, and P. Abbeel. Evolved Policy Gradients. In Advances in Neural Information Processing Systems, pages 5400-5409, 2018.
[18] W. Huang, Mordatch, and Pathak. One Policy to Control Them All: Shared Modular Policies for Agent-Agnostt Control. In International Conference on Machine Learning, 2020.
11

---

[19] L. Kirsch, J. Kunze, and D. Barber. Modular Networks: Learning to Decompose Neural Computation, Advances in Neural Information Processing Systems, 2018.
[20] L. Kirsch, S. van Steenkiste, and J. Schmidhuber. Improving Generalization in Meta Re- inforcement Learning using Learned Objectives. In International Conference Learning Representations, 2020.
[21] Y. LeCun, C. Cortes, and C. J. Burges, MNIST handwritten digit database. ATT Labs [Online]. Available: http://yyann www.ee...o//e///////iiiiieeeeeee 2, 2010.
[22] K. Li and Malik. Learning to Optimize, arXiv preprint (eGi:v666.011500 2016.
[23] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. PhD thesis, Univ. Helsinki, 1970.
[24] L. Metz, N. Maheswaranathan, C. D. Freeman, B. Poole, and Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009.1122
9 2020.
[25] T. Miconi, J. Clune, and K. O. Stanley. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, 2018.
[26] Miconi, A. Rawal, Clune, and K. O. Stanley. Backpropamine: training self- aefffyniiy neural networks with differentiable neuromodulated plasticity. In International Conference on Learning Representations, 2019.
[27] N. Mishra, M. Rohaninejad, and X. U. Chen Pieter Abbeel Berkeley. A Simple Neural Attentive Meta-Learner. In International Conference on Learning Reprexentations, 2018.
[28] A. Mordvintsev, E. Randazzo, Niklasson, and M. Levin, Growing neural cellular automata. Distill, 5(2))e23, 2020.
[29] M. C. Mozer and S. Das. A connectionist symbol manipulator that discovers the structure of context-free languages. In Advances in neural information processing systems, pages 863-870, 1993.
[30] E. Najarro and S. Risi. Meta-Learning through Hebbian Plasticity in Random Networks. Advances in Neural Information Processing Systems, 2020.
[31] J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. van Hasselt, S. Singh, and D. Silver. Discovering Reinforcement Learning Algorithms. arXiv preprint arXiv:2007.08794 2020.
[32] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros. Learning to Control lafe-Assemlling Morphologies: Study of Generalization via Modularity. In Advances in Neural Information Processing Systems, 2019.
[33] E. Randazzo. E. Niklasson, and A. Mordvintsev. MPLP: Learning a Message Passing Learning
Protocol. arXiv preprint arXiv:2007.00977 2020.
[34] S. Ravi and H. Larochelle, Optimization as model for few-shot learning. In International Conference on Learning Representations, 2016
[35] E. Real, C. Liang, D. R. So, and Q. V. Le. AutoML-Zero: Evolving Machine Learning Algorithms From Scratch. In International Conference on Machine Learning, pages 8007-8019, 2020.
[36] M. Rosa, O. Afanasjeva, S. Andersson, J. Davidson, N. Guttenberg, P. Hlubuček, M. Poliak, J. Vitku, and J. Feyereisl. BADGER: Learning (Learn [Learning Algorithms] through Multi-Agent Communication). arXiv preprint arXiv: 01511, 2019.
[37] C. Rosenbaum, T. Klinger, and M. Riemer. Routing Networks: Adaptive Selection of Non- Linear Functions for Multi-Task Learning. In International Conference on Learning Represen- tations, 2018
[38] T. Salimans, J. Ho. X. Chen, S. Sidor, and I. Sutskever. Evolution Strategies as Scalable Alternative to Reinforcement Learning. arXiv preprint 1733v31703.03864, 3 2017. ISSN ISSN 0302-9743. doi: 10.1.1.51.6328.
[39] M. Sandler, M. Vladymyrov, A. Zhmoginov, N. Miller, A. Jackson, T. Madams, and others. Meta-Learnig Bidirectional Update Rules. arXiv preprint arXiv:2104.04657 2021.
12

---

[40] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta- Meta-Learning with Keemor-Augmentter Neural Networks. International conference on machine learning, pages 1842-1850, 2016.
[41] I. Schlag and J. Schmidhuber. Gated Fast Weights for On-The-Fly Neural Program Generation.
In NIPS Metalearning Workshop, 2017.
[42] I. Schlag, T. Munkhdalai, and J. Schmidhuber. Learning Associative Inference Using Fast Weight Memory. In International Conference on Learning Representations, 2021.
[43] Schmidhuber. Evolutionary principles in self-referential learning. Diploma thesis, Institut für Informatik, Technische Universität München, 1987.
[44] J. Schmidhuber: Learning to Control Fast- Weight Memories: An Alternative to Recurrent Nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universitär München, 3 1991.
[45] J. Schmidhuber. Learning to Control Fast-Weiise Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation, 4(1):131-139, 1992. ISSN 0899-7667. doi: 10.1 10.1162/ncco.19922 1.9.4.1..11.
[46] J. Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks, pages 460-463. Springer, 1993.
[47] J. Schmidhuber. A self-referential weight matrix. In Proceedings the International Conference on Artificial Neural Networks, Amsterdam, pages 446-451. Springer, 1993.
[48] J. Schmidhuber. Discovering Problem Solutions with Low Kolmogorov Complexity and High Generalization Capability. Technical Report FKI-194-94, Fakultät für Informatik, Technische Universität München, 1994.
[49] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv preprint arXiv:1701.06538 2017.
[50] Sims, Karl. Evolving Virtual Creatures. ACM SIGGRAPH, 1994.
[51] A. Sperduti. Encoding labeled graphs by labeling raam. Advances in Neural Information Processing Systems, pages 1 1125-1125, 1994.
[52] S. Sudhakaran, D. Grbic, S. Li, A. Katona, E. Najarro, C. Glanois, and S. Risi. Growing 3d arte- facts and functional machines with neural cellular automata. arXiv preprint arXiv:2103.08737, 2021.
[53] G. Z. Sun. Neural networks with external memeory stack that learn context-free grammars from examples. In Proceedings of the Conference on Information Science and Systems, 1991, pages (49-653. Princeton University, 1991.
[54] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada,
K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-Dataset: Dataset of Datasets for Learning to Learn from Few Examples. 3 2019.
[55] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang. Cross-domain few-shot classification via learned feature- wise transformation. arXiv preprint arXiv:2001.08735, 2020.
[56] J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Ku- maran, and M. Botvinick. Learning reinforcement learn. arXiv preprint arXiv: 161 1.05663, 2016.
[57] D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber, Natural Evolution Strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pages 3381 3381-3387, 2008.
[58] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[59] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST Novel Image Dataset for Benchmarking Machine Learning Algorithms. CoRR, htbs//708.. 0, 2017.
13


Reviews:
[{"review_id": "vnEadZRyoMn", "reviewer": "Reviewer_9hFV", "summary": "The paper proposes to meta-learn the backpropagations by representing weights of a network by small RNNs. Different RNNs share the weights, and the weights can be thought of as representing the states of RNN. The approach can thus implement back-propagation solely in the forward dynamics of an RNN. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the general idea of the paper but have a few concerns regarding the approach and its motivation - \n\n1. The paper uses evolutionary algorithms to train their model. But evolutionary algorithms can also be used to directly learn the weights of a neural network, and it has been shown to achieve nice results.[1] What's the advantage of using VSML as an alternative to back-propagation?\n\n2. For meta-learning can the authors show results over Omniglot dataset, which is considered a more standard dataset for meta-learning.\n\n3. Since VSML is computationally expensive, what's the advantage of using it instead of back-propagation? \n\n[1] Designing neural networks through neuroevolution. Stanley et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors haven't addressed it. Though I don't feel that the paper would have any negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "vZf6KvaQFP4", "reviewer": "Reviewer_oKHU", "summary": "This paper proposes VSML RNN, an architecture for RNNs that is suitable for meta-learning problems in which the RNN activations represent task-specific quantities and RNN weights represent task-independent quantities. The authors use the insight that RNNs for meta learning (meta RNNs as called in the paper) perform better when the meta-parameters are constrained relative to the task-specific parameters. VSML RNN is essentially an RNN with a particular weight sharing and sparsity structure. The authors demonstrate that VSML RNN can implement *within its hidden state* backpropagation on a smaller model, where the parameters of the smaller model are represented in the VSML RNN's hidden state. Empirically, the authors show that the learning algorithm implemented by a VSML RNN can outperform baselines, and can learn faster than gradient descent. The authors also demonstrate that the learning algorithm is transferable to unseen problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe proposed approach appears novel. Unlike prior work on meta RNNs, the authors recognize that overparameterization of RNNs can lead to learning algorithms implemented by the RNN to overfit to the problems on which it is trained. The proposed method is explicitly designed to avoid such overfitting. Unlike prior work on learning learning rules, VSML RNN does not impose a particular architecture on  the task-specific learning algorithm.\n\n**Quality**\nThe theoretical claims in the paper appear well supported. However, the experimental results are limited, particularly when it comes to comparisons with other meta RNN approaches. The paper would be strengthened by comparing VSML RNN to meta RNNs other than a standard meta RNN- for example, adding memory mechanisms to the RNN. Moreover, interpreting the learning algorithm implemented by VSML RNN would be helpful. The current analysis views the output of the model over the course of training, but finding a way to interpret the internal structure of VSML RNN activations would provide insight into the nature of the learned learning algorithm.\n\nFurthermore, it is stated in the introduction that the results may be relevant in the field of biologically plausible learning algorithms. It would help if the authors could state how VSML RNN could be relevant here. Specifically, (how) can VSML RNN be used to discover biologically plausible learning rules in a way that existing approaches cannot? It seems that VSML RNN may be less useful for this compared to other approaches because it imposes fewer constraints on the form of the learning algorithm implemented by the RNN.\n\n\n**Clarity**\nVSML RNN is a difficult concept to understand- this makes clarity of the paper important. Unfortunately, there are some areas of the paper that are unclear or could be better explained. Specifically:\n1) The authors state in section 3 that \"A simple conceptual way of understanding VSML is taking a standard recurrent or feed-forward architecture and replacing all weights with small RNNs.\" This is hard to interpret since a weight is a parameter and a small RNN is a function- they do not seem comparable.\n2) The distinction between the view of VSML RNN as an RNN (where s is a hidden state) and the view of it as implementing a learning algorithm on a neural network (where s could be a parameter) is not very clear. This is partly because section 3 tries to introduce both viewpoints: the equations look like typical RNN equations and suggest the first view, while Figure 1 encourages the second view. It may be helpful to structure the section to introduce the two viewpoints separately and then explain the mapping between them.\n3) Visually illustrating the indexing of s would be helpful. Initially, it is not clear why the indexing in Equation 3 for example is constructed the way it is.\n4) Similarly, visually illustrating how stacking VSML RNNs works would be helpful. Figure 1 is not very clear- the distinctions and equivalences between VSML RNN states, parameters, messages and graph edges is not visually apparent.\n\n\n**Significance**\nIf the experimental results can be improved, VSML RNNs may become the state-of-the-art approach in the field of meta RNNs. The learning algorithms produced by VSML RNNs may also provide more insight into how to develop more general, better performing learning algorithms. Finally, VSML RNNs further unify meta RNNs and learning algorithms on neural networks, which may lead to more flexible meta-learning procedures.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and impact of this work are well-discussed in section 7.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "lUwI3jVOn8", "reviewer": "Reviewer_MHDx", "summary": "This paper proposes variable shared meta learning,  and implements the network forward and backward computation with RNN.  The proposed method can also be used for learning general learning algorithms. Experiments on MNIST and Fashion MNIST dataset demonstrate the effectiveness of the proposed method. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe proposed method is novel compared to existing works. The authors clearly describe the difference w.r.t to previous contributions and well-positioned with related works. \n\nQuality:\n\nThis submission is technically sound.  The main claims are supported by the experimental results. The methods are carefully designed and demonstrated its advantage. \n\nClarity:\n\nThis paper is clearly written and well-structured.\n\nSignificance:\n\n1. The experimental section only use simple datasets, e.g., MNIST. I would like to see more real dataset, e.g., CIFAR10 or CIFAR100. This would improve the applicability on the real tasks. \n\n2.  RNN is hard to train in practice, how to ensure the stability of the proposed methods in real datasets since the RNNs are interactived with each other in the proposed method.\n\n3.  I wonder if the proposed method can be applied on other network architectures, e.g., convolutional network. This will broaden the applicability of the proposed method. \n\n4. I would like to see what is the performance on larger-scale network structure, currently only experiment on small-scale networks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors sufficiently discuss the limitations and societal impact of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "2YC1acEt1kH", "reviewer": "Reviewer_v8jj", "summary": "The authors in this paper propose to learn learning algorithms (LAs) like backpropagation using an RNN/LSTM to replace all the weights of a network. Subsequently, they use LA on a different dataset to show that the LA has generalized.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors model each weight with an LSTM's state. The forward and backward passes of the NN are coded into the LSTM's state updates. The work is novel and is an attempt to find algorithms that can replace backpropagation for training NN. The authors show that the learned backpropagation can learn on other datasets. They also show that the learning is faster and can learn with lesser examples.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is hard to read in many places. For, e.g., in eq-3, the sudden use of $s_{cai}$ without defining the term kept me guessing. Also, it is unclear how the error passing is learned using gradient descent for the backward pass emulating backpropagation. \n\nI would recommend adding a step-by-step algorithm to the paper to make things more straightforward. Including the gradient descent update for both forward and backward weights would be helpful. \nAlso, given the paper's claim of learning the meta-algorithm, I think the authors should also include the results of using this method on conv networks. \n\nA single ablation using a small conv network trained on Cifar using LA learned over FC layers would significantly improve the impact of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in agreement that the method developed were novel and well evaluated, particularly with the additional details and experimental results that came up in the discussion. These results should be incorporated in the final version of the paper. Overall this paper should be of significant interest to the NeurIPS community.
Decision: Accept (Poster)