Meta Learning Backpropagation And Improving It
Year: 2021

Abstract:
Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VSML) unifies the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VSML where the weights of a neural network are replaced by tiny LSTMs allows for implementing the backpropagation LA solely by running in forward-mode. It can even meta learn new LAs that differ from online backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta learned LAs learn through fast association in a way that is qualitatively different from gradient descent.

heading: 1 Introduction
text: The shift from standard machine learning to meta learning involves learning the learning algorithm (LA) itself, reducing the burden on the human designer to craft useful learning algorithms [43]. Recent meta learning has primarily focused on generalization from training tasks to similar test tasks, e.g., few-shot learning [11], or from training environments to similar test environments [17]. This contrasts human-engineered LAs that generalize across a wide range of datasets or environments. Without such generalization, meta learned LAs can not entirely replace human-engineered variants. Recent work demonstrated that meta learning can also successfully generate more general LAs that generalize across wide spectra of environments [20, 1, 31], e.g., from toy environments to Mujoco and Atari. Unfortunately, however, many recent approaches still rely on a large number of human-designed and unmodifiable inner-loop components such as backpropagation.
Is it possible to implement modifiable versions of backpropagation or related algorithms as part of the end-to-end differentiable activation dynamics of a neural net (NN), instead of inserting them as separate fixed routines? Here we propose the Variable Shared Meta Learning (VSML) principle for this purpose. It introduces a novel way of using sparsity and weight-sharing in NNs for meta learning. We build on the arguably simplest neural meta learner, the meta recurrent neural network (Meta RNN) [16, 10, 56], by replacing the weights of a neural network with tiny LSTMs. The resulting system can be viewed as many RNNs passing messages to each other, or as one big RNN with a sparse shared weight matrix, or as a system learning each neuron’s functionality and its LA. VSML generalizes the principle behind end-to-end differentiable fast weight programmers [45, 46, 3, 41], hyper networks [14], learned learning rules [4, 13, 33], and hebbian-like synaptic plasticity [44, 46, 25, 26, 30]. Our mechanism, VSML, can implement backpropagation solely in the forward-dynamics of an RNN. Consequently, it enables meta-optimization of backproplike algorithms. We envision a future where novel methods of credit assignment can be meta learned while still generalizing across vastly different tasks. This may lead to improvements in sample efficiency, memory efficiency, continual learning, and others. As a first step, our system meta
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
learns online LAs from scratch that frequently learn faster than gradient descent and generalize to datasets outside of the meta training distribution (e.g., from MNIST to Fashion MNIST). Our VSML RNN is the first neural meta learner without hard-coded backpropagation that shows such strong generalization.

heading: 2 Background
text: Deep learning-based meta learning that does not rely on fixed gradient descent in the inner loop has historically fallen into two categories, 1) Learnable weight update mechanisms that allow for changing the parameters of an NN to implement a learning rule (FWPs / LLRs), and 2) Learning algorithms implemented in black-box models such as recurrent neural networks (Meta RNNs).
Fast weight programmers & Learned learning rules (FWPs / LLRs) In a standard NN, the weights are updated by a fixed LA. This framework can be extended to meta learning by defining an explicit architecture that allows for modifying these weights. This weight-update architecture augments a standard NN architecture. NNs that generate or change the weights of another or the same NN are known as fast weight programmers (FWPs) [44, 45, 46, 3, 41], hypernetworks [14], NNs with synaptic plasticity [25, 26, 30] or learned learning rules [4, 13, 33]. Often these architectures make use of local Hebbian-like update rules or outer-products, and we summarize this category as FWPs / LLRs. In FWPs / LLRs the variables VL that are subject to learning are the weights of the network, whereas the meta-variables VM that implement the LA are defined by the weight-update architecture. Note that the dimensionality of VL and VM can be defined independently of each other and often VM are reused in a coordinate-wise fashion for VL resulting in |VL| |VM |, where | · | is the number of elements.
Black-box learning in activations (Meta RNNs) It was shown that an RNN such as LSTM can learn to implement an LA [16] when the reward or error is given as an input [47]. After meta training, the LA is encoded in the weights of this RNN and determines learning during meta testing. The activations serve as the memory used for the LA solution. We refer to this as Meta RNNs [16, 10, 56] (Also sometimes referred to as memory-based meta learning.). They are conceptually simpler than FWPs / LLRs as no additional weight-update rules with many degrees of freedom need to be defined. In Meta RNNs VL are the RNN activations and VM are the parameters for the RNN. Note that an RNN with N neurons will yield |VL| = O(N) and |VM | = O(N2) [46]. This means that the LA is largely overparameterized whereas the available memory for learning is very small, making this approach prone to overfitting [20]. As a result, the RNN parameters often encode task-specific solutions instead of generic LAs. Meta learning a simple and generalizing LA would benefit from |VL| |VM |. Previous approaches have tried to mend this issue by adding architectural complexity through additional memory mechanisms [53, 29, 40, 27, 42].

heading: 3 Variable Shared Meta Learning (VSML)
text: In VSML we build on the simplicity of Meta RNNs while retaining |VL| |VM | from FWPs / LLRs. We do this by reusing the same few parameters VM many times in an RNN (via variable sharing) and introducing sparsity in the connectivity. This yields several interpretations for VSML:
A VSML as a single Meta RNN with a sparse shared weight matrix (Figure 1a). The most general description.
B VSML as message passing between RNNs (Figure 1b). We choose a simple sharing and sparsity scheme for the weight matrix such that it corresponds to multiple RNNs with shared parameters that exchange information.
C VSML as complex neurons with learned updates (Figure 1c). When choosing a specific connectivity between RNNs, states / activations VL of these RNNs can be interpreted as the weights of a conventional NN, consequently blurring the distinction between a weight and an activation.
Introducing variable sharing to Meta RNNs We begin by formalizing Meta RNNs which often use multiplicative gates such as the LSTM [12, 15] or its variant GRU [6]. For notational simplicity,
we consider a vanilla RNN. Let s ∈ RN be the hidden state of an RNN. The update for an element j ∈ {1, . . . , N} is given by
sj ← fRNN(s)j = σ( ∑ i siWij), (1)
where σ is a non-linear activation function, W ∈ RN×N , and the bias is omitted for simplicity. We also omit inputs by assuming a subset of s to be externally provided. Each application of Equation 1 reflects a single time tick in the RNN.
We now introduce variable sharing (reusing W ) into the RNN by duplicating the computation along two axes of size A,B (here A = B, which will later be relaxed) giving s ∈ RA×B×N . For a ∈ {1, . . . , A}, b ∈ {1, . . . , B} we have
sabj ← fRNN(sab)j = σ( ∑ i sabiWij). (2)
This can be viewed as multiple RNNs arranged on a 2-dimensional grid, with shared parameters that update independent states. Here, we chose a particular arrangement (two axes) that will facilitate the interpretation C of RNNs as weights.
VSML as message passing between RNNs The computation so far describes A ·B independent RNNs. We connect those by passing messages (interpretation B )
sab ← fRNN(sab,−→ma), (3)
where the message −→mb = ∑ a′ f−→m(sa′b) with b ∈ {1, . . . , A = B}, f−→m : R N → RN ′ is fed as an additional input to each RNN. This is related to Graph Neural Networks [51, 58]. Summing over the axis A (elements a′) corresponds to an RNN connectivity mimicking those of weights in an NN (to facilitate interpretation C ). We emphasise that other schemes based on different kinds of message passing and graph connectivity are possible. For a simple f−→m defined by the matrix C ∈ RN×N , we may equivalently write
sabj ← σ( ∑ i sabiWij + ∑ a′ f−→m(sa′a)) = σ( ∑ i sabiWij + ∑ a′,i sa′aiCij). (4)
This constitutes the minimal version of VSML with VM := (W,C) and is visualized in Figure 1b.
VSML as a Meta RNN with a sparse shared weight matrix It is trivial to see that with A = 1 and B = 1 we obtain a single RNN and Equation 4 recovers the original Meta RNN Equation 1. In the general case, we can derive an equivalent formulation that corresponds to a single standard RNN with a single matrix W̃ that has entries of zero and shared entries
sabj ← σ( ∑ c,d,i scdiW̃cdiabj), (5)
where the six axes can be flattened to obtain the two axes. For Equation 4 and Equation 5 to be equivalent, W̃ must satisfy (derivation in Appendix A)
W̃cdiabj =  Cij , if d = a ∧ (d 6= b ∨ c 6= a). Wij , if d 6= a ∧ d = b ∧ c = a. Cij +Wij , if d = a ∧ d = b ∧ c = a. 0, otherwise.
(6)
This corresponds to interpretation A with the weight matrix visualized in Figure 1a. To distinguish between the single sparse shared RNN and the connected RNNs, we now call the latter sub-RNNs.
VSML as complex neurons with learned updates The arrangement and connectivity of the subRNNs as described in the previous paragraphs corresponds to that of weights in a standard NN. Thus, in interpretation C , VSML can be viewed as defining complex neurons where each sub-RNN corresponds to a weight in a standard NN as visualized in Figure 1c. All these sub-RNNs share the same parameters but have distinct states. The current formulation corresponds to a single NN layer that is run recurrently. We will generalize this to other architectures in the next section. A corresponds to the dimensionality of the inputs and B to that of the outputs in that layer.
The role of weights in a standard neural network is now assigned to the states of RNNs. This allows these RNNs to define both the neural forward computation as well as the learning algorithm that determines how the network is updated (where the mechanism is shared across the network). In the case of backpropagation, this would correspond to the forward and backward passes being implemented purely in the recurrent dynamics. We will demonstrate the practical feasibility of this in Section 3.2. The emergence of
RNN states as weights quickly leads to confusing terminology when RNNs have ‘meta weights’. Instead, we simply refer to meta variables VM and learned variables VL.
With this interpretation, VSML can be seen as a generalization of learned learning rules [4, 13, 33] and Hebbian-like differentiable mechanisms or fast weights more generally [44, 46, 25, 26] where RNNs replace explicit weight updates. In standard NNs, weights and activations have multiplicative interactions. For VSML RNNs to mimic such computation we require multiplicative interactions between parts of the state s. Fortunately, LSTMs already incorporate this through gating and can be directly used in place of RNNs.
Stacking VSML RNNs and feeding inputs To get a structure similar to one of the non-recurrent deep feed-forward architectures (FNNs), we stack multiple VSML RNNs where their states are untied and their parameters are tied.1 This is visualized with two layers in Figure 2 where the states s(2) of the second column of sub-RNNs are distinct from the first column s(1). The parameters A(k) and B(k) describing layer sizes can then be varied for each layer k ∈ {1, . . . ,K} constrained by A(k) = B(k−1). The updated Equation 3 with distinct layers k is given by s (k) ab ← fRNN(s (k) ab , −→m(k)a ) where −→m(k+1)b := ∑ a′ f−→m(s (k) a′b) with b ∈ {1, . . . , B(k) = A(k+1)}.
1The resultant architecture as a whole is still recurrent. Note that even standard FNNs are recurrent if the LA (backpropagation) is taken into account.
To prevent information from flowing only forward in the network, we use an additional backward message
s (k) ab ← fRNN (s (k) ab , −→m(k)a ,←−m (k) b ), (7)
where ←−m(k−1)a := ∑ b′ f←−m(s (k) ab′) with a ∈ {1, . . . , A(k) = B(k−1)} (visualized in Figure 3). The backward transformation is given by f←−m : RN → RN ′′ .
Similarly, other neural architectures can be explicitly constructed (e.g. convolutional NNs, Section B.2). Some architectures may be learned implicitly if positional information is fed into each sub-RNN (Appendix C). We then update all states s(k) in sequence 1, . . . ,K to
mimic sequential layer execution. We may also apply multiple RNN ticks for each layer k.
To provide the VSML RNN with data, each time we execute the operations of the first layer, a single new datum x ∈ RA(1) (e.g. one flattened image) is distributed across all sub-RNNs. In our present experiments, we match the axis A(1) to the input datum dimensionality such that each dimension (e.g., pixel) is fed to different RNNs. This corresponds to initializing the forward message −→m(1)a1 := xa (padding
−→m with zeros if necessary). Similarly, we read the output ŷ ∈ RB(K) from ŷa := −→m(K+1)a1 . Finally, we feed the error e ∈ R B(K) at the output such that ←−m(K)b1 := eb. See Figure 2 for a visualization. Alternatively, multiple input or output dimensions could be patched together and fed into fewer sub-RNNs.

heading: 3.1 Meta learning general-purpose learning algorithms from scratch
text: Having formalized VSML, we can now use end-to-end meta learning to create LAs from scratch in Algorithm 1. We simply optimize the LSTM parameters VM to minimize the sum of prediction losses over many time steps starting with random states VL := {s(k)ab }. We focus on meta learning online LAs where one example is fed at a time as done in Meta RNNs [16, 56, 10]. Meta training may be performed using end-to-end gradient descent or gradient-free optimization such as evolutionary strategies [57, 38]. The latter is significantly more efficient on VSML compared to standard NNs due to the small parameter space VM . Crucially, during meta testing, no explicit gradient descent is used.
Algorithm 1 VSML: Meta Training Require: Dataset(s) D = {(xi, yi)} VM ← initialize LSTM parameters while meta loss has not converged do . Outer loop in parallel over datasets D
VL = {s(k)ab } ← initialize LSTM states ∀a, b, k for (x, y) ∈ {(x1, y1), . . . , (xT , yT )} ⊂ D do . Inner loop over T examples −→m(1)a1 := xa ∀a . Initialize from input image x for k ∈ {1, . . . ,K} do . Iterating over K layers
s (k) ab ← fRNN (s (k) ab , −→m(k)a ,←−m(k)b ) ∀a, b . Equation 7 −→m(k+1)b := ∑ a′ f−→m(s (k) a′b) ∀b . Create forward message
←−m(k−1)a := ∑ b′ f←−m(s (k) ab′) ∀a . Create backward message ŷa := −→m(K+1)a1 ∀a . Read output e := ∇ŷL(ŷ, y) . Compute error at outputs using loss L ←−m(K)b1 := eb ∀b . Input errors
VM ← VM − α∇VM ∑T
t=1 L(ŷ(t), y(t)), obtaining∇VM either by • back-propagation through the inner loop • evolution strategies, using a search distribution p(VM )

heading: 3.2 Learning to implement backpropagation in RNNs
text: in Figure 4.
We designate an element of each message −→m(k)a , ←−m(k)b , f−→m(s (k) ab ), f←−m(s (k) ab ) as the input x, error e, and output ŷ and error ê
′. Similarly, we set w := sab1 and b := sab2. We then optimize VM via gradient descent to regress ŷ, ∆w, ∆b, and ê′ toward their respective targets. We can either generate the training datasetD := {(x,w, b, y, e, e′)i} randomly or run a ‘shadow’ NN on some supervised problem and fit the VSML RNN to its activations and parameter updates. Multiple
iterations in the VSML RNN would then correspond to evaluating the network and updating it via backpropagation. The activations from the forward pass necessary for credit assignment could be memorized as part of the state s or be explicitly stored and fed back. For simplicity, we chose the latter to clone backpropagation. We continuously run the VSML RNN forward, alternately running the layers in order 1, . . . ,K and in reverse order K, . . . , 1.2
4 Experiments
First, we demonstrate the capabilities of the VSML RNN by showing that it can implement neural forward computation and backpropagation in its recurrent dynamics on the MNIST [21] and Fashion MNIST [59] dataset. Then, we show how we can meta learn an LA from scratch on one set of datasets and then successfully apply it to another (out of distribution). Such generalization is enabled by extensive variable sharing where we have very few meta variables |VM | ≈ 2, 400 and many learned variables |VL| ≈ 257, 200. We also investigate the robustness of the discovered LA. Finally, we introspect the meta learned LA and compare it to gradient descent.
Our implementation uses LSTMs and the message interpretation from Equation 7. Hyperparameters, training details, and additional experiments can be found in the appendix.

heading: 4.1 VSML RNNs can implement backpropagation
text: As described in Section 3.2, we optimize the VSML RNN to implement backpropagation. We structure the sub-RNNs to mimic a feed-forward NN with either one hidden layer or no hidden layers. To obtain training targets, we instantiate a standard NN, the shadow network, and feed it MNIST data. After cloning, we then run the LA encoded in the VSML RNN on the MNIST and Fashion MNIST dataset and observe that it performs learning purely in its recurrent dynamics, making explicit gradient calculations unnecessary. Figure 5 shows the learning curve on these two datasets. Notably,
2Executing layers in reverse order is not strictly necessary as information always also flows backwards through←−m but makes LA cloning easier.
learning works both on MNIST (within distribution) and on Fashion MNIST (out of distribution). We observe that the loss is decently minimized, albeit regular gradient descent still outperforms our cloned backpropagation. This may be due to non-zero errors during learning algorithm cloning, in particular when these errors accumulate in the deeper architecture. It is also possible that the VSML states (‘weights’) deviate too far from ranges seen during cloning, in particular in the deep case when the loss starts increasing. We obtain 87% (deep) and 90% (shallow) test accuracy on MNIST and 76% (deep) and 80% (shallow) on Fashion MNIST (focusing on successful cloning over performance).

heading: 4.2 Meta learning supervised learning from scratch
text: In the previous experiments, we have established that VSML is expressive enough to metaoptimize backpropagation-like algorithms. Instead of cloning an LA, we now meta learn from scratch as described in Section 3.1. Here, we use a single layer (K = 1) from input to output dimension and run it for two ticks per image with N = 16 and N ′ = N ′′ = 8. First, the VSML RNN is meta trained end-to-end using evolutionary strategies (ES) [38] on MNIST to minimize the sum of cross-entropies over 500 data points starting from random state initializations. As each image is unique and VM can not memorize the data, we are implicitly optimizing the VSML RNN to generalize to future inputs given all inputs it has seen so far. We do not pre-train VM with a human-engineered LA.
During meta testing on MNIST (Figure 6) we plot the cumulative accuracy on all previous inputs on the y axis ( 1T ∑T t=1 ct after example T
with binary ct indicating prediction correctness). For each example, the prediction when this example was fed to the RNN is used, thus measuring sample efficient learning. This evaluation protocol is similar to the one used in Meta RNNs [56, 10]. We observe that learning is considerably faster compared to the baseline of online gradient descent (no mini batching, the learning rate appropriately tuned). One possibility is that VSML simply overfits to the training distribution. We reject this possibility by meta testing the same unmodified RNN on a different dataset, here Fashion MNIST. Learning still works well, meaning we have meta learned a fairly general LA (although performance at convergence still lacks behind a little). This generalization is achieved without using any hardcoded gradients during meta testing purely by running the RNN forward. In comparison to VSML, a Meta RNN heavily overfits.

heading: 4.3 Robustness to varying inputs and outputs

heading: 10 classes (unseen)
text: A defining property of VSML is that the same parameters VM can be used to learn on varying input and output sizes. Further, the architecture and thus the meta learned LA is invariant to the order of inputs and outputs. In this experiment, we investigate how robust we are to such changes. We meta train across MNIST with 3, 4, 6, and 7 classes. Likewise, we train across rescaled versions with 14x14, 28x28, and 32x32 pixels. We also randomly project all inputs using a linear transformation, with the transforma-
tion fixed for all inner learning steps. In Figure 7 we meta test on 6 configurations that were not seen
during meta training. Performance on all of these configurations is comparable to the unchanged reference from the previous section. In particular, the invariance to random projections suggests that we have meta learned a learning algorithm beyond transferring learned representations [compare 11, 54, 55].

heading: 4.4 Varying datasets
text: To better understand how different meta training distributions and meta test datasets affect VSML RNNs and our baselines, we present several different combinations in Figure 8. The opaque bars represent tasks that were not seen during meta training and are thus most relevant for this analysis. This includes four additional datasets: (1) Kuzushiji MNIST [7] with 10 classes, (2) EMNIST [9] with 62 classes, (3) A randomly generated classification dataset (Random) with 20 data points that changes with each step in the outer loop, and (4) Sum Sign which generates random inputs and requires classifying the sign of the sum of all inputs. Meta training is done over 500 randomly drawn samples per outer iteration. Each algorithm is meta trained for 10k outer iterations. Inputs are randomly projected as in Section 4.3 (for VSML; the baselines did not benefit from these augmentations). We again report the cumulative accuracy on all data seen since the beginning of meta test training. We compare to SGD with a single layer, matching the architecture of VSML, and a hidden layer, matching the number of weights to the size of VL in VSML. We also have included a Hebbian fast weight baseline [25] and an external (fast weight) memory approach [42].
We observe that VSML generalizes much better than Meta RNNs, Hebbian fast weights, and the external memory. These baselines overfit to the training environments. Notably, VSML even generalizes to the unseen tasks Random and Sum Sign which have no shared structure with the other datasets. In many cases VSML’s performance is similar to SGD but a little more sample efficient in the beginning of training (learning curves in Appendix B). This suggests that our meta learned LAs are good at quickly associating new inputs with their labels. We further investigate this in the next Section 5.
5 Analysis
Given that VSML seems to learn faster than online gradient descent in many cases we would like to qualitatively investigate how learning differs. We first meta train on the full MNIST dataset as before. During meta testing, we plot the output probabilities for each digit against the number of samples seen in Figure 9. We highlight the ground truth input class as well as the predicted class ©. In this case, our meta test dataset consists of MNIST digits with two examples of each type. The same digit is always repeated twice. This allows us to observe and visualize the effect with only a few examples. We have done the same introspection with the full dataset in Appendix B.
We observe that in VSML almost all failed predictions are followed by the correct prediction with high certainty. In contrast, SGD
makes many incorrect predictions and fails to adapt correctly in only 20 steps. It seems that SGD learns qualitatively different from VSML. The VSML RNN meta learns to quickly associate new inputs with their class whereas SGD fails to do so. We tried several different SGD learning rates and considered multiple steps on the same input. In both cases, SGD does not behave similar to VSML, either learning much slower or forgetting previous examples. As evident from high accuracies in Figure 8, VSML does not only memorize inputs using this strategy of fast association, but the associations generalize to future unseen inputs.

heading: 6 Related Work
text: Memory based meta learning (Meta RNNs) Memory-based meta learning in RNNs [16, 10, 56] is a simple neural meta learner (see Section 2). Unfortunately, the LA encoded in the RNN parameters is largely overparameterized which leads to overfitting. VSML demonstrates that weight sharing can address this issue resulting in more general-purpose LAs.
Learned Learning Rules / Fast Weights NNs that generate or change the weights of another or the same NN are known as fast weight programmers [44], hypernetworks [14], NNs with synaptic plasticity [25] or learned learning rules [4] (see Section 2). In VSML we do not require explicit architectures for weight updates as weights are emergent from RNN state updates. In addition to the learning rule, we implicitly learn how the neural forward computation is defined. Concurrent to this work, fast weights have also been used to meta learn more general LAs [39].
Learned gradient-based optimizers Meta learning has been used to find optimizers that update the parameters of a model by taking the loss and gradient with respect to these parameters as an input [34, 2, 22, 24]. In this work, we are interested in meta learning that does not rely on fixed gradient calculation in the inner loop.
Discrete program search An interesting alternative to distributed variable updates in VSML is meta learning via discrete program search [48, 35]. In this paradigm, a separate programming language needs to be defined that gives rise to neural computation when its instructions are combined. This led to the automated rediscovery of backpropagation [35]. In VSML we demonstrate that a symbolic programming language is not required and general-purpose LAs can be discovered and encoded in variable-shared RNNs. Search over neural network parameters is usually easier compared to symbolic program search due to smoothness in the loss landscape.
Multi-agent systems In the reinforcement learning setting multiple agents can be modeled with shared parameters [50, 32, 18], also in the context of meta learning [36]. This is related to the variable sharing in VSML depending on how the agent-environment boundary is drawn. Unlike these works, we demonstrate the advantage of variable sharing in meta learning more general-purpose LAs and present a weight update interpretation.

heading: 7 Discussion and Limitations
text: The research community has perfected the art of leveraging backpropagation for learning for a long time. At the same time, there are open questions such as how to minimize memory requirements, effectively learn online and continually, learn sample efficiently, learn without separate backward phases, and others. VSML suggests that instead of building on top of backpropagation as a fixed routine, meta learning offers an alternative to discover general-purpose LAs. Nevertheless, this paper is only a proof of concept—until now we have only investigated small-scale problems and performance does not yet quite match the mini-batched setting with large quantities of data. In particular, we observed premature convergence of the solution at meta test time which calls for further investigations. Scaling our system to harder problems and larger meta task distributions will be important future work.
The computational cost of the current VSML variant is also larger than the one of standard backpropagation. If we run a sub-RNN for each weight in a standard NN with W weights, the cost is in O(WN2), where N is the state size of a sub-RNN. If N is small enough, and our experiments suggest small N may be feasible, this may be an acceptable cost. However, VSML is not bound to the interpretation of a sub-RNN as one weight. Future work may relax this particular choice.
Meta optimization is also prone to local minima. In particular, when the number of ticks between input and feedback increases (e.g. deeper architectures), credit assignment becomes harder. Early experiments suggest that diverse meta task distributions can help mitigate these issues. Additionally, learning horizons are limited when using backprop-based meta optimization. Using ES allowed for training across longer horizons and more stable optimization.
VSML can also be viewed as regularizing the NN weights that encode the LA through a representational bottleneck. It is conceivable that LA generalization as obtained by VSML can also be achieved through other regularization techniques. Unlike most regularizers, VSML also introduces substantial reuse of the same learning principle and permutation invariance through variable sharing.

heading: 8 Conclusion
text: We introduced Variable Shared Meta Learning (VSML), a simple principle of weight sharing and sparsity for meta learning powerful learning algorithms (LAs). Our implementation replaces the weights of a neural network with tiny LSTMs that share parameters. We discuss connections to meta recurrent neural networks, fast weight generators (hyper networks), and learned learning rules.
Using learning algorithm cloning, VSML RNNs can learn to implement the backpropagation algorithm and its parameter updates encoded implicitly in the recurrent dynamics. On MNIST it learns to predict well without any human-designed explicit computational graph for gradient calculation.
VSML can meta learn from scratch supervised LAs that do not explicitly rely on gradient computation and that generalize to unseen datasets. Introspection reveals that VSML LAs learn by fast association in a way that is qualitatively different from stochastic gradient descent. This leads to gains in sample efficiency. Future work will focus on reinforcement learning settings, improvements of meta learning, larger task distributions, and learning over longer horizons.

heading: Acknowledgements
text: We thank Sjoerd van Steenkiste, Imanol Schlag, Kazuki Irie, and the anonymous reviewers for their comments and feedback. This work was supported by the ERC Advanced Grant (no: 742870) and computational resources by the Swiss National Supercomputing Centre (CSCS, projects s978 and s1041). We also thank NVIDIA Corporation for donating several DGX machines as part of the Pioneers of AI Research Award, IBM for donating a Minsky machine, and weights & biases [5] for their great experiment tracking software and support.

Review summaries:
The paper proposes to meta-learn the backpropagations by representing weights of a network by small RNNs. Different RNNs share the weights, and the weights can be thought of as representing the states of RNN. The approach can thus implement back-propagation solely in the forward dynamics of an RNN. 

This paper proposes VSML RNN, an architecture for RNNs that is suitable for meta-learning problems in which the RNN activations represent task-specific quantities and RNN weights represent task-independent quantities. The authors use the insight that RNNs for meta learning (meta RNNs as called in the paper) perform better when the meta-parameters are constrained relative to the task-specific parameters. VSML RNN is essentially an RNN with a particular weight sharing and sparsity structure. The authors demonstrate that VSML RNN can implement *within its hidden state* backpropagation on a smaller model, where the parameters of the smaller model are represented in the VSML RNN's hidden state. Empirically, the authors show that the learning algorithm implemented by a VSML RNN can outperform baselines, and can learn faster than gradient descent. The authors also demonstrate that the learning algorithm is transferable to unseen problems.

This paper proposes variable shared meta learning,  and implements the network forward and backward computation with RNN.  The proposed method can also be used for learning general learning algorithms. Experiments on MNIST and Fashion MNIST dataset demonstrate the effectiveness of the proposed method.