Title: NORML: NODAL OPTIMIZATION FOR RECURRENT META-LEARNING

Abstract: Meta-learning is an exciting and powerful paradigm that aims to improve the effectiveness of current learning systems. By formulating the learning process as an optimization problem, a model can learn how to learn while requiring significantly less data or experience than traditional approaches. Gradient-based meta-learning methods aims to do just that, however recent work have shown that the effectiveness of these approaches are primarily due to feature reuse and very little has to do with priming the system for rapid learning (learning to make effective weight updates on unseen data distributions). This work introduces Nodal Optimization for Recurrent Meta-Learning (NORML), a novel meta-learning framework where an LSTM-based meta-learner performs neuron-wise optimization on a learner for efficient task learning. Crucially, the number of meta-learner parameters needed in NORML, increases linearly relative to the number of learner parameters. Allowing NORML to potentially scale to learner networks with very large numbers of parameters. While NORML also benefits from feature reuse it is shown experimentally that the meta-learner LSTM learns to make effective weight updates using information from previous data-points and update steps.

1. INTRODUCTION:

Humans have a remarkable capability to learn useful concepts from a small number of examples or a limited amount of experience. In contrast most machine learning methods require large, labelled datasets to learn effectively. Little is understood about the actual learning algorithm(s) used by the human brain, and how it relates to machine learning algorithms like backpropagation (Lillicrap & Körding (2019) ). Botvinick et al. (2019) argues that inductive bias and structured priors are some of the main factors that enable fast learning in animals. In order to build general-purpose systems we must be able to design and build learning algorithms that can quickly and effectively learn from a limited amount of data by utilizing prior knowledge and experience.

Supervised few-shot learning aims to challenge machine learning models to learn new tasks by leveraging only a handful of labelled examples. Vinyals et al. (2016) introduces the few-shot learning problem for image classification, where a model is tasked to classify a number of images while being provided either 1 or 5 examples of each class (hereafter referred to as 1-shot and 5-shot learning). One way to approach this problem is by way of meta-learning, a broad family of techniques that aim to learn how to learn (Thrun & Pratt (1998) ). One particularly powerful group of approaches, known as memory-based methods, use memory architectures that can leverage prior information to assist in learning (Santoro et al. (2016) ; Ravi & Larochelle (2017) ). Optimization-based methods (Finn et al. (2017) ) is another exciting area that aim to learn an initial set of parameters that can quickly adapt to new, unseen tasks with relatively little training data. This work introduces a novel technique where a recurrent neural network based meta-learner learns how to make parameter updates for a task learner. The meta-learner and the learner are jointly trained so as to learn how to learn new tasks with little data. This approach allows one to utilize aspects from both optimization-based and memory-based meta-learning methods. The recurrent architecture of the meta-learner can use important prior information when updating the learner, while the learner can learn a set of initial task parameters are easily optimized for a new task. The vanishing gradient challenge faced by gradient-based optimization is solved by using a Long shortterm memory (LSTM) based meta-learner (Hochreiter & Schmidhuber (1997) ). Memory-based methods (Ravi & Larochelle (2017) ) that use all of the learner's parameters as input to a metalearner tend to break down when using a learner with a large number of parameters (Andrychowicz et al., 2016) . The technique proposed in this work, Nodal Optimization for Recurrent Meta-Learning (NORML), solves this scaling problem by learning layer-wise update signals used to update the learners's parameters.

NORML is evaluated on the Mini-ImageNet dataset and is shown to improve on existing optimization-based and memory-based methods. An ablation study is done showing the effects of the different components of NORML. Furthermore a comparison is done between NORML and Model Agnostic Meta-Learning (MAML) using the Omniglot dataset. The comparison demonstrates that NORML makes superior parameter updates than the updates made by gradient descent.

2. PRELIMINARIES:



2.1. BACKPROPAGATION:

The backpropagation algorithm (Rumelhart et al., 1986) was developed in the 1980's and has since been the status quo for training neural networks. By recursively applying the chain rule, backpropagation sends gradient signals back through stacked layers of a deep network. Each gradient signal at a particular layer is used to calculate the gradient of the weights connected to that layer.

Consider a neural network with N hidden layers. Let the rows of matrix W l denote the weights connecting the nodes in the layer below to a hidden node in layer l and let the vector b l denote the biases connected to each node in layer l. Then:

EQUATION

Where f l is the activation function used at layer l, a l is the layer's pre-activation output, and h l is referred to as the activations of layer l. Note that h 0 = x is the input to the network. The output of the network is given by ŷ = f out (a out ). When choosing to use a softmax as the output activation and a cross-entropy loss function, the loss and the output layer's gradient are calculated as follows:

EQUATION

δ aout = ∂L ∂a out = ŷ -y (3)

where L is the cross-entropy loss, n is the network's output size, and δ aout is the gradient at the pre-activation of the output layer. Note that δ aout is an n-dimensional vector where each element of δ aout describes the corresponding node's contribution to the loss. By propagating δ aout back through the network one obtains the gradients at each of the hidden layers' nodes:

EQUATION

where f l () is the derivative of the activation function, is an element-wise multiplication and δ a l is the pre-activation gradient at layer l. In order to calculate the gradient with regards to the weights, δ a l is matrix multiplied with the activations of the previous layer:

∂L ∂W l = δ a l h l-1 (5) 2.2 META-LEARNING PROBLEM DEFINITION

The N -way K-shot problem is defined using an episodic formulation as proposed in Vinyals et al. (2016) . A task T i is sampled from a task distribution p(T ) and consists of an 

2.3. MODEL AGNOSTIC META-LEARNING:

Optimization-based meta-learning is an approach to meta-learning where an inner loop is used for fast adaptation to a new task, and an outer loop is used to optimize the inner loop's training steps. Model Agnostic Meta-Learning (MAML) is one such approach that contains all the key ingredients used in optimization-based meta-learning. The inner loop consists of training a model, f θ via gradient descent on a few-shot learning task, T i .

θ = G(θ, D tr ) (6)

where G is often implemented as a single step of gradient descent on the training set D tr ;

θ i = θ -η∇ θ L tr Ti (f θ ).

It is often advantageous for G to consist of multiple sequential update steps. The validation set of T i is then used to evaluate f θ i , and the task-specific validation loss, L val Ti , is calculated. The outer loop consists of optimizing the base-parameters θ, using the sum of M different tasks' validation losses:

EQUATION

where M is referred to as the meta-batch size.

This approach allows a model to learn a set of base-parameters θ, that can quickly adapt to a new unseen task. After meta-training a model, the inner loop procedure can update θ and return taskspecific parameters θ i using only a few examples of the new task. By differentiating through the inner loop, MAML learns to update the base-parameters θ in such a way that the task-specific paramaters, θ i , generalizes to unseen examples of task T i .

In many cases it is preferred to have an inner loop that consists of multiple sequential updates. However the inner loop's computational graph can become quite large if too many steps are taken. This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner loop Aravind Rajeswaran (2019) Antoniou et al. (2018) . This limits MAML to domains where a small amount of update steps are sufficient for learning. The LSTM-based metalearner proposed in this work, allow gradients to effectively flow through a large number of update steps. NORML can therefore be applied to a wide array of domains.

3. MODEL:



3.1. TRAINING:

NORML incoporates a neural network based learner with N layers and an LSTM-based metalearner that is used to optimize the learner's inner loop parameter updates. Figure 1 depicts the process by which the meta-learner calculates the parameter updates for a single layer learner. When multiple layers are optimized the additional gradient signals and layer inputs are added to the meta learner's input. The meta-learner outputs two update signals for every layer, so when a 4-layer network is used as the learner, the meta-learner outputs 8 update signals.

The high-level operation is as follows (Algorithm 1). First the learner's task-specific loss, L tr Ti (f θ ), and layer-gradients, δ a l , are calculated via cross-entropy and backpropagation. The meta-learner takes δ a l , h l-1 and the loss, L, as input, and outputs the current cell state and the update signals, ĥl-1 and δa l :

EQUATION

The update signals are matrix multiplied to determine the parameter updates for layer l: The learner's updated parameters, θ Ti , are then used on the next training example and the whole process gets repeated for U update steps. The inner loop terminates after calculating the task validation loss using f θ .

∆W l = η δa l ĥl-1 (9) ∆b l = η δa l (10)

EQUATION

The outer loop consists of optimizing both θ 0 and the meta-learner parameters Φ via gradient descent:

θ ← θ -β∇ θ Ti p(T )

EQUATION

3.2. META-LEARNER:

The meta-learner, m is implemented using a modified LSTM cell as shown in Figure 2 .

The input to the meta-learner is normalized, flattened and concatenated. x l is used to denote concatenating the loss, the layer gradients, and the activations of the previous layer.

The meta-learner consists of a forget gate, f l , an input gate, i l and an update gate, cl . The cell state at update step t is denoted via c l,t , and is calculated as follows:

EQUATION

In order to determine the update signals, the cell state, c l,t , is used as input to two separate fullyconnected layers followed by a sigmoid activation function and a pointwise multiplication with the original layer gradient and the previous layer's activations:

EQUATION

) The sigmoid activation and pointwise mutliplication operations allow the meta-learner to scale the update signal received by each node in the network. This node-wise scaling let the meta-learner control how the weights connected to a particular node will get changed at any given update step. By scaling the layer gradients, the meta-learner can dynamically control by how much each node's input should be changed at each update step. Likewise, by scaling the hidden layer activations, the meta-learner can control the change in weighting of each of the previous nodes' output. By controlling the magnitude of both the activations and the layer gradients, the meta-learner can learn a dynamically adaptive learning rate for each individual weight and bias of the learner. This can be achieved with a meta-learner that scales linearly in size relative to the size of the learner, i.e. increasing the number of learner parameters by a factor of p, will result in the same factor increase in the number of meta-learner parameters.

EQUATION

4. RELATED WORK:

Learning how to learn has a long history and variety of approaches have been proposed. Memorybased methods use memory architectures to either store key training examples or to encode fast adaptation algorithms (Santoro et al., 2016) (Andrychowicz et al., 2016) (Ravi & Larochelle, 2017) . Metric-based methods are designed to learn similarity metrics for examples of the same class (Koch, 2015) (Vinyals et al., 2016) (Snell et al., 2017) . Optimization-based methods aim to learn a set of initial parameters that can quickly adapt to new tasks (Finn et al., 2017) .

A subset of memory-based methods use recurrent neural networks to learn fast adaptation algorithms. In Ravi & Larochelle (2017) an LSTM-based meta-learner is trained to iteratively optimize a neural network. At a given timestep the meta-learner's hidden state consists of the optimizee's current parameters. At each timestep a training datapoint is used to calculate the optimizee's parameter gradients. The parameter gradients and the task loss is used as input to the meta-learner. The hidden state of the following timestep is then used as the updated parameters for the learner. A challenge of this approach is that if you want to optimize tens of thousands of parameters, you would have a massive hidden state and input size, and will therefore require an enormous number of parameters for the meta-learner.

In Andrychowicz et al. (2016) an alternative approach is introduced that avoids the aforementioned scaling problem by individually optimizing each parameter using a separate LSTM. This vastly decreases the number of meta-parameters required, however each meta-learner is only given information of a single parameter and can therefore not use information of other parts of the network when doing optimization. The approach introduced in this work uses a single meta-learner to optimize all the parameter's of the learner, while the meta-learner require less than 5 times the number of parameters used by the learner.

Optimization-based methods (Finn et al., 2017) , differentiate through the inner loop in order to optimize learning. Improvements to MAML include using a pre-trained network to extract features and to then only optimize the final fully-connected layer via gradient based optimization (?). Using a pre-trained feature extractor assists in overcoming task overfitting and have been demonstrated to give a large boost in performance (Sun et al., 2018) . In this work a pre-trained feature extractor was also used when evaluating NORML on Mini-Imagenet, but is not a requirement for implementing NORML.

Other work that use custom layer-wise update signals include Nø kland (2016) and Nkland & Eidnes (2019) . In feedback alignment layer gradient approximating update signals are used to train a neural network by passing the error signal through fixed random matrices. Backpropagation in contrast passes the error signal through the transpose of the subsequent layer's weight matrix. Surprisingly feedback alignment demonstrates that learning can still occur even when using random matrices for the backward pass.

5. EXPERIMENTAL RESULTS:

The proposed approach is evaluated on the Mini-Imagenet benchmark dataset. An ablation study comparing the performance against MAML is conducted on the Omniglot dataset. The evaluation aims to compare the performance of NORML against existing methods and to specifically test if node-wise optimization improves the update signals provided by backpropagation.

5.1. MINI-IMAGENET CLASSIFICATION:

The Mini-Imagenet dataset was proposed by Vinyals et al. (2016) and is a popular benchmark for few-shot learning methods. The dataset consists of 100 classes, each containing 600 images. The meta-training, meta-validation and meta-testing sets each contain 64, 16 and 20 classes respectively. It has been demonstrated that by using a pre-trained feature extractor the performance on few-shot learning tasks can be significantly improved (Sun et al. (2018) ). For evaluation of NORML on the Mini-Imagenet benchmark, a Resnet-12 network is pre-trained using only data and classes from the meta-training set. The pre-train model is trained to perform 64-way classification. After pretraining, the final FC-layer is replaced by a randomly initialized FC-layer and the convolutional layers are frozen.

The meta-learner network and the final layer of the learner are optimized during meta-training. The inner loop consisted of 10 epochs for both the 5-shot and 1-shot cases, while using a meta-batch size of 4. The model was trained for 10 000 meta-iterations and the meta-validation set was used for model selection for testing.

Classification accuracies for NORML and other baselines are shown in Table 1 .

Model:

Mini-Imagenet Test Accuracy 1-shot 5-shot Matching network (Vinyals et al. (2016) 

5.2. ABLATION STUDY:

For the Mini-Imagenet benchmark the learner consisted of a single FC-layer which was proceeded by a pre-trained feature extractor. NORML can however be applied to networks with multiple layers.

To assess the effect of the different components of the meta-learner and to compare NORML to MAML, an ablation study was conducted on the Omniglot 20-way dataset. The learner consisted of a fully-connected network with 4 hidden layers, each including batch normalization and ReLU activations, followed by a linear layer and softmax. The hidden layer sizes were 256, 128, 64 and 64. The same learner network was used for all approaches while only changing the setup of the meta-learner between experiments.

The cell state and hidden state of the meta-learner carries information of all the previous losses, gradients and update steps that was calculated for the task at hand. This enables NORML to potentially improve on the update signals calculated via backpropagation, since the gradient for a particular data-point only contains information from the current time step. To assess the contribution of the cell state and hidden state to the performance of NORML, the following three setups were tested:

(1) Setting the cell input state and hidden input state to zero, (2) only setting the hidden state to zero and (3) only setting the cell state to zero.

The meta-learner used in NORML can learn to dynamically control the learning rates of each learner node's input and output weights. To assess the effect of the node-wise learning rates and to compare NORML to MAML, three different training setups are used for MAML: (1) MAML with a constant learning rate, (2) MAML with a single learnable learning rate and (3) MAML with learnable per-layer learning rates. The results for the 1-shot and 5-shot tasks are shown in Table 2 . Note that the results of the ablation study are obtained using a fully-connected network; a convolutional neural network will give higher accuracies. NORML learns to dynamically control the learning rates of each learner node's input and output weights. This is shown to give a significant improvement when compared to using constant, learnable, and per-layer learnable learning rates. Intuitively it makes sense, since NORML can effectively control the learning rates of each parameter in the learner network. Additionally NORML can control the learning rates dynamically. Dynamic control seems crucial to NORML's success, since a parameter that receives a large update at step t, might need to make a small update at step t + 1. The importance of being able to dynamically adapt the learning rate using prior update information, is confirmed in the experiments where the input hidden state and input cell state of the meta-learner are set to zero. Although the meta-learner can still learn nodal learning rates, it is unable to use information of previous update steps and ends up performing the same as MAML with a constant learning rate.

6. CONCLUSION AND FUTURE WORK:

Recent work has shown that the successes of most meta-learning techniques are primarily due to feature reuse (Raghu et al., 2019) . Ideally meta-learning techniques should be learning to also make efficient and effective parameter updates. This work introduces a novel meta-learning method that allows node-wise scaling of learner update signals, thereby enabling the meta-learner to dynamically control the learning rate of each of the learner's parameters. Importantly this is done by a meta-learner who's size scales linearly relative to the size of the learner.

Future work would focus on scaling NORML to optimize large learner networks. Furthermore, applying NORML to domains where a large number of update steps are required, such as Deep Reinforcement Learning, could prove to be an exciting avenue of further research since the LSTMbased meta-learner allow gradients to propagate back much farther than gradient-based optimization methods (Finn et al., 2017) .

:



Figures and Tables:

Figure 1: Training process of NORML.

Figure 2: Meta-learner LSTM.

N -way classification problem. A meta-dataset is divided into a training meta-set S tr , a validation meta-set S val and a testing meta-set S test where the classes contained in each meta-set is disjointed (i.e. none of the classes in S test is present in S tr and visa versa). The task T i consists of a training set D tr and validation set D val . The training set D tr contains K examples from each of T i 's N classes. The validation set D val usually contains a larger set of examples from each class in order to give an estimate of the model's generalization performance on task T i . Note that a task's validation set D val (used to optimize the meta training objective) should not be confused with the meta validation set S val , which is used for model selection.

Table content: <table/>

Algorithm 1 Nodal Optimization for Rapid Meta-Learning Require: : Learner f with parameters θ Require: : Meta-learner m with parameters Φ Require: : p(T ): distribution over tasks Require: : η, β: step size hyperparameters 1: randomly initialize θ, Φ 2: while not converged do

Table content: <table><tr><td>3:</td><td colspan="3">Sample batch of tasks T i ∼ p(T )</td></tr><tr><td>4:</td><td/><td/><td/></tr><tr><td>5: 6:</td><td colspan="2">for all T i do Let (D tr i , D val i ) = T i</td><td/></tr><tr><td>7:</td><td>Initialize θ 0 = θ</td><td/><td/></tr><tr><td>8:</td><td/><td/><td/></tr><tr><td>9: 10:</td><td colspan="4">for t = 1, len(D tr i ) do Compute learner's training loss L tr Ti (f θ t-1 ) and layer gradients</td><td>∂L tr T i ∂a l for all layers l</td></tr><tr><td>11:</td><td/><td/><td/><td>∂L tr T i ∂a l , h l-1 )</td></tr><tr><td>13:</td><td>end for</td><td/><td/></tr><tr><td>14: 15:</td><td colspan="3">Compute validation loss L val Ti (f θ t ) end for</td></tr><tr><td>16:</td><td/><td/><td/></tr><tr><td/><td>Cell State</td><td/><td/><td>Cell State</td></tr><tr><td/><td>at t</td><td/><td/><td>at t+1</td></tr><tr><td/><td/><td/><td/><td>FC</td><td>FC</td></tr><tr><td/><td/><td/><td/><td>Layer</td><td>Layer</td></tr><tr><td/><td/><td>Forget Gate</td><td>Input Gate</td><td>Update Gate</td></tr><tr><td/><td/><td/><td/><td>sigmoid</td><td>sigmoid</td></tr><tr><td/><td/><td>sigmoid</td><td>sigmoid</td><td>tanh</td></tr><tr><td/><td/><td>FC Layer</td><td>FC Layer</td><td>FC Layer</td></tr><tr><td/><td/><td/><td/><td>Hidden</td></tr><tr><td/><td/><td>Concat</td><td/><td>State</td></tr><tr><td/><td>Hidden State</td><td>and Norm</td><td/><td>Concat</td><td>at t+1</td></tr><tr><td/><td>at t</td><td/><td/></tr><tr><td/><td>Layer Input</td><td/><td/><td>Gradient Update</td><td>Layer Input</td></tr><tr><td/><td/><td/><td/><td>Signal</td><td>Update</td></tr><tr><td/><td>Layer Gradient</td><td/><td/><td>Signal</td></tr></table>

Test accuracies on the Mini-ImageNet benchmark. The first set of results use convolutional networks, while the second use much deeper residual networks, predominantly in conjunction with pre-training.

Table content: <table><tr><td>)</td><td>43.56 ±0.84%</td><td>55.31 ±0.73%</td></tr></table>

Ablation study and comparison to MAML using different training setups

Table content: <table><tr><td>Model</td><td colspan="2">Omniglot 20-way Test Accuracy</td></tr><tr><td/><td>1-shot</td><td>5-shot</td></tr><tr><td>MAML (constant lr)</td><td>86.3%</td><td>95.7%</td></tr><tr><td>MAML (global lr)</td><td>86.5%</td><td>95.7%</td></tr><tr><td>MAML (layer-wise lr)</td><td>87.1%</td><td>96.0%</td></tr><tr><td colspan="2">NORML (zero hidden &amp; zero cell) 86.3%</td><td>95.7%</td></tr><tr><td>NORML (zero cell)</td><td>86.3%</td><td>95.7%</td></tr><tr><td>NORML (zero hidden)</td><td>87.9%</td><td>96.5%</td></tr><tr><td>NORML</td><td>90.3%</td><td>97.2%</td></tr></table>

Peer review (iclr2020):
- The paper targets the scalability issue of certain meta-learning frameworks, and is therefore addressing an important and interesting topic. - Theoretical and technical novelty is rather minimal. - The paper writing is well beyond the ICLR level, and is honestly beyond the level required by a scientific manuscript in general. Writing needs a massive overhaul. - There are way too many grammatical mistakes. In addition, there are citations written the wrong way -e.g. Rajeswaran (2019)-, and also the flow of the ideas has room for improvement. - For the aforementioned reference, the author ordering is wrong, and not consistent with the way it is cited in the text. - page 1: *while the learner can learn a set of initial task parameters are easily optimized for a new task. *: Please fix and/or clarify. - page 2: *where the classes contained in each meta-set is disjointed*. - page 3: *The LSTM-based meta-learner proposed in this work, allow gradients to* - *Memory-based Under review as a conference paper at ICLR 2020 methods (Ravi & Larochelle (2017)) that use all of the learner’s parameters as input to a meta-learner tend to break down when using a learner with a large number of parameters (Andrychowicz et al., 2016).*: Just to clarify: The latter paper, which is criticising the former category, is older than the paper representing the former category. Is that right? - page 2: *superior parameter updates*. I see that the result has been based on classification accuracy. Notwithstanding the ablative study in the experiments secion, maybe superior parameter updates can be either replaced by a more unequivocal description of the mentioned comparison, or formally defined from there onwards. - At the ICLR level, I do not think that all this detailed description of backpropagation would be necessary. - Equation 1 and the description that follows: *a_l is the layer’s pre-activation output*. Is a_l the post-activation output as well? Equation 1 implies so, doesn*t it? - page 3: *This limits MAML to domains where a small amount of update steps are sufficient for learning.*: What do you mean by *This*? Is it to have inner loops consisting of multiple sequential updates, which is what was referred to as *preferable* in the beginning of the same paragraph? i.e. Is the MAML limitation noted in this paragraph a limitation of the vanilla MAML (plus the other versions) as well or solely due to the *preferred, yet detrimental* extension of adopting multiple sequential updates? Minor: - page 1: Supervised few-shot learning *aims to challenge machine learning models to ...*: It does not challenge ML models; it is a specific ML paradigm.