NORML: NODAL OPTIMIZATION FOR RECURRENT META-LEARNING
Year: 2019

Abstract:
Meta-learning is an exciting and powerful paradigm that aims to improve the effectiveness of current learning systems. By formulating the learning process as an optimization problem, a model can learn how to learn while requiring significantly less data or experience than traditional approaches. Gradient-based meta-learning methods aims to do just that, however recent work have shown that the effectiveness of these approaches are primarily due to feature reuse and very little has to do with priming the system for rapid learning (learning to make effective weight updates on unseen data distributions). This work introduces Nodal Optimization for Recurrent Meta-Learning (NORML), a novel meta-learning framework where an LSTM-based meta-learner performs neuron-wise optimization on a learner for efficient task learning. Crucially, the number of meta-learner parameters needed in NORML, increases linearly relative to the number of learner parameters. Allowing NORML to potentially scale to learner networks with very large numbers of parameters. While NORML also benefits from feature reuse it is shown experimentally that the meta-learner LSTM learns to make effective weight updates using information from previous data-points and update steps.

heading: 1 INTRODUCTION
text: Humans have a remarkable capability to learn useful concepts from a small number of examples or a limited amount of experience. In contrast most machine learning methods require large, labelled datasets to learn effectively. Little is understood about the actual learning algorithm(s) used by the human brain, and how it relates to machine learning algorithms like backpropagation (Lillicrap & Körding (2019)). Botvinick et al. (2019) argues that inductive bias and structured priors are some of the main factors that enable fast learning in animals. In order to build general-purpose systems we must be able to design and build learning algorithms that can quickly and effectively learn from a limited amount of data by utilizing prior knowledge and experience.
Supervised few-shot learning aims to challenge machine learning models to learn new tasks by leveraging only a handful of labelled examples. Vinyals et al. (2016) introduces the few-shot learning problem for image classification, where a model is tasked to classify a number of images while being provided either 1 or 5 examples of each class (hereafter referred to as 1-shot and 5-shot learning). One way to approach this problem is by way of meta-learning, a broad family of techniques that aim to learn how to learn (Thrun & Pratt (1998)). One particularly powerful group of approaches, known as memory-based methods, use memory architectures that can leverage prior information to assist in learning (Santoro et al. (2016); Ravi & Larochelle (2017)). Optimization-based methods (Finn et al. (2017)) is another exciting area that aim to learn an initial set of parameters that can quickly adapt to new, unseen tasks with relatively little training data.
This work introduces a novel technique where a recurrent neural network based meta-learner learns how to make parameter updates for a task learner. The meta-learner and the learner are jointly trained so as to learn how to learn new tasks with little data. This approach allows one to utilize aspects from both optimization-based and memory-based meta-learning methods. The recurrent architecture of the meta-learner can use important prior information when updating the learner, while the learner can learn a set of initial task parameters are easily optimized for a new task. The vanishing gradient challenge faced by gradient-based optimization is solved by using a Long shortterm memory (LSTM) based meta-learner (Hochreiter & Schmidhuber (1997)). Memory-based
methods (Ravi & Larochelle (2017)) that use all of the learner’s parameters as input to a metalearner tend to break down when using a learner with a large number of parameters (Andrychowicz et al., 2016). The technique proposed in this work, Nodal Optimization for Recurrent Meta-Learning (NORML), solves this scaling problem by learning layer-wise update signals used to update the learners’s parameters.
NORML is evaluated on the Mini-ImageNet dataset and is shown to improve on existing optimization-based and memory-based methods. An ablation study is done showing the effects of the different components of NORML. Furthermore a comparison is done between NORML and Model Agnostic Meta-Learning (MAML) using the Omniglot dataset. The comparison demonstrates that NORML makes superior parameter updates than the updates made by gradient descent.

heading: 2 PRELIMINARIES

heading: 2.1 BACKPROPAGATION
text: The backpropagation algorithm (Rumelhart et al., 1986) was developed in the 1980’s and has since been the status quo for training neural networks. By recursively applying the chain rule, backpropagation sends gradient signals back through stacked layers of a deep network. Each gradient signal at a particular layer is used to calculate the gradient of the weights connected to that layer.
Consider a neural network with N hidden layers. Let the rows of matrix Wl denote the weights connecting the nodes in the layer below to a hidden node in layer l and let the vector bl denote the biases connected to each node in layer l. Then:
al = Wlhl−1 + bl,hl = fl(al) (1)
Where fl is the activation function used at layer l, al is the layer’s pre-activation output, and hl is referred to as the activations of layer l. Note that h0 = x is the input to the network. The output of the network is given by ŷ = fout(aout). When choosing to use a softmax as the output activation and a cross-entropy loss function, the loss and the output layer’s gradient are calculated as follows:
L = − 1 n ∑ n yn log ŷn + (1− yn) log(1− ŷn) (2)
δaout = ∂L
∂aout = ŷ − y (3)
where L is the cross-entropy loss, n is the network’s output size, and δaout is the gradient at the pre-activation of the output layer. Note that δaout is an n-dimensional vector where each element of δaout describes the corresponding node’s contribution to the loss. By propagating δaout back through the network one obtains the gradients at each of the hidden layers’ nodes:
δal−1 = ∂L ∂al−1 = (W Tl δal) f ′l−1(al−1) (4)
where f ′l () is the derivative of the activation function, is an element-wise multiplication and δal is the pre-activation gradient at layer l. In order to calculate the gradient with regards to the weights, δal is matrix multiplied with the activations of the previous layer:
∂L
∂Wl = δalhl−1 (5)

heading: 2.2 META-LEARNING PROBLEM DEFINITION
text: The N -way K-shot problem is defined using an episodic formulation as proposed in Vinyals et al. (2016). A task Ti is sampled from a task distribution p(T ) and consists of an N -way classification problem. A meta-dataset is divided into a training meta-set Str, a validation meta-set Sval and a testing meta-set Stest where the classes contained in each meta-set is disjointed (i.e. none of the classes in Stest is present in Str and visa versa).
The task Ti consists of a training set Dtr and validation set Dval. The training set Dtr contains K examples from each of Ti’s N classes. The validation set Dval usually contains a larger set of examples from each class in order to give an estimate of the model’s generalization performance on task Ti. Note that a task’s validation set Dval (used to optimize the meta training objective) should not be confused with the meta validation set Sval, which is used for model selection.

heading: 2.3 MODEL AGNOSTIC META-LEARNING
text: Optimization-based meta-learning is an approach to meta-learning where an inner loop is used for fast adaptation to a new task, and an outer loop is used to optimize the inner loop’s training steps. Model Agnostic Meta-Learning (MAML) is one such approach that contains all the key ingredients used in optimization-based meta-learning. The inner loop consists of training a model, fθ via gradient descent on a few-shot learning task, Ti.
θ′ = G(θ,Dtr) (6)
where G is often implemented as a single step of gradient descent on the training set Dtr; θ′i = θ − η∇θLtrTi(fθ). It is often advantageous for G to consist of multiple sequential update steps.
The validation set of Ti is then used to evaluate fθ′i , and the task-specific validation loss, L val Ti , is calculated. The outer loop consists of optimizing the base-parameters θ, using the sum of M different tasks’ validation losses:
θ ← θ − η∇θ M∑
Ti∼p(T )
LvalTi (fθ′i) (7)
where M is referred to as the meta-batch size.
This approach allows a model to learn a set of base-parameters θ, that can quickly adapt to a new unseen task. After meta-training a model, the inner loop procedure can update θ and return taskspecific parameters θ′i using only a few examples of the new task. By differentiating through the inner loop, MAML learns to update the base-parameters θ in such a way that the task-specific paramaters, θ′i, generalizes to unseen examples of task Ti.
In many cases it is preferred to have an inner loop that consists of multiple sequential updates. However the inner loop’s computational graph can become quite large if too many steps are taken. This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner loop Aravind Rajeswaran (2019) Antoniou et al. (2018). This limits MAML to domains where a small amount of update steps are sufficient for learning. The LSTM-based metalearner proposed in this work, allow gradients to effectively flow through a large number of update steps. NORML can therefore be applied to a wide array of domains.

heading: 3 MODEL

heading: 3.1 TRAINING
text: NORML incoporates a neural network based learner with N layers and an LSTM-based metalearner that is used to optimize the learner’s inner loop parameter updates. Figure 1 depicts the process by which the meta-learner calculates the parameter updates for a single layer learner. When multiple layers are optimized the additional gradient signals and layer inputs are added to the meta learner’s input. The meta-learner outputs two update signals for every layer, so when a 4-layer network is used as the learner, the meta-learner outputs 8 update signals.
The high-level operation is as follows (Algorithm 1). First the learner’s task-specific loss, LtrTi(fθ), and layer-gradients, δal , are calculated via cross-entropy and backpropagation. The meta-learner takes δal , hl−1 and the loss, L, as input, and outputs the current cell state and the update signals, ĥl−1 and δ̂al :  δ̂alĥl−1
cl,t+1
= mΦ(δal ,hl−1, L, cl,t) (8)
The update signals are matrix multiplied to determine the parameter updates for layer l:
∆Wl = ηδ̂alĥl−1 (9)
∆bl = ηδ̂al (10)
The learner’s updated parameters, θ′Ti , are then used on the next training example and the whole process gets repeated for U update steps. The inner loop terminates after calculating the task validation loss using fθ′ . LvalTi = CrossEntropy(yval, fθ′(xval)) (11) The outer loop consists of optimizing both θ0 and the meta-learner parameters Φ via gradient descent:
θ ← θ − β∇θ ∑
Ti p(T )
LvalTi (fθ′i) (12)
Φ← Φ− β∇Φ ∑
Ti p(T )
LvalTi (fθ′i) (13)

heading: 3.2 META-LEARNER
text: The meta-learner, m is implemented using a modified LSTM cell as shown in Figure 2.
The input to the meta-learner is normalized, flattened and concatenated. xl is used to denote concatenating the loss, the layer gradients, and the activations of the previous layer.
The meta-learner consists of a forget gate, fl, an input gate, il and an update gate, c̃l. The cell state at update step t is denoted via cl,t, and is calculated as follows:
fl,t = σ(Wl,fxl,t + bl,f ) (14)
il,t = σ(Wl,ixl,t + bl,i) (15) c̃l,t = tanh(Wl,c̃xl,t + bi,c̃) (16) cl,t = fl,t cl,t−1 + il,t c̃l,t (17)
In order to determine the update signals, the cell state, cl,t, is used as input to two separate fullyconnected layers followed by a sigmoid activation function and a pointwise multiplication with the original layer gradient and the previous layer’s activations:
δ̂al = δal σ(Wl,gcl,t + bl,g) (18)
ĥl = hl σ(Wl,hcl,t + bl,h) (19)
Algorithm 1 Nodal Optimization for Rapid Meta-Learning Require: : Learner f with parameters θ Require: : Meta-learner m with parameters Φ Require: : p(T ): distribution over tasks Require: : η, β: step size hyperparameters
1: randomly initialize θ,Φ 2: while not converged do 3: Sample batch of tasks Ti ∼ p(T ) 4: 5: for all Ti do 6: Let (Dtri , D val i ) = Ti 7: Initialize θ′0 = θ 8: 9: for t = 1, len(Dtri ) do
10: Compute learner’s training loss LtrTi(fθ′t−1) and layer gradients ∂LtrTi ∂al for all layers l 11: Compute the update signals: ĥl−1, ĝl = mΦ(LtrTi(fθ′t−1), ∂LtrTi ∂al ,hl−1) 12: Update task parameters: θ′t ← θ′t−1 − ηĝlĥl−1 13: end for 14: Compute validation loss LvalTi (fθ′t) 15: end for 16: Update θ ← θ − β∇θ ∑ Ti p(T ) LvalTi (fθ′t)
17: Update Φ← Φ− β∇Φ ∑ Ti p(T ) LvalTi (fθ′t) 18: end while
The sigmoid activation and pointwise mutliplication operations allow the meta-learner to scale the update signal received by each node in the network. This node-wise scaling let the meta-learner control how the weights connected to a particular node will get changed at any given update step. By scaling the layer gradients, the meta-learner can dynamically control by how much each node’s input should be changed at each update step. Likewise, by scaling the hidden layer activations, the meta-learner can control the change in weighting of each of the previous nodes’ output. By controlling the magnitude of both the activations and the layer gradients, the meta-learner can learn a dynamically adaptive learning rate for each individual weight and bias of the learner. This can be achieved with a meta-learner that scales linearly in size relative to the size of the learner, i.e. increasing the number of learner parameters by a factor of p, will result in the same factor increase in the number of meta-learner parameters.

heading: 4 RELATED WORK
text: Learning how to learn has a long history and variety of approaches have been proposed. Memorybased methods use memory architectures to either store key training examples or to encode fast adaptation algorithms (Santoro et al., 2016) (Andrychowicz et al., 2016) (Ravi & Larochelle, 2017). Metric-based methods are designed to learn similarity metrics for examples of the same class (Koch, 2015) (Vinyals et al., 2016) (Snell et al., 2017). Optimization-based methods aim to learn a set of initial parameters that can quickly adapt to new tasks (Finn et al., 2017).
A subset of memory-based methods use recurrent neural networks to learn fast adaptation algorithms. In Ravi & Larochelle (2017) an LSTM-based meta-learner is trained to iteratively optimize a neural network. At a given timestep the meta-learner’s hidden state consists of the optimizee’s current parameters. At each timestep a training datapoint is used to calculate the optimizee’s parameter gradients. The parameter gradients and the task loss is used as input to the meta-learner. The hidden state of the following timestep is then used as the updated parameters for the learner. A challenge of this approach is that if you want to optimize tens of thousands of parameters, you would have a massive hidden state and input size, and will therefore require an enormous number of parameters for the meta-learner.
In Andrychowicz et al. (2016) an alternative approach is introduced that avoids the aforementioned scaling problem by individually optimizing each parameter using a separate LSTM. This vastly decreases the number of meta-parameters required, however each meta-learner is only given information of a single parameter and can therefore not use information of other parts of the network when doing optimization. The approach introduced in this work uses a single meta-learner to optimize all the parameter’s of the learner, while the meta-learner require less than 5 times the number of parameters used by the learner.
Optimization-based methods (Finn et al., 2017), differentiate through the inner loop in order to optimize learning. Improvements to MAML include using a pre-trained network to extract features and to then only optimize the final fully-connected layer via gradient based optimization (?). Using a pre-trained feature extractor assists in overcoming task overfitting and have been demonstrated to give a large boost in performance(Sun et al., 2018). In this work a pre-trained feature extractor was also used when evaluating NORML on Mini-Imagenet, but is not a requirement for implementing NORML.
Other work that use custom layer-wise update signals include Nø kland (2016) and Nkland & Eidnes (2019). In feedback alignment layer gradient approximating update signals are used to train a neural network by passing the error signal through fixed random matrices. Backpropagation in contrast passes the error signal through the transpose of the subsequent layer’s weight matrix. Surprisingly feedback alignment demonstrates that learning can still occur even when using random matrices for the backward pass.

heading: 5 EXPERIMENTAL RESULTS
text: The proposed approach is evaluated on the Mini-Imagenet benchmark dataset. An ablation study comparing the performance against MAML is conducted on the Omniglot dataset. The evaluation aims to compare the performance of NORML against existing methods and to specifically test if node-wise optimization improves the update signals provided by backpropagation.

heading: 5.1 MINI-IMAGENET CLASSIFICATION
text: The Mini-Imagenet dataset was proposed by Vinyals et al. (2016) and is a popular benchmark for few-shot learning methods. The dataset consists of 100 classes, each containing 600 images. The meta-training, meta-validation and meta-testing sets each contain 64, 16 and 20 classes respectively. It has been demonstrated that by using a pre-trained feature extractor the performance on few-shot learning tasks can be significantly improved (Sun et al. (2018)). For evaluation of NORML on the Mini-Imagenet benchmark, a Resnet-12 network is pre-trained using only data and classes from the meta-training set. The pre-train model is trained to perform 64-way classification. After pretraining, the final FC-layer is replaced by a randomly initialized FC-layer and the convolutional layers are frozen.
The meta-learner network and the final layer of the learner are optimized during meta-training. The inner loop consisted of 10 epochs for both the 5-shot and 1-shot cases, while using a meta-batch size of 4. The model was trained for 10 000 meta-iterations and the meta-validation set was used for model selection for testing.
Classification accuracies for NORML and other baselines are shown in Table 1.

heading: 5.2 ABLATION STUDY
text: For the Mini-Imagenet benchmark the learner consisted of a single FC-layer which was proceeded by a pre-trained feature extractor. NORML can however be applied to networks with multiple layers. To assess the effect of the different components of the meta-learner and to compare NORML to MAML, an ablation study was conducted on the Omniglot 20-way dataset. The learner consisted of a fully-connected network with 4 hidden layers, each including batch normalization and ReLU activations, followed by a linear layer and softmax. The hidden layer sizes were 256, 128, 64 and 64. The same learner network was used for all approaches while only changing the setup of the meta-learner between experiments.
The cell state and hidden state of the meta-learner carries information of all the previous losses, gradients and update steps that was calculated for the task at hand. This enables NORML to potentially improve on the update signals calculated via backpropagation, since the gradient for a particular data-point only contains information from the current time step. To assess the contribution of the cell state and hidden state to the performance of NORML, the following three setups were tested: (1) Setting the cell input state and hidden input state to zero, (2) only setting the hidden state to zero and (3) only setting the cell state to zero.
The meta-learner used in NORML can learn to dynamically control the learning rates of each learner node’s input and output weights. To assess the effect of the node-wise learning rates and to compare NORML to MAML, three different training setups are used for MAML: (1) MAML with a constant learning rate, (2) MAML with a single learnable learning rate and (3) MAML with learnable per-layer learning rates. The results for the 1-shot and 5-shot tasks are shown in Table 2. Note that the results of the ablation study are obtained using a fully-connected network; a convolutional neural network will give higher accuracies.
NORML learns to dynamically control the learning rates of each learner node’s input and output weights. This is shown to give a significant improvement when compared to using constant, learnable, and per-layer learnable learning rates. Intuitively it makes sense, since NORML can effectively control the learning rates of each parameter in the learner network. Additionally NORML can control the learning rates dynamically. Dynamic control seems crucial to NORML’s success, since a parameter that receives a large update at step t, might need to make a small update at step t+ 1. The importance of being able to dynamically adapt the learning rate using prior update information, is confirmed in the experiments where the input hidden state and input cell state of the meta-learner are set to zero. Although the meta-learner can still learn nodal learning rates, it is unable to use information of previous update steps and ends up performing the same as MAML with a constant learning rate.

heading: 6 CONCLUSION AND FUTURE WORK
text: Recent work has shown that the successes of most meta-learning techniques are primarily due to feature reuse (Raghu et al., 2019). Ideally meta-learning techniques should be learning to also make efficient and effective parameter updates. This work introduces a novel meta-learning method that allows node-wise scaling of learner update signals, thereby enabling the meta-learner to dynamically control the learning rate of each of the learner’s parameters. Importantly this is done by a meta-learner who’s size scales linearly relative to the size of the learner.
Future work would focus on scaling NORML to optimize large learner networks. Furthermore, applying NORML to domains where a large number of update steps are required, such as Deep Reinforcement Learning, could prove to be an exciting avenue of further research since the LSTMbased meta-learner allow gradients to propagate back much farther than gradient-based optimization methods (Finn et al., 2017).

heading: A APPENDIX
text: You may include other additional sections here.

Review summaries:
This submission proposes NORML, a meta-learning method that 1) learns initial parameters for a base model that leads to good few-shot learning performance and 2) where a recurrent neural network (LSTM) is used to control the learning updates on a small support set for a given task. The method is derived specifically for full connected neural networks, where the meta-learner produces gating factors on the normal gradients (one for each neuron that the parameter is connecting). The method is compared with various published few-shot learning methods on miniImageNet, and an ablation study and detailed comparison with MAML is presented on Omniglot.

This paper proposes a meta-learner that learns how to make parameter updates for a model on a new few-shot learning task. The proposed meta-learner is an LSTM that proposes at each time-step, a point-wise multiplier for the gradient of the hidden units and for the hidden units of the learner neural network, which are then used to compute a gradient update for the hidden-layer weights of the learner network. By not directly producing a learning rate for the gradient, the meta-learner’s parameters are only proportional to the square of the number of hidden units in the network rather than the square of the number of weights of the network. Experiments are performed on few-shot learning benchmarks. The first experiment is on Mini-ImageNet. The authors build upon the method of Sun et al, where they pre-train the network on the meta-training data and then do meta-training where the convolutional network weights are frozen and only the fully-connected layer is updated on few-shot learning tasks using their meta-learner LSTM. The other experiment is on Omniglot 20-way classification, where they consider a network with only full-connected layers and show that their meta-learner LSTM performs better than MAML.