NORML: Nodal Optimization for Recurrent Meta-Learning

1 INTRODUCTION . Humans have a remarkable capability to learn useful concepts from a small number of examples or a limited amount of experience . In contrast most machine learning methods require large , labelled datasets to learn effectively . Little is understood about the actual learning algorithm ( s ) used by the human brain , and how it relates to machine learning algorithms like backpropagation ( Lillicrap & Körding ( 2019 ) ) . Botvinick et al . ( 2019 ) argues that inductive bias and structured priors are some of the main factors that enable fast learning in animals . In order to build general-purpose systems we must be able to design and build learning algorithms that can quickly and effectively learn from a limited amount of data by utilizing prior knowledge and experience . Supervised few-shot learning aims to challenge machine learning models to learn new tasks by leveraging only a handful of labelled examples . Vinyals et al . ( 2016 ) introduces the few-shot learning problem for image classification , where a model is tasked to classify a number of images while being provided either 1 or 5 examples of each class ( hereafter referred to as 1-shot and 5-shot learning ) . One way to approach this problem is by way of meta-learning , a broad family of techniques that aim to learn how to learn ( Thrun & Pratt ( 1998 ) ) . One particularly powerful group of approaches , known as memory-based methods , use memory architectures that can leverage prior information to assist in learning ( Santoro et al . ( 2016 ) ; Ravi & Larochelle ( 2017 ) ) . Optimization-based methods ( Finn et al . ( 2017 ) ) is another exciting area that aim to learn an initial set of parameters that can quickly adapt to new , unseen tasks with relatively little training data . This work introduces a novel technique where a recurrent neural network based meta-learner learns how to make parameter updates for a task learner . The meta-learner and the learner are jointly trained so as to learn how to learn new tasks with little data . This approach allows one to utilize aspects from both optimization-based and memory-based meta-learning methods . The recurrent architecture of the meta-learner can use important prior information when updating the learner , while the learner can learn a set of initial task parameters are easily optimized for a new task . The vanishing gradient challenge faced by gradient-based optimization is solved by using a Long shortterm memory ( LSTM ) based meta-learner ( Hochreiter & Schmidhuber ( 1997 ) ) . Memory-based methods ( Ravi & Larochelle ( 2017 ) ) that use all of the learner ’ s parameters as input to a metalearner tend to break down when using a learner with a large number of parameters ( Andrychowicz et al. , 2016 ) . The technique proposed in this work , Nodal Optimization for Recurrent Meta-Learning ( NORML ) , solves this scaling problem by learning layer-wise update signals used to update the learners ’ s parameters . NORML is evaluated on the Mini-ImageNet dataset and is shown to improve on existing optimization-based and memory-based methods . An ablation study is done showing the effects of the different components of NORML . Furthermore a comparison is done between NORML and Model Agnostic Meta-Learning ( MAML ) using the Omniglot dataset . The comparison demonstrates that NORML makes superior parameter updates than the updates made by gradient descent . 2 PRELIMINARIES . 2.1 BACKPROPAGATION . The backpropagation algorithm ( Rumelhart et al. , 1986 ) was developed in the 1980 ’ s and has since been the status quo for training neural networks . By recursively applying the chain rule , backpropagation sends gradient signals back through stacked layers of a deep network . Each gradient signal at a particular layer is used to calculate the gradient of the weights connected to that layer . Consider a neural network with N hidden layers . Let the rows of matrix Wl denote the weights connecting the nodes in the layer below to a hidden node in layer l and let the vector bl denote the biases connected to each node in layer l. Then : al = Wlhl−1 + bl , hl = fl ( al ) ( 1 ) Where fl is the activation function used at layer l , al is the layer ’ s pre-activation output , and hl is referred to as the activations of layer l. Note that h0 = x is the input to the network . The output of the network is given by ŷ = fout ( aout ) . When choosing to use a softmax as the output activation and a cross-entropy loss function , the loss and the output layer ’ s gradient are calculated as follows : L = − 1 n ∑ n yn log ŷn + ( 1− yn ) log ( 1− ŷn ) ( 2 ) δaout = ∂L ∂aout = ŷ − y ( 3 ) where L is the cross-entropy loss , n is the network ’ s output size , and δaout is the gradient at the pre-activation of the output layer . Note that δaout is an n-dimensional vector where each element of δaout describes the corresponding node ’ s contribution to the loss . By propagating δaout back through the network one obtains the gradients at each of the hidden layers ’ nodes : δal−1 = ∂L ∂al−1 = ( W Tl δal ) f ′l−1 ( al−1 ) ( 4 ) where f ′l ( ) is the derivative of the activation function , is an element-wise multiplication and δal is the pre-activation gradient at layer l. In order to calculate the gradient with regards to the weights , δal is matrix multiplied with the activations of the previous layer : ∂L ∂Wl = δalhl−1 ( 5 ) 2.2 META-LEARNING PROBLEM DEFINITION . The N -way K-shot problem is defined using an episodic formulation as proposed in Vinyals et al . ( 2016 ) . A task Ti is sampled from a task distribution p ( T ) and consists of an N -way classification problem . A meta-dataset is divided into a training meta-set Str , a validation meta-set Sval and a testing meta-set Stest where the classes contained in each meta-set is disjointed ( i.e . none of the classes in Stest is present in Str and visa versa ) . The task Ti consists of a training set Dtr and validation set Dval . The training set Dtr contains K examples from each of Ti ’ s N classes . The validation set Dval usually contains a larger set of examples from each class in order to give an estimate of the model ’ s generalization performance on task Ti . Note that a task ’ s validation set Dval ( used to optimize the meta training objective ) should not be confused with the meta validation set Sval , which is used for model selection . 2.3 MODEL AGNOSTIC META-LEARNING . Optimization-based meta-learning is an approach to meta-learning where an inner loop is used for fast adaptation to a new task , and an outer loop is used to optimize the inner loop ’ s training steps . Model Agnostic Meta-Learning ( MAML ) is one such approach that contains all the key ingredients used in optimization-based meta-learning . The inner loop consists of training a model , fθ via gradient descent on a few-shot learning task , Ti . θ′ = G ( θ , Dtr ) ( 6 ) where G is often implemented as a single step of gradient descent on the training set Dtr ; θ′i = θ − η∇θLtrTi ( fθ ) . It is often advantageous for G to consist of multiple sequential update steps . The validation set of Ti is then used to evaluate fθ′i , and the task-specific validation loss , L val Ti , is calculated . The outer loop consists of optimizing the base-parameters θ , using the sum of M different tasks ’ validation losses : θ ← θ − η∇θ M∑ Ti∼p ( T ) LvalTi ( fθ′i ) ( 7 ) where M is referred to as the meta-batch size . This approach allows a model to learn a set of base-parameters θ , that can quickly adapt to a new unseen task . After meta-training a model , the inner loop procedure can update θ and return taskspecific parameters θ′i using only a few examples of the new task . By differentiating through the inner loop , MAML learns to update the base-parameters θ in such a way that the task-specific paramaters , θ′i , generalizes to unseen examples of task Ti . In many cases it is preferred to have an inner loop that consists of multiple sequential updates . However the inner loop ’ s computational graph can become quite large if too many steps are taken . This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner loop Aravind Rajeswaran ( 2019 ) Antoniou et al . ( 2018 ) . This limits MAML to domains where a small amount of update steps are sufficient for learning . The LSTM-based metalearner proposed in this work , allow gradients to effectively flow through a large number of update steps . NORML can therefore be applied to a wide array of domains . 3 MODEL . 3.1 TRAINING . NORML incoporates a neural network based learner with N layers and an LSTM-based metalearner that is used to optimize the learner ’ s inner loop parameter updates . Figure 1 depicts the process by which the meta-learner calculates the parameter updates for a single layer learner . When multiple layers are optimized the additional gradient signals and layer inputs are added to the meta learner ’ s input . The meta-learner outputs two update signals for every layer , so when a 4-layer network is used as the learner , the meta-learner outputs 8 update signals . The high-level operation is as follows ( Algorithm 1 ) . First the learner ’ s task-specific loss , LtrTi ( fθ ) , and layer-gradients , δal , are calculated via cross-entropy and backpropagation . The meta-learner takes δal , hl−1 and the loss , L , as input , and outputs the current cell state and the update signals , ĥl−1 and δ̂al : δ̂alĥl−1 cl , t+1 = mΦ ( δal , hl−1 , L , cl , t ) ( 8 ) The update signals are matrix multiplied to determine the parameter updates for layer l : ∆Wl = ηδ̂alĥl−1 ( 9 ) ∆bl = ηδ̂al ( 10 ) The learner ’ s updated parameters , θ′Ti , are then used on the next training example and the whole process gets repeated for U update steps . The inner loop terminates after calculating the task validation loss using fθ′ . LvalTi = CrossEntropy ( yval , fθ′ ( xval ) ) ( 11 ) The outer loop consists of optimizing both θ0 and the meta-learner parameters Φ via gradient descent : θ ← θ − β∇θ ∑ Ti p ( T ) LvalTi ( fθ′i ) ( 12 ) Φ← Φ− β∇Φ ∑ Ti p ( T ) LvalTi ( fθ′i ) ( 13 ) 3.2 META-LEARNER . The meta-learner , m is implemented using a modified LSTM cell as shown in Figure 2 . The input to the meta-learner is normalized , flattened and concatenated . xl is used to denote concatenating the loss , the layer gradients , and the activations of the previous layer . The meta-learner consists of a forget gate , fl , an input gate , il and an update gate , c̃l . The cell state at update step t is denoted via cl , t , and is calculated as follows : fl , t = σ ( Wl , fxl , t + bl , f ) ( 14 ) il , t = σ ( Wl , ixl , t + bl , i ) ( 15 ) c̃l , t = tanh ( Wl , c̃xl , t + bi , c̃ ) ( 16 ) cl , t = fl , t cl , t−1 + il , t c̃l , t ( 17 ) In order to determine the update signals , the cell state , cl , t , is used as input to two separate fullyconnected layers followed by a sigmoid activation function and a pointwise multiplication with the original layer gradient and the previous layer ’ s activations : δ̂al = δal σ ( Wl , gcl , t + bl , g ) ( 18 ) ĥl = hl σ ( Wl , hcl , t + bl , h ) ( 19 ) Algorithm 1 Nodal Optimization for Rapid Meta-Learning Require : : Learner f with parameters θ Require : : Meta-learner m with parameters Φ Require : : p ( T ) : distribution over tasks Require : : η , β : step size hyperparameters 1 : randomly initialize θ , Φ 2 : while not converged do 3 : Sample batch of tasks Ti ∼ p ( T ) 4 : 5 : for all Ti do 6 : Let ( Dtri , D val i ) = Ti 7 : Initialize θ′0 = θ 8 : 9 : for t = 1 , len ( Dtri ) do 10 : Compute learner ’ s training loss LtrTi ( fθ′t−1 ) and layer gradients ∂LtrTi ∂al for all layers l 11 : Compute the update signals : ĥl−1 , ĝl = mΦ ( LtrTi ( fθ′t−1 ) , ∂LtrTi ∂al , hl−1 ) 12 : Update task parameters : θ′t ← θ′t−1 − ηĝlĥl−1 13 : end for 14 : Compute validation loss LvalTi ( fθ′t ) 15 : end for 16 : Update θ ← θ − β∇θ ∑ Ti p ( T ) LvalTi ( fθ′t ) 17 : Update Φ← Φ− β∇Φ ∑ Ti p ( T ) LvalTi ( fθ′t ) 18 : end while The sigmoid activation and pointwise mutliplication operations allow the meta-learner to scale the update signal received by each node in the network . This node-wise scaling let the meta-learner control how the weights connected to a particular node will get changed at any given update step . By scaling the layer gradients , the meta-learner can dynamically control by how much each node ’ s input should be changed at each update step . Likewise , by scaling the hidden layer activations , the meta-learner can control the change in weighting of each of the previous nodes ’ output . By controlling the magnitude of both the activations and the layer gradients , the meta-learner can learn a dynamically adaptive learning rate for each individual weight and bias of the learner . This can be achieved with a meta-learner that scales linearly in size relative to the size of the learner , i.e . increasing the number of learner parameters by a factor of p , will result in the same factor increase in the number of meta-learner parameters .

Summary: This paper proposes a meta-learner that learns how to make parameter updates for a model on a new few-shot learning task. The proposed meta-learner is an LSTM that proposes at each time-step, a point-wise multiplier for the gradient of the hidden units and for the hidden units of the learner neural network, which are then used to compute a gradient update for the hidden-layer weights of the learner network. By not directly producing a learning rate for the gradient, the meta-learner’s parameters are only proportional to the square of the number of hidden units in the network rather than the square of the number of weights of the network. Experiments are performed on few-shot learning benchmarks. The first experiment is on Mini-ImageNet. The authors build upon the method of Sun et al, where they pre-train the network on the meta-training data and then do meta-training where the convolutional network weights are frozen and only the fully-connected layer is updated on few-shot learning tasks using their meta-learner LSTM. The other experiment is on Omniglot 20-way classification, where they consider a network with only full-connected layers and show that their meta-learner LSTM performs better than MAML.