NORML: Nodal Optimization for Recurrent Meta-Learning

1 INTRODUCTION . Humans have a remarkable capability to learn useful concepts from a small number of examples or a limited amount of experience . In contrast most machine learning methods require large , labelled datasets to learn effectively . Little is understood about the actual learning algorithm ( s ) used by the human brain , and how it relates to machine learning algorithms like backpropagation ( Lillicrap & Körding ( 2019 ) ) . Botvinick et al . ( 2019 ) argues that inductive bias and structured priors are some of the main factors that enable fast learning in animals . In order to build general-purpose systems we must be able to design and build learning algorithms that can quickly and effectively learn from a limited amount of data by utilizing prior knowledge and experience . Supervised few-shot learning aims to challenge machine learning models to learn new tasks by leveraging only a handful of labelled examples . Vinyals et al . ( 2016 ) introduces the few-shot learning problem for image classification , where a model is tasked to classify a number of images while being provided either 1 or 5 examples of each class ( hereafter referred to as 1-shot and 5-shot learning ) . One way to approach this problem is by way of meta-learning , a broad family of techniques that aim to learn how to learn ( Thrun & Pratt ( 1998 ) ) . One particularly powerful group of approaches , known as memory-based methods , use memory architectures that can leverage prior information to assist in learning ( Santoro et al . ( 2016 ) ; Ravi & Larochelle ( 2017 ) ) . Optimization-based methods ( Finn et al . ( 2017 ) ) is another exciting area that aim to learn an initial set of parameters that can quickly adapt to new , unseen tasks with relatively little training data . This work introduces a novel technique where a recurrent neural network based meta-learner learns how to make parameter updates for a task learner . The meta-learner and the learner are jointly trained so as to learn how to learn new tasks with little data . This approach allows one to utilize aspects from both optimization-based and memory-based meta-learning methods . The recurrent architecture of the meta-learner can use important prior information when updating the learner , while the learner can learn a set of initial task parameters are easily optimized for a new task . The vanishing gradient challenge faced by gradient-based optimization is solved by using a Long shortterm memory ( LSTM ) based meta-learner ( Hochreiter & Schmidhuber ( 1997 ) ) . Memory-based methods ( Ravi & Larochelle ( 2017 ) ) that use all of the learner ’ s parameters as input to a metalearner tend to break down when using a learner with a large number of parameters ( Andrychowicz et al. , 2016 ) . The technique proposed in this work , Nodal Optimization for Recurrent Meta-Learning ( NORML ) , solves this scaling problem by learning layer-wise update signals used to update the learners ’ s parameters . NORML is evaluated on the Mini-ImageNet dataset and is shown to improve on existing optimization-based and memory-based methods . An ablation study is done showing the effects of the different components of NORML . Furthermore a comparison is done between NORML and Model Agnostic Meta-Learning ( MAML ) using the Omniglot dataset . The comparison demonstrates that NORML makes superior parameter updates than the updates made by gradient descent . 2 PRELIMINARIES . 2.1 BACKPROPAGATION . The backpropagation algorithm ( Rumelhart et al. , 1986 ) was developed in the 1980 ’ s and has since been the status quo for training neural networks . By recursively applying the chain rule , backpropagation sends gradient signals back through stacked layers of a deep network . Each gradient signal at a particular layer is used to calculate the gradient of the weights connected to that layer . Consider a neural network with N hidden layers . Let the rows of matrix Wl denote the weights connecting the nodes in the layer below to a hidden node in layer l and let the vector bl denote the biases connected to each node in layer l. Then : al = Wlhl−1 + bl , hl = fl ( al ) ( 1 ) Where fl is the activation function used at layer l , al is the layer ’ s pre-activation output , and hl is referred to as the activations of layer l. Note that h0 = x is the input to the network . The output of the network is given by ŷ = fout ( aout ) . When choosing to use a softmax as the output activation and a cross-entropy loss function , the loss and the output layer ’ s gradient are calculated as follows : L = − 1 n ∑ n yn log ŷn + ( 1− yn ) log ( 1− ŷn ) ( 2 ) δaout = ∂L ∂aout = ŷ − y ( 3 ) where L is the cross-entropy loss , n is the network ’ s output size , and δaout is the gradient at the pre-activation of the output layer . Note that δaout is an n-dimensional vector where each element of δaout describes the corresponding node ’ s contribution to the loss . By propagating δaout back through the network one obtains the gradients at each of the hidden layers ’ nodes : δal−1 = ∂L ∂al−1 = ( W Tl δal ) f ′l−1 ( al−1 ) ( 4 ) where f ′l ( ) is the derivative of the activation function , is an element-wise multiplication and δal is the pre-activation gradient at layer l. In order to calculate the gradient with regards to the weights , δal is matrix multiplied with the activations of the previous layer : ∂L ∂Wl = δalhl−1 ( 5 ) 2.2 META-LEARNING PROBLEM DEFINITION . The N -way K-shot problem is defined using an episodic formulation as proposed in Vinyals et al . ( 2016 ) . A task Ti is sampled from a task distribution p ( T ) and consists of an N -way classification problem . A meta-dataset is divided into a training meta-set Str , a validation meta-set Sval and a testing meta-set Stest where the classes contained in each meta-set is disjointed ( i.e . none of the classes in Stest is present in Str and visa versa ) . The task Ti consists of a training set Dtr and validation set Dval . The training set Dtr contains K examples from each of Ti ’ s N classes . The validation set Dval usually contains a larger set of examples from each class in order to give an estimate of the model ’ s generalization performance on task Ti . Note that a task ’ s validation set Dval ( used to optimize the meta training objective ) should not be confused with the meta validation set Sval , which is used for model selection . 2.3 MODEL AGNOSTIC META-LEARNING . Optimization-based meta-learning is an approach to meta-learning where an inner loop is used for fast adaptation to a new task , and an outer loop is used to optimize the inner loop ’ s training steps . Model Agnostic Meta-Learning ( MAML ) is one such approach that contains all the key ingredients used in optimization-based meta-learning . The inner loop consists of training a model , fθ via gradient descent on a few-shot learning task , Ti . θ′ = G ( θ , Dtr ) ( 6 ) where G is often implemented as a single step of gradient descent on the training set Dtr ; θ′i = θ − η∇θLtrTi ( fθ ) . It is often advantageous for G to consist of multiple sequential update steps . The validation set of Ti is then used to evaluate fθ′i , and the task-specific validation loss , L val Ti , is calculated . The outer loop consists of optimizing the base-parameters θ , using the sum of M different tasks ’ validation losses : θ ← θ − η∇θ M∑ Ti∼p ( T ) LvalTi ( fθ′i ) ( 7 ) where M is referred to as the meta-batch size . This approach allows a model to learn a set of base-parameters θ , that can quickly adapt to a new unseen task . After meta-training a model , the inner loop procedure can update θ and return taskspecific parameters θ′i using only a few examples of the new task . By differentiating through the inner loop , MAML learns to update the base-parameters θ in such a way that the task-specific paramaters , θ′i , generalizes to unseen examples of task Ti . In many cases it is preferred to have an inner loop that consists of multiple sequential updates . However the inner loop ’ s computational graph can become quite large if too many steps are taken . This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner loop Aravind Rajeswaran ( 2019 ) Antoniou et al . ( 2018 ) . This limits MAML to domains where a small amount of update steps are sufficient for learning . The LSTM-based metalearner proposed in this work , allow gradients to effectively flow through a large number of update steps . NORML can therefore be applied to a wide array of domains . 3 MODEL . 3.1 TRAINING . NORML incoporates a neural network based learner with N layers and an LSTM-based metalearner that is used to optimize the learner ’ s inner loop parameter updates . Figure 1 depicts the process by which the meta-learner calculates the parameter updates for a single layer learner . When multiple layers are optimized the additional gradient signals and layer inputs are added to the meta learner ’ s input . The meta-learner outputs two update signals for every layer , so when a 4-layer network is used as the learner , the meta-learner outputs 8 update signals . The high-level operation is as follows ( Algorithm 1 ) . First the learner ’ s task-specific loss , LtrTi ( fθ ) , and layer-gradients , δal , are calculated via cross-entropy and backpropagation . The meta-learner takes δal , hl−1 and the loss , L , as input , and outputs the current cell state and the update signals , ĥl−1 and δ̂al : δ̂alĥl−1 cl , t+1 = mΦ ( δal , hl−1 , L , cl , t ) ( 8 ) The update signals are matrix multiplied to determine the parameter updates for layer l : ∆Wl = ηδ̂alĥl−1 ( 9 ) ∆bl = ηδ̂al ( 10 ) The learner ’ s updated parameters , θ′Ti , are then used on the next training example and the whole process gets repeated for U update steps . The inner loop terminates after calculating the task validation loss using fθ′ . LvalTi = CrossEntropy ( yval , fθ′ ( xval ) ) ( 11 ) The outer loop consists of optimizing both θ0 and the meta-learner parameters Φ via gradient descent : θ ← θ − β∇θ ∑ Ti p ( T ) LvalTi ( fθ′i ) ( 12 ) Φ← Φ− β∇Φ ∑ Ti p ( T ) LvalTi ( fθ′i ) ( 13 ) 3.2 META-LEARNER . The meta-learner , m is implemented using a modified LSTM cell as shown in Figure 2 . The input to the meta-learner is normalized , flattened and concatenated . xl is used to denote concatenating the loss , the layer gradients , and the activations of the previous layer . The meta-learner consists of a forget gate , fl , an input gate , il and an update gate , c̃l . The cell state at update step t is denoted via cl , t , and is calculated as follows : fl , t = σ ( Wl , fxl , t + bl , f ) ( 14 ) il , t = σ ( Wl , ixl , t + bl , i ) ( 15 ) c̃l , t = tanh ( Wl , c̃xl , t + bi , c̃ ) ( 16 ) cl , t = fl , t cl , t−1 + il , t c̃l , t ( 17 ) In order to determine the update signals , the cell state , cl , t , is used as input to two separate fullyconnected layers followed by a sigmoid activation function and a pointwise multiplication with the original layer gradient and the previous layer ’ s activations : δ̂al = δal σ ( Wl , gcl , t + bl , g ) ( 18 ) ĥl = hl σ ( Wl , hcl , t + bl , h ) ( 19 ) Algorithm 1 Nodal Optimization for Rapid Meta-Learning Require : : Learner f with parameters θ Require : : Meta-learner m with parameters Φ Require : : p ( T ) : distribution over tasks Require : : η , β : step size hyperparameters 1 : randomly initialize θ , Φ 2 : while not converged do 3 : Sample batch of tasks Ti ∼ p ( T ) 4 : 5 : for all Ti do 6 : Let ( Dtri , D val i ) = Ti 7 : Initialize θ′0 = θ 8 : 9 : for t = 1 , len ( Dtri ) do 10 : Compute learner ’ s training loss LtrTi ( fθ′t−1 ) and layer gradients ∂LtrTi ∂al for all layers l 11 : Compute the update signals : ĥl−1 , ĝl = mΦ ( LtrTi ( fθ′t−1 ) , ∂LtrTi ∂al , hl−1 ) 12 : Update task parameters : θ′t ← θ′t−1 − ηĝlĥl−1 13 : end for 14 : Compute validation loss LvalTi ( fθ′t ) 15 : end for 16 : Update θ ← θ − β∇θ ∑ Ti p ( T ) LvalTi ( fθ′t ) 17 : Update Φ← Φ− β∇Φ ∑ Ti p ( T ) LvalTi ( fθ′t ) 18 : end while The sigmoid activation and pointwise mutliplication operations allow the meta-learner to scale the update signal received by each node in the network . This node-wise scaling let the meta-learner control how the weights connected to a particular node will get changed at any given update step . By scaling the layer gradients , the meta-learner can dynamically control by how much each node ’ s input should be changed at each update step . Likewise , by scaling the hidden layer activations , the meta-learner can control the change in weighting of each of the previous nodes ’ output . By controlling the magnitude of both the activations and the layer gradients , the meta-learner can learn a dynamically adaptive learning rate for each individual weight and bias of the learner . This can be achieved with a meta-learner that scales linearly in size relative to the size of the learner , i.e . increasing the number of learner parameters by a factor of p , will result in the same factor increase in the number of meta-learner parameters .

Summary: This submission proposes NORML, a meta-learning method that 1) learns initial parameters for a base model that leads to good few-shot learning performance and 2) where a recurrent neural network (LSTM) is used to control the learning updates on a small support set for a given task. The method is derived specifically for full connected neural networks, where the meta-learner produces gating factors on the normal gradients (one for each neuron that the parameter is connecting). The method is compared with various published few-shot learning methods on miniImageNet, and an ablation study and detailed comparison with MAML is presented on Omniglot.