CAML: Fast Context Adaptation via Meta-Learning

1 INTRODUCTION . A key challenge in meta-learning is fast adaptation : learning on previously unseen tasks fast and with little data . In principle , this can be achieved by leveraging knowledge obtained in other , related tasks . However , the best way to do so remains an open question . A popular recent method for fast adaptation is model agnostic meta learning ( MAML ) ( Finn et al. , 2017a ) , which learns a model initialisation , such that at test time the model can be adapted to solve the new task in only a few gradient steps . MAML has an interleaved training procedure , comprised of inner loop and outer loop updates that operate on a batch of tasks at each iteration . In the inner loop , MAML learns task-specific parameters by performing one gradient step on a task-specific loss . Then , in the outer loop , the model parameters from before the inner loop update are updated to reduce the expected loss across tasks after the inner loop update on the individual tasks . Hence , MAML learns a model initialisation that , at test time , can generalise to a new task after only a few gradient updates . However , while MAML adapts the entire model to the new task , many transfer learning algorithms adapt only a fraction of the model ( Kokkinos , 2017 ) , keeping the rest fixed across tasks . For example , representations learned for images can be transferred to different image classification domains ( Donahue et al. , 2014 ) or reused for object tracking in videos ( Wojke et al. , 2017 ) . This suggests that some model parameters can be considered task independent and others task specific . Adapting only some model parameters can make learning faster and easier , as well as mitigate overfitting and catastrophic forgetting . To this end , we propose context adaptation for meta-learning ( CAML ) , a new method for fast adaptation via meta-learning . Like MAML , CAML learns a model initialisation that can quickly be adapted to new tasks . However , unlike MAML , it adapts only a subset of the model parameters to the new task . While restricting adaptation in this way is straightforward , it raises a key question : how should we decide which parameters to adapt and which to keep fixed ? The main insight behind CAML is that , for many fast adaptation problems , the inner loop reduces to a task identification problem , rather than learning how to solve the whole task , which is typically infeasible with only a few gradient updates . Thus , it suffices if the part of the model that varies across tasks is an additional input to the model , and is independent of its other inputs . These additional inputs , which we call context parameters ( see Figure 1 ) , can be interpreted as a task embedding that modulates the behaviour of the model . This embedding is learned via backpropagation during the inner loop of a meta-learning procedure similar to MAML , while the rest of the model is updated only in the outer loop . This allows CAML to explicitly optimise the task-independent parameters ✓ for good performance across tasks , while ensuring that the task-specific context parameters can quickly adapt to new tasks . This separation of task solver and task embedding has several advantages . First , the size of both components can be chosen appropriately for the task . In particular , the network can be made expressive enough without overfitting to a single task in the inner loop , which we show empirically MAML is prone to . Model design and architecture choice also benefit from this separation , since for many practical problems we have prior knowledge of which aspects vary across tasks and hence how much capacity the context parameter should have . Like MAML , our method is model-agnostic , i.e. , it can be applied to any model that is trained via gradient descent . However , CAML is easier to implement : assigning the correct computational graphs for higher order gradients is done only on the level of the context parameters , avoiding manual access and operations on the network weights and biases . Furthermore , parameter copies are not necessary which saves memory writes , a common bottleneck for running on GPUs . CAML can also help distributed machine learning systems , where the same model is deployed to different machines and we wish to learn different contexts concurrently . Network communication is often the bottleneck , which is mitigated by only sharing the ( gradients of ) context parameters . We show empirically that CAML outperforms MAML on a regression and classification task and performs similarly on a reinforcement learning problem , while adapting significantly fewer parameters at test time . We observe that CAML is less sensitive to the inner-loop learning rate , and can be scaled up to larger networks without overfitting . We also demonstrate that the context parameters represent meaningful embeddings of tasks , confirming that the inner loop acts as a task identification step . 2 BACKGROUND : META-LEARNING FOR FAST ADAPTATION . We consider settings where the goal is to learn models that can quickly adapt to a new task with only little data . To this end , learning on the new task is preceded by meta-learning on a set of related tasks . Here we describe the meta-learning problem for supervised and reinforcement learning , as well as the method MAML . 2.1 PROBLEM SETTING . In few-shot learning problems , we are given distributions over training tasks ptrain ( T ) and test tasks ptest ( T ) . Training tasks can be used to learn how to adapt fast to any of the tasks with little per-task data , and evaluation is done on previously unseen test tasks . Unless stated otherwise , we assume that ptrain = ptest and refer to both as p. Tasks in p typically share some structure , so that transferring knowledge between tasks speeds learning . During each meta-training iteration , a batch of N tasks T = { Ti } Ni=1 is sampled from p. Supervised Learning . In a supervised learning setting , we learn a model f that maps data points x 2 X that have a true label y 2 Y to predictions ŷ 2 Y . A task Ti is defined as a tuple Ti = ( X , Y , L , q ) , where X is the input space , Y is the output space , L ( y , ŷ ) is a task-specific loss function , and q ( x , y ) is a distribution over labelled data points . We assume that all data points are drawn i.i.d . from q . Different tasks can be created by changing any element of Ti . Training in the supervised meta-learning setting proceeds over meta-training iterations , where for every task Ti 2 T from the current batch , we sample two datasets Dtraini ( for training ) and Dtesti from qTi : Dtrain i = { ( x , y ) i , m } M train i m=1 , Dtesti = { ( x , y ) i , m } M test i m=1 , ( 1 ) where ( x , y ) ⇠ qTi . M traini and M testi are the number of training and test datapoints , respectively . The training data is used to update f , and the test data is then used to evaluate how good this update was , and adjust f or the update rule accordingly . Reinforcement Learning . In a reinforcement learning ( RL ) setting , we aim to learn a policy ⇡ that maps states s 2 S to actions a 2 A . Each task corresponds to a Markov decision process ( MDP ) : a tuple Ti = ( S , A , r , q , q0 ) , where S is a set of states , A is a set of actions , r ( st , at , st+1 ) is a reward function , q ( st+1|st , at ) is a transition function , and q0 ( s0 ) is an initial state distribution . The goal is to maximise the expected cumulative reward J under ⇡ , J ( ⇡ ) = Eq0 , q , ⇡ '' H 1X t=0 t r ( st , at , st+1 ) # , ( 2 ) where H 2 N [ 1 is the horizon and 2 [ 0 , 1 ] is the discount factor . Again , different tasks can be created by changing any element of Ti . During each meta-training iteration , for every task Ti 2 T from the current batch , we first collect a trajectory ⌧ train i = { s0 , a0 , r0 , s1 , a1 , r1 , . . . , sM traini 1 , aM traini 1 , rM traini 1 , sM traini } , ( 3 ) where the initial state s0 is sampled from q0 , the actions are chosen by the current policy ⇡ , the state transitions according to q , and M train i is the number of environment interactions available , We unify several episodes in this formulation : if the horizon H is reached within the trajectory , the environment is reset using q0 . Once the trajectory is collected , this data is used to update the policy . Another trajectory ⌧ test i is then collected by rolling out the updated policy for M test i time steps . This test trajectory is used to evaluate the quality of the update on that task , and to adjust ⇡ or the update rule accordingly . Evaluation for both supervised and reinforcement learning problems is done on a new ( unseen ) set of tasks drawn from p ( or ptest if the test distribution of task is different ) . For each such task , the model is updated using L or J and only few datapoints ( Dtrain or ⌧ train ) . Performance of the updated model is reported on Dtest or ⌧ test . 2.2 MODEL-AGNOSTIC META-LEARNING . One method for few-shot learning is model-agnostic meta-learning ( Finn et al. , 2017a , MAML ) . Here , we describe the application of MAML to a supervised learning setting . MAML learns an initialisation for the parameters ✓ of a model f✓ such that , given a new task , a good model for that task can be learned with only a small number of gradient steps and data points . In the inner loop , MAML computes new task-specific parameters ✓i ( starting from ✓ ) via one1 gradient update , ✓i = ✓ ↵r✓ 1 M i train X ( x , y ) 2Dtraini LTi ( f✓ ( x ) , y ) . ( 4 ) For the meta-update in the outer loop , the original model parameters are then updated with respect to the performance after the inner-loop update , i.e. , ✓ ✓ r✓ 1 N X Ti2T 1 M i test X ( x , y ) 2Dtesti LTi ( f✓i ( x ) , y ) . ( 5 ) The result of training is a model initialisation ✓ that can be adapted with just a few gradient steps to any new task that we draw from p. Since the gradient is taken with respect to the parameters ✓ before the inner-loop update ( 4 ) , the outer-loop update ( 5 ) involves higher order derivatives in ✓ .

Summary: They are proposing a meta-learning method inspired by previous method, MAML. Their idea is separating the parameters in to two groups of context and shared parameters. The context parameters are learned through back-propagation of inner-loop and represents embedding for individual task. Shared-parameters on the other hand are shared between all tasks, and are learned in the outer-loop.