CAML: Fast Context Adaptation via Meta-Learning

1 INTRODUCTION . A key challenge in meta-learning is fast adaptation : learning on previously unseen tasks fast and with little data . In principle , this can be achieved by leveraging knowledge obtained in other , related tasks . However , the best way to do so remains an open question . A popular recent method for fast adaptation is model agnostic meta learning ( MAML ) ( Finn et al. , 2017a ) , which learns a model initialisation , such that at test time the model can be adapted to solve the new task in only a few gradient steps . MAML has an interleaved training procedure , comprised of inner loop and outer loop updates that operate on a batch of tasks at each iteration . In the inner loop , MAML learns task-specific parameters by performing one gradient step on a task-specific loss . Then , in the outer loop , the model parameters from before the inner loop update are updated to reduce the expected loss across tasks after the inner loop update on the individual tasks . Hence , MAML learns a model initialisation that , at test time , can generalise to a new task after only a few gradient updates . However , while MAML adapts the entire model to the new task , many transfer learning algorithms adapt only a fraction of the model ( Kokkinos , 2017 ) , keeping the rest fixed across tasks . For example , representations learned for images can be transferred to different image classification domains ( Donahue et al. , 2014 ) or reused for object tracking in videos ( Wojke et al. , 2017 ) . This suggests that some model parameters can be considered task independent and others task specific . Adapting only some model parameters can make learning faster and easier , as well as mitigate overfitting and catastrophic forgetting . To this end , we propose context adaptation for meta-learning ( CAML ) , a new method for fast adaptation via meta-learning . Like MAML , CAML learns a model initialisation that can quickly be adapted to new tasks . However , unlike MAML , it adapts only a subset of the model parameters to the new task . While restricting adaptation in this way is straightforward , it raises a key question : how should we decide which parameters to adapt and which to keep fixed ? The main insight behind CAML is that , for many fast adaptation problems , the inner loop reduces to a task identification problem , rather than learning how to solve the whole task , which is typically infeasible with only a few gradient updates . Thus , it suffices if the part of the model that varies across tasks is an additional input to the model , and is independent of its other inputs . These additional inputs , which we call context parameters ( see Figure 1 ) , can be interpreted as a task embedding that modulates the behaviour of the model . This embedding is learned via backpropagation during the inner loop of a meta-learning procedure similar to MAML , while the rest of the model is updated only in the outer loop . This allows CAML to explicitly optimise the task-independent parameters ✓ for good performance across tasks , while ensuring that the task-specific context parameters can quickly adapt to new tasks . This separation of task solver and task embedding has several advantages . First , the size of both components can be chosen appropriately for the task . In particular , the network can be made expressive enough without overfitting to a single task in the inner loop , which we show empirically MAML is prone to . Model design and architecture choice also benefit from this separation , since for many practical problems we have prior knowledge of which aspects vary across tasks and hence how much capacity the context parameter should have . Like MAML , our method is model-agnostic , i.e. , it can be applied to any model that is trained via gradient descent . However , CAML is easier to implement : assigning the correct computational graphs for higher order gradients is done only on the level of the context parameters , avoiding manual access and operations on the network weights and biases . Furthermore , parameter copies are not necessary which saves memory writes , a common bottleneck for running on GPUs . CAML can also help distributed machine learning systems , where the same model is deployed to different machines and we wish to learn different contexts concurrently . Network communication is often the bottleneck , which is mitigated by only sharing the ( gradients of ) context parameters . We show empirically that CAML outperforms MAML on a regression and classification task and performs similarly on a reinforcement learning problem , while adapting significantly fewer parameters at test time . We observe that CAML is less sensitive to the inner-loop learning rate , and can be scaled up to larger networks without overfitting . We also demonstrate that the context parameters represent meaningful embeddings of tasks , confirming that the inner loop acts as a task identification step . 2 BACKGROUND : META-LEARNING FOR FAST ADAPTATION . We consider settings where the goal is to learn models that can quickly adapt to a new task with only little data . To this end , learning on the new task is preceded by meta-learning on a set of related tasks . Here we describe the meta-learning problem for supervised and reinforcement learning , as well as the method MAML . 2.1 PROBLEM SETTING . In few-shot learning problems , we are given distributions over training tasks ptrain ( T ) and test tasks ptest ( T ) . Training tasks can be used to learn how to adapt fast to any of the tasks with little per-task data , and evaluation is done on previously unseen test tasks . Unless stated otherwise , we assume that ptrain = ptest and refer to both as p. Tasks in p typically share some structure , so that transferring knowledge between tasks speeds learning . During each meta-training iteration , a batch of N tasks T = { Ti } Ni=1 is sampled from p. Supervised Learning . In a supervised learning setting , we learn a model f that maps data points x 2 X that have a true label y 2 Y to predictions ŷ 2 Y . A task Ti is defined as a tuple Ti = ( X , Y , L , q ) , where X is the input space , Y is the output space , L ( y , ŷ ) is a task-specific loss function , and q ( x , y ) is a distribution over labelled data points . We assume that all data points are drawn i.i.d . from q . Different tasks can be created by changing any element of Ti . Training in the supervised meta-learning setting proceeds over meta-training iterations , where for every task Ti 2 T from the current batch , we sample two datasets Dtraini ( for training ) and Dtesti from qTi : Dtrain i = { ( x , y ) i , m } M train i m=1 , Dtesti = { ( x , y ) i , m } M test i m=1 , ( 1 ) where ( x , y ) ⇠ qTi . M traini and M testi are the number of training and test datapoints , respectively . The training data is used to update f , and the test data is then used to evaluate how good this update was , and adjust f or the update rule accordingly . Reinforcement Learning . In a reinforcement learning ( RL ) setting , we aim to learn a policy ⇡ that maps states s 2 S to actions a 2 A . Each task corresponds to a Markov decision process ( MDP ) : a tuple Ti = ( S , A , r , q , q0 ) , where S is a set of states , A is a set of actions , r ( st , at , st+1 ) is a reward function , q ( st+1|st , at ) is a transition function , and q0 ( s0 ) is an initial state distribution . The goal is to maximise the expected cumulative reward J under ⇡ , J ( ⇡ ) = Eq0 , q , ⇡ '' H 1X t=0 t r ( st , at , st+1 ) # , ( 2 ) where H 2 N [ 1 is the horizon and 2 [ 0 , 1 ] is the discount factor . Again , different tasks can be created by changing any element of Ti . During each meta-training iteration , for every task Ti 2 T from the current batch , we first collect a trajectory ⌧ train i = { s0 , a0 , r0 , s1 , a1 , r1 , . . . , sM traini 1 , aM traini 1 , rM traini 1 , sM traini } , ( 3 ) where the initial state s0 is sampled from q0 , the actions are chosen by the current policy ⇡ , the state transitions according to q , and M train i is the number of environment interactions available , We unify several episodes in this formulation : if the horizon H is reached within the trajectory , the environment is reset using q0 . Once the trajectory is collected , this data is used to update the policy . Another trajectory ⌧ test i is then collected by rolling out the updated policy for M test i time steps . This test trajectory is used to evaluate the quality of the update on that task , and to adjust ⇡ or the update rule accordingly . Evaluation for both supervised and reinforcement learning problems is done on a new ( unseen ) set of tasks drawn from p ( or ptest if the test distribution of task is different ) . For each such task , the model is updated using L or J and only few datapoints ( Dtrain or ⌧ train ) . Performance of the updated model is reported on Dtest or ⌧ test . 2.2 MODEL-AGNOSTIC META-LEARNING . One method for few-shot learning is model-agnostic meta-learning ( Finn et al. , 2017a , MAML ) . Here , we describe the application of MAML to a supervised learning setting . MAML learns an initialisation for the parameters ✓ of a model f✓ such that , given a new task , a good model for that task can be learned with only a small number of gradient steps and data points . In the inner loop , MAML computes new task-specific parameters ✓i ( starting from ✓ ) via one1 gradient update , ✓i = ✓ ↵r✓ 1 M i train X ( x , y ) 2Dtraini LTi ( f✓ ( x ) , y ) . ( 4 ) For the meta-update in the outer loop , the original model parameters are then updated with respect to the performance after the inner-loop update , i.e. , ✓ ✓ r✓ 1 N X Ti2T 1 M i test X ( x , y ) 2Dtesti LTi ( f✓i ( x ) , y ) . ( 5 ) The result of training is a model initialisation ✓ that can be adapted with just a few gradient steps to any new task that we draw from p. Since the gradient is taken with respect to the parameters ✓ before the inner-loop update ( 4 ) , the outer-loop update ( 5 ) involves higher order derivatives in ✓ .

Summary: CAML is a gradient-based meta-learning method closely related to MAML. It divides model parameters into disjoint sets of task-specific parameters $\phi$ which are adapted to each task and task-independent parameters $\theta$ with are meta-learned across tasks. $\phi$ are then interpreted as an embedding and fed as input to the model (parameterized by $\theta$). Experiments demonstrate that this approach performs on par with MAML while adapting far fewer parameters. An additional benefit is that this approach is less sensitive to the adaptation learning rate and is easier to implement and faster to compute.