Model-Agnostic Meta-Learning for Multimodal Task Distributions

1 INTRODUCTION . Humans are capable of effectively utilizing prior knowledge to acquire new skills as well as adapt to new environments . For example , imagine that we are learning how to carve on a snowboard on a mountain that we just arrived at . While we only know the basics of snowboarding and barely know the terrain , we can still accomplish this feat quickly . Even when the skill that we are interested in learning is only vaguely related to the set of skills that we have acquired in the past , we are usually able to exploit the relationship among skills and generalize . For the snowboarding example , our knowledge about skiing and skateboarding can play the role that prepares us for learning snowboarding skills more efficiently . Similarly , can machines rapidly master a novel skill based on a variety of related skills they have already acquired ? Recent advances in meta-learning offer machines a way to rapidly adapt to a new task using few samples by first learning an internal representation that matches similar tasks . Such representations can be learned by considering a distribution over similar tasks as the training data distribution . Model-based meta-learning approaches ( Duan et al. , 2016 ; Wang et al. , 2016 ; Munkhdalai & Yu , 2017 ; Mishra et al. , 2018 ) propose to recognize the task identity from a few sample data , use the task identity to adjust a model ’ s state ( e.g . RNN ’ s internal states ) and make the appropriate predictions with the adjusted model . Those methods demonstrate good performance at the expense of having to hand-design architectures , yet the optimal strategy of designing a meta-learner for arbitrary tasks may not be obvious to humans . On the other hand , model-agnostic gradient-based meta-learning frameworks ( Finn et al. , 2017 ; 2018 ; Kim et al. , 2018 ; Lee & Choi , 2018 ; Grant et al. , 2018 ) seek an initialization of model parameters such that a small number of gradient updates will lead to high performance on a new task , while having flexibility in the choice of models . While most of the existing gradient-based meta-learners rely on a single initialization , different tasks from the task distribution can require substantially different parameters , making finding an initialization that is a short distance away from all of them infeasible . If the task distribution is multimodal with disjoint and far apart modes , one can imagine that a set of separate meta-learners with each covering one mode could better master the full distribution . However , associating each task with one of the meta-learners not only requires additional identity information about the task , which is not always available or could be ambiguous when the modes are not clearly disjoint , but also eliminates the possibility of transferring knowledge across different modes of the task distribution . To overcome this issue , we aim to develop a meta-learner that is able to acquire a meta-learned prior and adapt quickly given tasks sampled from a multimodal task distribution . To this end , we leverage the strengths of the two main lines of existing meta-learning techniques : model-based and gradient-based meta-learning . Specifically , we propose to augment gradient-based meta-learners with the capability of generalizing across a multimodal task distribution . Instead of learning a single initialization point in the parameter space , we propose to first compute the task identity of a sampled task by examining task related samples . Given the estimated task identity , our model then performs a step of model-based adaptation to condition the meta-learned initialization on the sampled task in a step we call modulation . Then , from this modulated meta-prior , a few steps of gradient-based adaptation are performed towards the target task to progressively improve the performance on the task . The main contributions of this paper are as follows : • We identify and empirically demonstrate a limitation in a family of widely used gradientbased meta-learners , including Finn et al . ( 2017 ; 2018 ) ; Kim et al . ( 2018 ) ; Lee & Choi ( 2018 ) ; Grant et al . ( 2018 ) ; Nichol & Schulman ( 2018 ) . • We propose a model that is able to acquire a set of meta-learned prior parameters and adapt quickly given tasks sampled from a multimodal task distribution by taking advantage of both model-based and gradient-based meta-learning . • We design a set of multimodal meta-learning problems and demonstrate that our model offers a better generalization ability on tasks including regression , classification , and reinforcement learning . 2 RELATED WORK . We review the several families of meta-learning , which our work builds upon : Optimization-based meta-learning . The approaches that learn to optimize a learner model are known as optimization-based meta-learning . Pioneered by Schmidhuber ; Bengio et al . ( 1992 ) , optimization algorithms with learned parameters have been proposed , enabling the automatic exploitation of the structure of learning problems . From a reinforcement learning perspective , Li & Malik ( 2016 ) represent an optimization algorithm as a learning policy . Andrychowicz et al . ( 2016 ) train LSTM optimizers to learn update rules from the history of gradients , and Ravi & Larochelle ( 2016 ) train a meta-learner LSTM to update a learner ’ s parameters for few-shot image classification . Unlike these methods , our algorithm does not require additional parameters to learn update rules ; instead , we augment our meta-learner with the ability to compute the identity of tasks . Model-based meta-learning . Model-based meta-learning frameworks learn to recognize the identities of tasks and adjust the model state ( e.g . the internal state of an RNN ) to fit the task . Santoro et al . ( 2016 ) train a network with an external memory that is able to assimilate new samples and leverage this data to make accurate predictions . Duan et al . ( 2016 ) represent a fast RL algorithm as an RNN and learn it from data . Munkhdalai & Yu ( 2017 ) propose to learn task-agnostic knowledge and use it to shift its fast parameters for rapid generalization . Though our model has a model-based meta-learning component , the main adaptation mechanism is gradient-based , which has preferable behavior on tasks outside the task distribution ( Finn & Levine , 2018 ) . Gradient-based meta-learning . Finn et al . ( 2017 ) and its extensions ( Finn et al. , 2018 ; Kim et al. , 2018 ; Lee & Choi , 2018 ; Grant et al. , 2018 ) , known as gradient-based meta-learning , aim to estimate a parameter initialization among the task-specific models , that provides a favorable inductive bias for fast adaptation . With the model agnostic nature , appealing results have been shown on a variety of learning problems . However , assuming tasks are sampled from a concentrated distribution and pursuing a common initialization to all tasks can substantially limit the performance of such methods on multimodal task distributions where the center in the task space becomes ambiguous . In this paper , we propose to first identify the mode of a sampled task , in a procedure which is similar to model-based meta-learning approaches Santoro et al . ( 2016 ) ; Mishra et al . ( 2018 ) . Then , we modulate the meta-learned prior in the parameter space to make the model better fit to the mode and take gradient steps to rapidly improve the performance on the task afterwards . 3 PRELIMINARIES . We aim to rapidly adapt to a novel task sampled from a multimodal task distribution . We consider a target dataset D consisting of tasks sampled from a multimodal distribution . The dataset is split into meta-training and meta-testing sets , which are further divided into task-specific training DtrainT and validationDvalT sets . A meta-learner learns about the underlying structure of the task distribution through training on the meta-training set and is evaluated on meta-testing set . Our work builds upon Model-Agnostic Meta-Learning ( MAML ) algorithm ( Finn et al. , 2017 ) . MAML seeks an initialization of parameters θ of a meta-learner such that after a small number of gradient steps to minimize the task-specific loss on the training set of a new task DtrainT , the adapted parameters produce good results on the validation set of that task DvalT . The initialization of the parameters is trained by sampling minibatches of tasks from D , computing the adapted parameters for all DtrainT in the batch , evaluating the validation losses on the corresponding DvalT using the adapted parameters and finally backpropagating the sum of the validation losses through the adaptation step to the initial parameters θ . 4 METHOD We aim to develop a Multi-Modal Model-Agnostic Meta-Learner ( MUMOMAML ) that is able to quickly master a novel task sampled from a multimodal task distribution . To this end , we propose to leverage the ability of model-based metalearners to identify tasks sampled from a task distribution as well as the ability of gradient-based metalearners to consistently improve the performance with a few gradient steps . Specifically , we propose to learn a model-based meta-learner that produces a set of task-specific parameters to modulate the meta-learned prior parameters . Then , this modulated prior learns to adapt to a target task rapidly through gradient-based optimization . An illustration of our model is shown in Figure 1 . In this section , we first introduce our model-based meta-learner and a variety of modulation operators in section 4.1 . Then we describe the training details for MUMOMAML in section 4.2 and present the implementation details in section 4.3 . 4.1 MODEL-BASED META-LEARNER . Given K data points and labels { xk , yk } k=1 , ... , K , the task embedding network f produces an embedding vector υ that encodes the characteristics of a task according to υ = f ( { xk , yk } k=1 , ... , K ; ωf ) . The modulation network g modulates the metalearned prior of the gradient-based meta-learner in the parameter space based on the task embedding vector υ . To enable specialization of each block of the gradient-based meta-learner to the task , we apply the modulation block-wise to activate or deactivate the units of a block ( i.e . a channel of a convolutional layer or a neuron of a fully-connected layer ) . Specifically , modulation network produces the modulation vectors for each block i by τ1 , ... , τN = g ( υ ; ωg ) . We formalize the procedure of applying modulation as : φi = θi τi , where φi represents the modulated prior parameters for the gradient-based meta-learner , and represents a general modulation function . In the experiments , we investigate some representative modulation operations including attention-based modulation ( Mnih et al. , 2014 ; Vaswani et al. , 2017 ) and feature-wise linear modulation ( FiLM ) ( Perez et al. , 2017 ) . Attention based modulation has been widely used in modern deep learning models and has proved its effectiveness across various tasks ( Yang et al. , 2016 ; Mnih et al. , 2014 ; Zhang et al. , 2018 ; Xu et al. , 2015 ) . Inspired by the previous works , we employed attention to modulate the prior model . In concrete terms , attention over the outputs of all neurons ( Softmax ) or a binary gating value ( Sigmoid ) on each neuron ’ s output is computed by the model-based meta-learner . These modulation vectors τ are then used to scale the pre-activation of each neural network layer Fθ , such that Fφ = Fθ ⊗ τ . Note that here ⊗ represents a channel-wise multiplication . Feature-wise linear modulation ( FiLM ) has been proposed to modulate neural networks for achieving the conditioning effects of data from different modalities . We adopt FiLM as an option for modulating our gradient-based meta-learner . Specifically , the modulation vectors τ are divided into two components τγ and τβ such that for a certain layer of the neural network with its pre-activation Fθ , we would have Fφ = Fθ ⊗ τγ + τβ . It can be viewed as a more generic form of attention mechanism . Please refer to Perez et al . ( 2017 ) for the complete details . Interestingly , in a recent few-shot learning paper FiLM modulation is used in a metric learning model and high performance on few-shot learning tasks is attained ( Oreshkin et al. , 2018 ) .

Summary: This paper presents an interesting meta-learning algorithm that can learn from multimodal task distributions, by combining model-based and gradient-based meta-learning. It first represents a task with a latent feature vector produced by a recurrent network, and then modulates the meta-learned prior with this task-specific latent feature vector before applying gradient-based adaptation. Experimental results are shown to validate the proposed algorithm. While the idea appears to be quite novel for meta-learning, further efforts are needed to improve this work.