Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning

Multimodal meta-learning is a recent problem that extends conventional few-shot meta-learning by generalizing its setup to diverse multimodal task distributions . This setup makes a step towards mimicking how humans make use of a diverse set of prior skills to learn new skills . Previous work has achieved encouraging performance . In particular , in spite of the diversity of the multimodal tasks , previous work claims that a single meta-learner trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions . The improvement is attributed to knowledge transfer between different modes of task distributions . However , there is no deep investigation to verify and understand the knowledge transfer between multimodal tasks . Our work makes two contributions to multimodal meta-learning . First , we propose a method to quantify knowledge transfer between tasks of different modes at a micro-level . Our quantitative , task-level analysis is inspired by the recent transference idea from multi-task learning . Second , inspired by hard parameter sharing in multi-task learning and a new interpretation of related work , we propose a new multimodal meta-learner that outperforms existing work by considerable margins . While the major focus is on multimodal meta-learning , our work also attempts to shed light on task interaction in conventional meta-learning . The code for this project is available at https : //miladabd.github.io/KML . 1 Introduction . Multimodal meta-learning was recently proposed as an extension of conventional few-shot metalearning . In [ 1 ] , it is defined as a meta-learning problem that involves classification tasks from multiple different input and label domains . An example in their work is a 3-mode few-shot image classification which includes tasks to classify characters ( Omniglot ) and natural objects of different characteristics ( FC100 , mini-ImageNet ) . The multimodal extension of meta-learning is proposed with two objectives . First , it generalizes the conventional meta-learning setup for more diverse task distributions . Second , it makes a step towards mimicking humans ’ ability to acquire a new skill via prior knowledge of a set of diverse skills . For example , humans can quickly learn a novel snowboarding trick by exploiting not only fundamental snowboarding knowledge but also skiing and skateboarding experience [ 1 ] . Multimodal Model-Agnostic Meta-Learning ( MMAML ) [ 1 ] proposes a framework to better handle multimodal task distributions and achieves encouraging performance . As one of the most intriguing findings , MMAML claims that a single meta-learner trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions . This was observed in spite of the diversity of the multimodal tasks . In [ 1 ] , this observation is attributed to knowledge transfer across different modes of multimodal task distribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In our work , we delve into understanding knowledge transfer in multimodal meta-learning . While improved performance using a multimodal task distribution is reported in [ 1 ] , there is no deep investigation to verify and understand how tasks from different modes benefit from each other . Towards understanding knowledge transfer in multimodal meta-learning at a micro-level , we propose a new quantification method inspired by the idea of transference recently proposed in multi-task learning ( MTL ) [ 2 ] . Despite the large number of meta-learning algorithms proposed in the literature , we remark that little work has been done in analyzing meta-learning at the task sample level . In particular , to the best of our knowledge , there is no previous work on understanding knowledge transfer among tasks and quantitative analysis of task samples in the context of meta-learning . Because of the lack of such study , the interaction between task samples remains rather opaque in meta-learning . Interestingly , despite the notion of a task in meta-learning and MTL , there has not been much intersections between these two branches of research , perhaps due to several fundamental differences between meta-learning and MTL1 ( e.g , meta-learning optimizes the risk over a large number of future tasks sampled from an unknown distribution of tasks , while MTL optimizes the average risk over a finite number of known tasks ; this will be further discussed ) . We note that because of these fundamental differences , we propose adaptations to develop our method to quantify knowledge transfer for multimodal tasks in meta-learning . Another contribution of our work is a new multimodal meta-learner . Our idea is inspired by hard parameter sharing in MTL [ 4 ] . Furthermore , we discuss our own interpretation of the modulation mechanism in [ 1 ] . These lead us to propose a new method that achieves substantial improvement over the best results in multimodal meta-learning [ 1 ] . While our major focus in this work is on multimodal meta-learning , we have also performed experiments on conventional meta-learning for our proposed knowledge transfer quantification . Our work makes an attempt to shed light on task interaction in conventional meta-learning . Our main contributions are • Focusing on multimodal meta-learning , we propose a method to understand and quantify knowledge transfer across different modes at a micro-level . • We propose a new multimodal meta-learner that outperforms existing state-of-the-art methods by substantial margins . 2 Related Work . Few-Shot Learning . In a few-data regime , conventional learning methods mostly fail due to overfitting . Fine-tuning a pre-trained network [ 5 , 6 , 7 , 8 ] sometimes prevents overfitting but at the cost of computation [ 9 ] . Therefore , recent successful approaches tackle this problem by meta-learning [ 10 ] . These methods can be classified into several categories . In metric-based approaches , a similarity metric between support and query samples is learned by learning an embedding space , in which samples from similar classes are close and samples from different classes are further apart [ 11 , 12 , 13 , 14 , 15 , 16 ] . Optimization-based methods focus on learning an optimizer , including an LSTM meta-learner for replacing stochastic gradient descent optimizer [ 17 ] , a mechanism to update weights using an external memory [ 18 ] , or finding a good initialization point for model parameters for fast adaptation [ 19 , 20 , 21 , 22 ] . Augmentation-based methods learn a generator from the existing labeled data to further use it for data augmentation in novel classes [ 23 , 24 ] . Finally , weight-generation methods directly generate the classification weights for unseen classes [ 25 , 26 ] . Multi-Task Learning . MTL algorithms can generally be divided into hard or soft parameter sharing methods [ 4 ] . Hard parameter sharing is the most prevalent design to MTL where a subset of the hidden layers are shared among all tasks , while several task-specific layers are stacked on top of the shared base [ 27 ] . Hard parameter sharing reduces the risk of overfitting and enables parameter efficiency across tasks [ 3 , 28 ] . In soft parameter sharing , each task has a separate model . Then , either the parameters of the models are encouraged to be similar by regularizing a distance [ 29 , 30 ] or the knowledge of tasks are linearly combined to produce output for each task [ 31 ] . Other works try to improve the multi-task performance by addressing what to share [ 32 , 33 , 34 ] , which tasks to train together [ 35 , 36 ] , or inferring task-specifc model weights [ 37 , 30 , 38 ] . Recently , the transference is 1Here we follow the MTL definition in Zhang and Yang [ 3 ] instead of a loose definition of MTL : learning problems that involve more that one task . proposed to analyze the information transfer in a general MTL framework [ 2 ] . This metric is then used for selecting the tasks for updating shared parameters . 3 Preliminaries . In the N-way , K-shot image classification task , given N classes and K labeled examples per class , the final goal is to learn a model that can generalize well to unseen examples from the same classes . Each task T consists of a support set ST = { ( xi , yi ) } N×Ki=1 and a query set QT = { ( x̃i , ỹi ) } N×M i=1 . The model learns from the samples in the support set ST and is evaluated on the query set QT . As a solution , meta-learning assumes that tasks are sampled from a distribution of tasks p ( T ) , and are split into meta-training and meta-test sets . The meta-learner learns the prior knowledge about the underlying structure of tasks using the meta-training set , such that later , the meta-test tasks can benefit from this prior knowledge . While early meta-learning algorithms directly minimize the average training error of a set of training tasks , Vinyals et al . [ 12 ] propose a novel training strategy called episodic training . Episodic training utilizes sampled mini-batches called episodes during training , where each episode is designed to mimic the few-shot task and generated by subsampling of classes and samples from the training dataset . The use of episodes improves the generalization by making the training process more faithful to the test environment and has become a dominant approach for the few-shot learning [ 39 , 19 , 11 ] . Meta-learning algorithms usually suppose the task distribution p ( T ) is unimodal , meaning that all generated classification tasks belong to a single input-label domain ( e.g . classification tasks with different combination of digits ) . Then , a multimodal counterpart can be considered where it contains classification tasks from multiple different input-label domains ( e.g . few-shot digits classification and birds classification ) . In our work , multimodal meta-learning refers to the multimodality occurs in task distribution p ( T ) due to using tasks from multiple domains . This should not be confused with multimodality in data type [ 40 ] ( e.g. , combination of image and text ) . 4 Information Transfer Among Tasks . Transference in Episodic Training . The episodic training process can be viewed from a multitask learning point of view , where multiple tasks ( episodes ) collaborate to build a shared feature Algorithm 1 : Measuring Transference on a Target Task . Require : task distribution p ( T ) , learning rate α , current state of network parameters θt 1 Sample a batch of tasks ξ = { Ti } ∼ p ( T ) 2 Sample a target task Tj 3 Use support set of target task Sj for adaption 4 Use query set Qj to evaluate the adapted model and compute the loss LTj ( Qj ; θt , Sj ) 5 for all Ti ∈ ξ do 6 Use support set Si to adapt to that task 7 Use query set Qi to evaluate the adapted model and compute the loss LTi ( Qi ; θt , Si ) 8 Update model parameters with respect to task i using ( 1 ) 9 Compute the loss of target task using updated parameters LTj ( Qj ; θt+1i , Sj ) 10 Compute transference from task i to target task LRi→j using ( 2 ) 11 end representation that is broadly suitable for many tasks . Consequently , training episodes implicitly transfer information to each other by updating this shared feature representation with successive gradient updates . Then , the information transfer ( transference ) in episodic training can be viewed as the effect of gradients update from one episode ( or a group of episodes ) to the network parameters on the generalization performance on other episodes . Like MTL , in an episodic training scenario , some learning episodes can be constructive regarding a target task and some can be destructive [ 41 ] . Analysis of Transference . To gain insights on the interaction between different episodes during episodic training , we adapt the transference idea [ 2 ] from multi-task learning to episodic training scenario of meta-learning . Consider a meta-learner parameterized by θ . At time-step t of episodic training , a batch of tasks are sampled from task distribution p ( T ) . Then for task i denoted by Ti , the model uses its support set Si to adapt to it , and the quality of adaption is evaluated by the loss on query set Qi denoted by LTi . The quantity θt+1i is defined as updated model parameters after a SGD step with respect to task i : θt+1i = θ t − α∇θtLTi ( Qi ; θt , Si ) ( 1 ) We can use this quantity to calculate a loss which reflects the effect of task i on the performance of others . More specifically , in order to assess the effect of the gradient update of task i on a given target task j , we calculate the ratio between loss of task j before and after applying the gradient update on the shared parameters with respect to i : LRi→j = LTj ( Qj ; θt+1i , Sj ) LTj ( Qj ; θt , Sj ) ( 2 ) In episodic training , the loss ratio LRi→j can be considered as a measure of transference from meta-train task i to meta-test task j . If LRi→j has a value smaller than one , the update on the network parameters results in a lower loss on target task j than the original parameter values , meaning that task i has a constructive effect on the generalization of the model on task j . On the other hand , if LRi→j has a value greater than one , it indicates the destructive effect of task i on the target task . For i = j , the loss ratio denotes the self-transference , i.e . the effect of a task ’ s gradient update in its own loss which can be used as a baseline for transference . One major limitation of the transference algorithm proposed in [ 2 ] is that while the knowledge transfer is measured by generalization performance , they have only considered the LR improvement on the target training tasks . This can not necessarily guarantee the improvement in the generalization performance . We adrress this by sampling source tasks from meta-train dataset and target tasks from meta-test dataset . Then , our transference metric can be considered as a generalization metric in micro-level . So we expect the algorithms with better generalization to have a more positive knowledge transfer in terms of our transference metric , and vice versa . The overall procedure of calculating transference within an episodic training framework is summarized in Algorithm 1 . Experiments . To investigate the interaction between tasks during training in a multimodal task distribution , we have sampled 300 meta-train tasks from the mini-ImageNet dataset as source tasks . Then we have analyzed the transference from these tasks to a single FC100 meta-test task using algorithm 1 . For this experiment , we have used ProtoNet [ 11 ] as meta-learner , and the analysis is performed in the middle of training on the combination of mini-ImageNet and FC100 datasets . The details of the experimental setup can be found in the supplementary . The histogram of the transference is shown in Figure 1a which indicates both positive and negative knowledge transfer from miniImageNet tasks to the target task . An exemplar task for both positive and negative knowledge transfer is shown in figure 1 . The task including animal classes has positive knowledge transfer to target task while the task including non-animal classes has negative transfer . In figure 2 , the transference to a different meta-test FC100 target task from mini-ImageNet meta-train tasks is shown . While the target task includes classification from people and insect classes , two source tasks with animal classes are among the best and worst knowledge transferring source tasks . This can be attributed to the quality of samples in these tasks . When a task includes noisy data samples , it is much harder to solve meaning that the transference can also happen based on task hardness [ 42 ] . Figure 2 also indicates that in the cross mode knowledge transfer , the negative transference occurs at the beginning iterations and increasingly more positive transference occurs as training proceeds . Based on the experience from MTL literature , the negative knowledge transfer occurs when different tasks fight for the capacity [ 41 ] . In the next section we will propose a new modulation scheme to reduce negative transfer and improve generalization ( Figure 1c ) .

Summary: This work proposes a method to augment multi-modal meta learning by improving information transfer among tasks. In particular, a modulation matrix is generated for each layer and each task using the outer-product of two learned parameters. The meta learning algorithm then proceeds, but with the inner loop using the per-parameter modulation kernel to generate an output feature map. The proposed method improves positive information transfer among tasks as measured by transference and model performance when evaluated with two meta learning algorithms (MAML and ProtoNet) on several benchmark datasets.