Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning
Authors: Milad Abdollahzadeh
Touba Malekzadeh
Ngai-man Cheung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-Learning
Few-Shot Learning
Multi-Task Learning
Transference Analysis

Abstract: Multimodal meta-learning is a recent problem that extends conventional few-shot meta-learning by generalizing its setup to diverse multimodal task distributions. This setup makes a step towards mimicking how humans make use of a diverse set of prior skills to learn new skills. Previous work has achieved encouraging performance. In particular, in spite of the diversity of the multimodal tasks, previous work claims that a single meta-learner trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions. The improvement is attributed to knowledge transfer between different modes of task distributions. However, there is no deep investigation to verify and understand the knowledge transfer between multimodal tasks. Our work makes two contributions to multimodal meta-learning. First, we propose a method to quantify knowledge transfer between tasks of different modes at a micro-level. Our quantitative, task-level analysis is inspired by the recent transference idea from multi-task learning. Second, inspired by hard parameter sharing in multi-task learning and a new interpretation of related work, we propose a new multimodal meta-learner that outperforms existing work by considerable margins. While the major focus is on multimodal meta-learning, our work also attempts to shed light on task interaction in conventional meta-learning. The code for this project is available at https://miladabd.github.io/KML.

Revisit Multimodal Meta-Learning through the Lens of Multi-Tash Learning
Milad Abdollahzadeh, Touba Malekzadeh, Ngai-Man Cheung
Singapore University of Technology and Design (mil lmilad__bb6ollabzadee, souba_malekzadei agaiman_cheeugg_ eew.euudddddu........ edu.
Abstract
Multimodal meta-leaming is recent problem that extends conventional few-shot meta-learning by generalizing its setup to diverse multimodal task distributions. This setup makes step towards mimicking how humans make of diverse set of prior skills to learn new skills. Previous work has achieved encouraging per- formance, In particular, spite of the diversity of the multimodal tasks, previous work claims that single meta-learner trained on multimodal distribution can sometimes outperform multiple specialized meta-learners trained individual uni- modal distributions. The improvement is attributed to knowledge transfer between different modes of task distributions. However, there no deep investigation verify and understand the knowledge transfer between multimodal tasks. Our work makes two contributions to multimodal meta-learning. First, we propose method to quantify knowledge transfer between tasks of different modes micro-level. Our quantitative, task-level analysis is inspired by the recent transference idea from multi-task learning. Second, inspired by hard parameter sharing in multi-task learning and a new interpretation related work, we propose a new multimodal meta-learner that outperforms existing work by considerable margins. While the major focus is on multimodal meta-learning, our work also attempts to shed light task interaction in conventional meta-learning. The code for this project is available at https::///// / htt///miaaaall ladabd, github. 1o/KML.
Introduction
Multimodal meta-learning was recently proposed as an extension of conventional few-shot meta- learning. In [1, it is defined a meta-learnnng problem that involves classification tasks from multiple different input and label domains, An example their work is a 3-mode few-shot image classification which includes tasks to classify characters (Omniglot) and natural objects of different characteristics (FC100, mini-ImageNee with two objectives. First, it generalizes the conventional meta-learning setup for more diverse task distributions. Second, it makes a step towards mimicking humans' ability to acquire a new skill via prior knowledge set of diverse skills. For example, humans can quickly learn novel snowboarding trick by exploiting not only fundamental snowboarding knowledge but also skiing and skateboarding experience .
The multimodal extension of meta-learning is proposed
Multimodal Model-Agnostic Meta-Learming (MMAML) 1 proposes a framework to better handle multimodal task distributions and achieves encouraging performance. As of the most intriguing findings, MMAML claims that a single meta-leamer trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions. This was observed in spite of the diversity of the multimodal tasks. In (1) this observation is attributed knowledge transfer across different modes of multimodal task distribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In our work, we delve into understanding knowledge transfer in multimodal meta-learning. While improved performance using a multimodal task distribution is reported in M. there no deep investigation to verify and understand how tasks from different modes benefit from each other. Towards understanding knowledge transfer in multimodal meta-learning at micro-level, we propose new quantification method inspired by the idea of transference recently proposed in multi-task learning (MTL) [2]. Despite the large number of meta-learning algorithms proposed in the literature, we remark that little work has been done in analyzing meta-learning at the task sample level. In particular, to the best of our knowledge, there is no previous work on understanding knowledge transfer among tasks and quantitative analysis of task samples in the context of meta-learning Because of the lack of such study, the interaction between task samples remains rather opaque in meta-learning, Interestingly, despite the notion of task in meta-learning and MTL, there has not been much intersections between these two branches of research, perhaps due to several fundamental differences between meta-learning and MTI (e.g, meta-learning optimizes the risk over large number of future tasks sampled from an unknown distribution of tasks, while MTL optimizes the average risk over finite number of known tasks; this will be further discussed). We note that because of these fundamental differences, we propose adaptations to develop our method to quantify knowledge transfer for multimodal tasks in meta-learning.
Another contribution of our work is a new multimodal meta-learner. Our idea is inspired by hard parameter sharing in MTL [4]] Furthermore, we discuss our interpretation of the modulation mechanism in [.. These lead us to propose a new method that achieves substantial improvement over the best results in multimodal meta-learning (.. While our major focus in this work is multimodal meta-learning, we have also performed experiments on conventional meta-learning for our proposed knowledge transfer quantification. Our work makes attempt to shed light on task interaction in conventional meta-learning.
Our main contributions are
Focusing on multimodal meta-learning, we propose a method to understand and quantify knowledge transfer across different modes micro-level.
We propose new multimodal meta-learner that outperforms existing state-of-the art meth- ods by substantial margins.
2 Related Work
Few-Shot Learning. In a few-data regime, conventional learning methods mostly fail due to over- fitting. Fine-tuning a pre-trained network 6 7 g sometimes prevents overfitting but at the cost of computation
[9. Therefore, recent successful approaches tackle this problem by meta-learning [10 These methods can be classified into several categories. In metric-based approaches, sim- ilarity metric between support and query samples is learned by learning an embedding space, in which samples from similar classes are close and samples from different classes are further apart [5, [6]. Optimization- based methods focus on learning an optimizer, including an LSTM meta- meta-learner for replacing stochastic gradient descent optimizer [17]. mechanism to update weights using an external memory [88] or finding a good initialization point for model
12 [3 14
parameters for fast adaptation 199 20 21 22] Arrgmentation-basse methods learn a generator from
the existing labeled data to further use it for data augmentation in novel classes 23 24 Finally,
weight- gengrtttnnnration methods directly generate the classification weights for unseen classes
25 26
Multi-Task Learning. MTL algorithms can generally be divided into hard or soft parameter sharing methods (4. Hard parameter sharing is the most prevalent design to MTL where a subset of the hidden layers are shared among all tasks, while several task-specific layers are stacked on top of the shared base (27). Hard parameter sharing reduces the risk of overfitting and enables parameter efficiency across tasks [3.28] In soft parameter sharing, each task has a separate model. Then, either the parameters of the models are encouraged to be similar by regularizing distance [29] 30 the knowledge of tasks are linearly combined to produce output for each task
33. Other works try to which tasks to train Recently, the transference is
improve the multi-task performance by addressing what share
32
33
together B5 36], or inferring task-specifc model weights B7
30
38
Herre we follow the MTL definition in Zhang and Yang 3 instead of a loose detinition of MTL: learning problems that involve more that one task.

---

Targel Task (FC100)
Source Task (mini- Lmaill-mmaeeee
Source Task (mini-TmageNNet
Transfer Type
Supportt
Supporr Sez
Sugpoor See
Query
Query
Query
GIV
IE%
59%
Lefl ProteNel
Self-Transfermee
Positive Knowledge
Negative Knowledge Transfer
Negative Transte
Nextral
Right MProlNNi-KML (ours)
Middle) M MrPootNet
Transfer
Transforance Histogram
Transference Histogram
Transferonce Histogram
vete jent
Numbzz of TURKS
0.
0.9
1G
1.
Loss Ratio
1.
15
Rabio
(b)
Figure I: Information transfer (transference) from 300 meta-train train mini-Imagell tasks to meta-test
FC100 task, Transference Histogram for: (a) ProtoNet, (b) MProtoNet (with FiLM modulation), (c) MProtoNet with proposed KMI. method. For both positive knowledge transfer (T.R < 1), and negative knowledge transfer (LR > 1) an exemplar task shown. Proposed method increases the positive transfer from average of 27% to 41%. Here, we simply use the LR threshold to classify the transference a task as positive or negative.
proposed analyze the information transfer in general MTL framework used for selecting the tasks for updating shared parameters.
2
This metric is then
3 Preliminaries
In the N-way, K-shot image classification task, given N classes and K labeled examples per class, the final goal is to learn model that can generalize well to unseen examples from the same classes. Each task T consists of a support set Sy = {(z,, 9t) N ] The model learns from the samples in the support set Sy and is evaluated on the query set Q As solution, meta-learning assumes that tasks are sampled from a distribution of tasks p(T), and are split into meta-training and meta-test sets. The meta-learner learns the prior knowledge about the underlying structure of tasks using the meta-training set, such that later, the meta-test tasks can benefit from this prior knowledge. While early meta-learning algorithms directly minimize the average training error of a set training tasks, Vinyals et al. [12) propose a novel training strategy called episodic training. Episodic training utilizes sampled mini-batches called episodes during training, where each episode is designed to mimic the few-shot task and generated by subsampling of classes and samples from the training dataset. The use of episodes improves the generalization by making the training process more faithful to the test environment and has become a dominant approach for the few-shot learning 339 19 III Meta-lcarning algorithms usually suppose the task distribution p(T) unimodal, meaning that all generated classification tasks belong to a single input-label domain (e.g. classification tasks with different combination of digits). Then, multimodal counterpart can be considered where it contains classification tasks from multiple different input-label domains (e.g. few-shot digits classification and birds classification). In our work, multimodal meta-learning refers to the multimodality occurs in task distribution p(7) due to using tasks from multiple domains. This should not be confused with multimodality in data type 40 (e.g., combination of image and text).
and a query set Qf = ( 9t) N )
4 Information Transfer Among Tasks
Transference in Episodic Training. The episodic training process can be viewed from a multi- task learning point of view, where multiple tasks (episodes) collaborate to build a shared feature
3

---

Algorithm 1: Measuring Transference on a Target Task.
Require:
task distribution p(7), learning rate a, current state of network parameters 0
Sample batch of tasks {T:) ~ PT)
2 Sample a target task 7
Use support set of target task S, for adaption
Use query set 2 to evaluate the adapted model and compute the loss LT₃ (T,(0
5 for all Ti E ç do
Use support set S; to adapt to that task
Use query set 2; evaluate the adapted model and compute the loss C (2;; 6* S.)
Update model parameters with respect to task using {
Compute the loss of target task using updated parameters LT, (2:: 0f
10
Compute transference from task i to target task LR using
11 end
representation that is broadly suitable for many tasks. Consequently, training episodes implicitly transfer information to each other by updating this shared feature representation with successive gradient updates. Then, the information transfer (transference) in episodic training can be viewed as the effect of gradients update from one episode (or a group of episodes) to the network parameters on the generalization performance on other episodes. Like MTL, in an episodic training scenario, some learning episodes can be constructive regarding a target task and some can be destructive
41
Analysis of Transference. To gain insights on the interaction between different episodes during episodic training, we adapt the transference idea 2 from multi-task learning episodic training scenario of meta-learning. Consider a meta-learner parameterized by 0. At time-step L. of episodic training, batch of tasks are sampled from task distribution p(T). Then for task i denoted by T, the model uses its support set S, to adapt to it, and the quality of adaption is evaluated by the loss on query set Qi denoted by LT,- The quantity ge+1 defined as updated model parameters after a SGD step with respect to task i:
We can use this quantity to calculate a loss which reflects the effect of task on the performance of others. More specifically, in order to assess the effect of the gradient update of task on a given target task caleulate the ratio between loss of task j before and after applying the gradient update on the shared parameters with respect to i:
LR = (Q, SS))
(Q,;
In episodic training, the loss ratio LR:4; can be considered as a measure of transference from meta-train task to meta-test task j. If LR:-> has a value smaller than one, the update on the network parameters results in a lower loss on target task j than the original parameter values, meaning that task i has a constructive effect on the generalization of the model on task j. On the other hand, if LR has value greater than one, it indicates the destructive effect of task on the target task. For i the loss ratio denotes the self-transserence, i.e. the effect of a task's gradient update in its own loss which can be used as a baseline for transference. One major limitation of the transference algorithm proposed in 2 is that while the knowledge transfer is measured by generalization performance, they have only considered the LR improvement the target training tasks. This can not necessarily guarantee the improvement in the generalization performance. We adrress this by sampling source tasks from meta-train dataset and target tasks from meta-test dataset. Then, our transference metric can be considered as a generalization metric in micro-level. So we expect the algorithms with better generalization to have more positive knowledge transfer in terms of our transference metric, and vice versa. The overall procedure of calculating transference within an episodic training framework is summarized in Algorithm
Experiments. To investigate the interaction between tasks during training in a multimodal task distribution, we have sampled 300 meta-train tasks from the mini- ImageNe dataset source tasks.

---

Transference Histogram (5K Training (terations)
Targel Task E C108
Source Imagehee
Source Tack (miel-lnageeel Suagurt Set
QUARPT
Sell-T Tral-Treeeeence
Poblice Kaandedge Trandfi
Negathe Krureledge
13
14
60
Transference Histogram (10K Training iterations)
Transference Histogram (15K Training Iterations)
Transference Histogram (20K Training Iterations)
Marbbo
Rablo
Loca Rutio
Loss (LR1
Figure 2: Transference from 300 mini- mini-ImageNet meta-train tasks to a FC100 meta-test task. In the beginning of the training, when network learns low-level features, both tasks transfer negative knowledge. As the training proceeds, one becomes more and more positive, and then, consistently transfers positive knowledge to target task.
Then we have analyzed the transference from these tasks single FC100 meta-test task using algorithm|] [ For this experiment, we have used ProtoNet [1]] as meta-learner, and the analysis is performed in the middle of training on the combination of mini-ImageNet and FC100 datasets. The details of the experimental setup can be found in the supplementary. The histogram of the transference shown in Figure T which indicates both positive and negative knowledge transfer from mini- ImageNet tasks to the target task. An exemplar task for both positive and negative knowledge transfer shown in figure I The task including animal classes has positive knowledge transfer to target task while the task including non-animal classes has negative transfer.
In figure 2 the transference to a different meta-test FC100 target task from tasks shown. While the target task includes classification from people and insect classes, two source tasks with animal classes are among the best and worst knowledge transferring source tasks. This can be attributed to the quality of samples in these tasks. When a task includes noisy data samples, it is much harder to solve meaning that the transference can also happen based on task hardness (42) Figure also indicates that in the cross mode knowledge transfer. the negative transference occurs at the beginning iterations and increasingly more positive transference occurs as training proceeds. Based on the experience from MTL literature, the negative knowledge transfer occurs when different tasks fight for the capacity [4T] In the next section we will propose new modulation scheme to reduce negative transfer and improve generalization (Figure Te).
meta-train
[
Proposed Multimodal Meta-Learner
In the previous section, the transference analysis shows that different tasks in episodic training can have a positive or negative impact on the learning process of other tasks. Recent works have analyzed the compatibility of tasks in multi-task learning (36]] 2 during training to select some grouping of tasks for co-training that can improve performance. In our episodic training scenario, idcally, we would like select the grouping of learning episodes that are compatible with the meta-test task. However, direct application of the methods in MTL to episodic training is not possible, due to two main reasons. First, the novel tasks in meta-testing are unseen and unknown during meta-training Second, even if the tasks in meta-testing were known, it can be very computationally expensive to determine the group of cooperative tasks, and then assign a task-specific layer for each task, because episodic training involves tens of thousands of training tasks. Here, we propose a new interpretation of the modulation mechanism in MMAMI.. These lead us to propose masudo-task-specffe. (task-aware) layers inspired by hard parameter sharing in MTL.
5

---

Input Task
Base Network
Support Set
Query Set
CONV
Rop
Input
Output Renturemap
Modulated festuremap
Fneader
Modulation Parameters
Modulation Network
Embeshinngs
Porameter
Generator
[M,
Output
CONV
CONV
Ieput
Modialated Kernel
Octput
Modulated Network f
Kernel
lesturemap
(8)
(c)
Figure 3: (a) General multimodal meta-Learning framework; Modulation scheme in (b) MMAML,
(c) Proposed KML method.
5.1 Multimodal Model-Agnostic Meta-Learning
To tackle multimodal few-shot tasks, MMAML [ proposes multimodal meta-lcarning framework which consists of modulation network and a base network fo (FigureBh) The modulation network, predicts the mode of a task and generates a set of taec-ipec modulation parameters to help the base network to better fit to the identified mode. It includes a task encoder network and modulation parameter generator. Task encoder takes the support samples a task T as input and produces an embedding vector UT (ST) encode its characteristics. Then, task-specific modulation parameters are generated using the embedding vector of task by the parameter generator network WT (1))) Since MMAMI. uses feature-wise linear modulation (FiLM) 433. the generated modulation parameters are split into scaling and shifting parameters wT = (nT 7r Let Y, denote the 1th output channel of layer in the base network fa. Then, the corresponding modulated output channel in MMAML is calculated as:
Y, = Y, +Y++. +
where ?/ E '7 and 7 E YT are scalar values. For task encoder, MMAML uses a 4-layer convo- lutional network which is followed by multiple MLPs parameter generator. MMAML uses the model-agnostic meta-learning (MAML) algorithm [19] as meta-learner in base network.
Limitations of Feature-Wise Modulation. We believe the major limitation of MMAML results from using FiLM modulation scheme. For each convolutional layer of CNN, the 1h output channel (feature map) is computed by convolving the 1 kernel of that layer W; (the layer index is removed for the sake of simplicity) with input the layer X and adding the bias term b,:
W. x
where denotes the convolution operator. Considering this, the modulation in 3) can be rewritten as:
Y, (n. X + (nibs + +%)
(5)
We define the modulated kernel, and the modulated bias W: = 7) W. and b, n:b; + %. Note that the modulated kernel is calculated by scaling the whole elements of the original kernel with single scalar number. Then considering (5) the modulation in MMAML can be interpreted as convolving the input with this modulated kernel and adding the modulated bias term. The validity of this new interpretation is verified with experimental results (please refer to supplementary material). Based on this new interpretation, the modulated kernels used for different tasks are heavily bound

---

Algorithm Kernel Modulation Algorithm
Require: task distribution p(7), learning rate a
Randomly initialize 0, s.
while not done do
Sample batch of tasks T, p(T)
for all T. do
Infer UT, (S,)
g.
[ Φ)
Generate M and A (1) using and 9
Task Eimeedding
Modulation
Modulate using 6 and to obtain ST. Compute LT, Q1: OT. S)) using Sōr,
end
94
(b ()
V ET. TTppET (Qi 0T, Update + 4 a ET>p(T) LT, (4; TT,, St) 11
Update
Offict
Update o a TTmpTT CT. Q 8y,
Figure 4: Simplified design for modulation parameter generator.
end
together by W and are just scaled versions of it. Since the base kernel shared across all tasks and scaling gives not many degrees of freedom for each task to produce its desired modulated kernel, different tasks fight for the model capacity, and the negative transfer between tasks from different distributions lead to training compromised base kernel.
5.2 Kernel Modulation
We propose modulate the base network by changing the whole parameters within each layer using the modulation parameters conditioned on the task. This gives more capacity for each task to generate its desired kernel. To do this, for every single parameter within the network, a modulation parameter is generated. For task T, and for each layer / 1..... I. of the network, let M' (VTT, o) denote the modulation matrix generated for modulating the base kernel W' of that layer. Among different design choices, intuitively, we consider the modulated kernels as perturbations around the base kernel, by learning to generate the residual weights, more powerful base kernel can be learned via enabling proper knowledge transfer across different modes the task distribution. In our case, this can be achieved by adding the unit matrix J (all-ones matrix) to the modulation matrix and then perform Hadamard multiplication between the result and the base kernel as follows:
W
w' (J (J+M((UT,))
For the bias of each layer we simply add an offset term Ab' (DT,
b/y bb'+Ab'(uur)
The modulated parameters for task T are therefore a collection of modulated kernels and biases, The modulation parameters for each task are generated through processing task embeddings of that task UT by parameter generator network go (Figure 3). Similar to M. assuming Be to be MLP, saves lots of computations where separate parameter generation network s used for each layer. The main concern with using an MLP is the large number of parameters required in g to generate the modulation parameters Ab' and M Because to produce the modulation parameters for the whole parameters of deep neural network, need to employ some sort of parameter reduction in g& to prevent an explosion of parameters. Instead of following current complicated algorithms such pruning the weights of network 44 redesigning the operations [45), we propose simple design for ge which substitutes an MLP module with three smaller ones (Figure 4.. In this design, the modulation matrix is generated by processing task embeddings Uy using two modules, and then performing the outer product operation on the output of these two modules:
i.e. 07 - [WW,,.. W bt.... 64).
M'(UT, Φ) = g'o (UT) (VT)
(8)
7

---

The produced matrix is then reshaped to match the kernel shape in that layer. The bias offset of each layer is generated by processing the embedding vector oy using the third module:
Ab (UT,, Ø) = Bigg (UT)
Our experiments show that compared to a single MLP, the proposed simplified structure decreases the number of parameters in ga by factor of 150, and also provides better generalization performance. A detailed discussion can be found in supplementary. A similar approach to our simplified structure for parameter generator is proposed in 46. However in [6], it is used to filter out noisy gradients rather than generating task-aware parameter for few-shot learning done in our work. The overall multimodal meta-learning algorithm using the proposed Kernel ModuLation (KML) scheme is summarized in Algorithm 2 Note that considering the new interpretation presented in (5) our KML can be thought of as a generalization of FiLM. So we can expect that applying KML to the areas improved by FiLM may bring some further improvements. In addition to the few-shot classification results provided in section [6] we have also provided some results on visual reasoning in the supplementary which shows better performance of proposed KML over FiLM.
6 Experiments and Analysis
We evaluate the proposed model in both multimodal and unimodal few-shot classification scenarios. Following [.] to create a multimodal few-shot image classification meta-dataset, we combine
multiple widely used datasets (Omniglot [47),
[12]. FC100 48]. CUB 40 and
Aircraft (50). Most of our experiments were performed by modifying the code accompanying [. and for fair comparison, we follow their experimental protocol unless specified. The details of all datasets, experimental setups, and network structure can be found in the supplementary material. Note that when using an aptimization-basee meta-learner (fa) like MAML, our KML idea can easily be extended to Reinforcement Learning (RL) environments. More details and experimental results on RL can be found in supplementary.
Multimodal Few-shot Classification Results. For multimodal few-shot classification, similar to (1. we train and evaluate models on the meta-datasets with two modes (Omniglot and mini-Imagenn three modes (Omniglot, mini- Ima--emaeene,, and FC100), and five modes (all the five datasets). This meta-dataset includes few-shot classification of characters, natural objects with different statistics and also fine-grained classification of different bird and aircraft types. Due to the large discrepancies between datasets, the possibility of negative transfer increases which makes few-shot classification more challenging. We also combine the mini-ImageNet and FC100 datasets to construct another two-mode meta-dataser. Both mini-ImageNNt and FC100 include samples of natural objects and few-shot tasks generated from them probably have a similar underlying structure. However, the few-shot classification of FC-100 100 tasks a bit challenging due the smaller image size. So, here we aim to investigate the possibility of knowledge transfer from mini-Imagek
to FC100. We use two different meta-learners namely ProtoNet [I], and MAML [1), and several baselines considering each meta-learner. The first baseline is training and testing multiple separate meta-learnerss one for each dataset. The second baseline is considered when meta- leta-learner has no access to task mode information and simply trained and tested in the combination of tasks from different modes. The third baseline is the MMAML [I] which the best of our knowledge is the only proposed algorithm for multimodal few-shot learning. A variant of this baseline (MProtoNet) is also used here by replacing the ProtoNet meta-learner with MAML in the general framework of 1.
Multimodal few-shot image classification results are shown in Table T. Comparing MProtoNet with MMAML, we can see that replacing ProtoNet with MAML considerably improves the classification accuracy. We believe that most of the extra accuracy gained in the multimodal framework by replacing MAML with ProtoNet results from the proper training of modulation network due to improved gradient flow from meta-learner. A detailed experiment can be found in supplementary. Additionally, as the results in Table I suggest, the proposed KML scheme considerably improves the performance of multimodal few-shot classification with both meta-learners. The improvement in 5-shot scenarios is higher because more clues about the task provide more accurate embedding vectors to generate the modulation parameters. We have also included the detailed accuracy for 2 scenario consisting of mini-ImageNet and FC100 in table 2 This shows on average a positive knowledge transfer from mini- mini--mageNet to FC100 dataset in terms of increased classification
8

---

Table 1: Meta-test accuracies the multimodal few-shot image classification with 2, 3, and 5 modes. The accuracy values are mean of 1000 randomly generated test tasks, and the + shows 95% confidence interval over tasks. 2Mode* and 2Mode indicate the combination of Omniglot, respectively. Results produced by code provided in ... Our Implementation.
with FC100 and
Method
Setup
MAML
[19
Multi-MAML
MMAML I
MMAML+KML (ours)
1-shot
40.53
39.27+0.76%%
39.11+0.622%
40.73+00.66% %
2Mode
5-shot
54.11=0.63%
53.51
52.02:40.63%
53.72-10.60%
1-shot
65.18=0A %
66.77:10.68
67.67:10.63%
68.01:10.599
2Mode
5-shot
00.5%%
73.07+00.61 %
73.52++0.71%%
77.02+00.66 %
1-shot
54.40=0.56%
56.01
57.35::0.61%
57.68:10.59
3 Mode
5-shot 66.51=0.54%
65.92:1:0.255
64.21110.57%
67.12:10.55%
5Mode
1-shot
47.19-0.49%
48.33++0.58%
49.53++0.5055
50.31-+00.995 %
5-shot
58.13=0.489%
59.20=
58.89:10.47%
60.51+:0.47%
ProtoNet [11"
Multi-ProtoNet
MProtoNet (1)
MProtoNet-KMI (ours)
1-shot
43.05=0.58%
43.42:1:0.664
43.57:10.59%
44.40-10.65%
2Mode
5-shot
57.70=0.59%
56.73:t0.6490
56.03-t0.64%
59.31=:0.62%
1-shot
69.55=0.54%%
70.17++0.61 %
70.60++0.56%
73.69:+0,52%
2Mode
5-shot
15.12-0.4199 1%
75.33+00.46%
75.72+00.47%
79.82++0.40% %
1-shot
$8.14=0.499
59.89:+0.50%%
39.62:10.54%
62.08:10.54%%
3 Mode
5-shot 66.84=0.449% 44%
67.03:10.444
67.51
70.03-10.43% %
1-shot
49.31=0.53%
50.69
51.75::0.52%
56.72-10.46% %
5Mode
5-shot
58.91=0.51%
59.88++0.54%
59.95:::.422%
64.91-+0.389
Table 2: Meta-test accuracies including the performance on each dataset.
Datasets
Method
mini-Ima
FC100
Overall
shot
5shot
Ishot
5shot
shot
5xhot
MAMLI
41.70=0.915%
60.03110.88%
39.27:10.86%
53.51110.25%
MAML2
36.84-10811 S.
47,08=0.74%
MAML 19
44.11:10.88%
59.34:10.8%%
37.0110.82%
49.01:10.73%
40.53:10.68%
54.11:10.63%
MMAML.
42.70=10.866
56.93:10.78%
35.50
0.711:
39.11:10.72%
52.00-100.69%
(AML-KML(Ouus)
1..01110.82%
58.95:10.7%%
37.67=0.86%
48.45:20.76%
40.73:10.66%
53.72:10.60%
ProtoNet1
47.19+0.588%
61.11++0.53%
ProtoNet2
39.61=0.54%%
52.35:0.57%%
43 43.42:+0.56%%
56.73-10.64%
ProtoNet I1
45.62-+0.88 822%
61.74:+0.55%
40.36-t-0.6%%
53.66=0.77%%
43.05=+0.58%
57.70-+0.559%
MProtoNet 1
47.06: 0.80
60.56: 0.77
40.09: 0.77
51.501 0.83
43.57= 0.59
56.03:t 0.64
MProto-KMLLours))
48.34 0.86 %
61.20:: 0.77%
40.48:: 0.77% %
54.42 0.76%
44.40:1 0.65%
59.31: 0.62%
accuracy compared to a single meta-learnrr trained and tested on FC100. The improved accuracy of proposed KML over previous modulation comes at the cost of parameter and computational overhead. A detailed discussion can be found in the supplementary. In addition to multimodal scenario, KML also brings considerable improvement the conventional unimodal scenario (c.g., mini-ImageNNtt.. Please supplementary for more details and results.
Transference Results. In addition to classification results, here we analyse the performance of the proposed KML method on improving knowledge transfer. To this end, we apply the proposed transference metric to analyse the knowledge transfer from 300 randomly sampled mini-ImageNer meta-train tasks to another random meta-test FC100 task. We use Algorithm
] to calculate the
transference during training and the transference histogram plotted in figure for both ProtoNet and proposed MProtoNet+KMML In each sub-figure, we have included the results for both methods

---

Transference Histogram (5K Training Iterations)
Transference Histogram (10K Training Iterations)
Prothiiet
P0.67%, 13%, Ma.0 60%.
Proster: P13.338, 1ar16.3%
N8L.75% Na:21.7%
Number of Tacks
of
13
Loss Ratio (UR)
Rabio
(a)
(b)
Transference Histogram (15K Training Iterations)
Transference Histogram (20K Training Iterations)
25
Pressler P3.67% k:te c0%, kar..155
25-
Drstolitt
9.10 320% N:72 00l, Nr.3.7%
Mirotalte CXL 2:33 67%, N.70 57% Nu..67%
MProtakat DIL 7...0%. N.G 02%. Nu:26.05%
Number of Tacks
25
(c)
(d)
Figure Transference Histogram from mini-ImageNer to FC100 task.
for ease of comparison. A vertical line is also drawn to specify the threshold between positive and negative transference. The percentage the Positive (P), Negative (N) and Neutral (Nu) transference also shown for each method. For classifying source task neutral, the threshold that the loss change is negligible and corresponds to less than 0.05% change in the meta-test accuracy. Note that in average, proposed KMI. also performs better in micro-level in terms of increasing positive knowledge transfer and reducing the negative transfer.
interval around
The cross mode knowledge transfer can be investigated between different datasets. An interesting behavior that we observed during our analysis that even Omniglot few-shot tasks -which are considered easy tasks and be handled very well with current meta-learning algorithms- have positive knowledge transfer to challenging FC100 tasks specially later training iterations. We believe in this Omniglot tasks act more like regularizer they emphasize simple yet strong features and prevent the model from overfitting to FC100 meta- train task (please find the analysis results in supplementary). Note that knowledge transfer also be analysed within tasks of single dataset as in some datasets like Image- classes vary so much in few-shot setup (please see supplementary for more details).
Discussion and Conclusion
In this work, we propose a new quantification method for understanding knowledge transfer in multimodal meta-learning We then propose a new interpretation of the modulation mechanism in MMAML. These lead us to propose a new multi-modal meta-learner inspired by hard parameter sharing in multi-task learning. Our extensive experimental results show that the proposed method achieves substantial improvement over the best results in multimodal meta-learning. While our major focus is on multimodal meta-learning, our work also attempts to shed light on conventional unimodal meta-learning. Limitations discussed in the supplementary.
10

---

Acknowledgement
This project was supported by SUTD project PIE-SGP-AI-2018-00 This research was also supported by the National Research Foundation Singapore under its AT Singapore Programme [Award Number: 100-1000021100000 The authors are grateful for the fruitful discussion with Yuval Elovici, and Alexander Binder.
References
[1] Risto Vuorio, Shao- Sun, Hexiang Hu. and Joseph J. Lim. Multimodal model-agnostie meta-learning via task-aware modulation. Neural Information Processing Systems, 2019.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Measuring and harnessing transference in multi-task learning. arXiv preprint S.iv:20100.541.. 2020.
[3] Yu Zhang and Qiang Yang. survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
[4] Sebastian Ruder. An overview of multi-task learning deep neural networks. arXiv preprint OrXiv:1706.05000 2017.
[5] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Can features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806-813, 2014.
[6] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 1409 1556, 2014.
[7] Touba Malekzadeh, Milad Abdollahzadeh, Hossein Nejati, and Ngai-Man Cheung. Aireraft fuselage defect detection using deep neural networks. arXiv preprint arXiv:1712.09211. 2017.
[8] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang. Xiaodan Liang, and I. Liang Lin. Meta r-cnn: Towards general solver for instance-level low-shot learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
[9] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin. Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-datased A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
[10] Sebastian Thrun and Lorien Pratt. Learning to lear. Springer Science & Business Media, 2012.
[11] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[12] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[13] Flood Sung. Yongxin Yang. Li Zhang. Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1199-120, 2018
[14] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. International Conference on Learning Representations, 2019.
[15] Seong Min Kye. Hae Beom Lee, Hoirin Kim, and Sung Ju Hwang. Meta-learned confidence for few-shot learning. arXiv e-prints, pages arXiv-2002, 2020.
[16] Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot- meta training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 331 339, 2019.
[17] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017.
[18] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pages 3554-2563 PMLR. 2017.

---

[19] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR. 2017.
[20] Andrei A Rusu, Dushyant Rao, Jakuh Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
[21] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: Stt.:1777.098... 2017.
[22] Sungyong Baik, Seokil Hong, and Kyoung Mu Lee. Learning to forget for meta-learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[23] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. 2017.
[24] Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pages 3018-3033 302700272222 2017.
[25] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Patten Recognition, pages 7229-7238, 2018.
[26] Yiluan Guo and Ngai-Man Cheung. Attentive weights generation for few shot learning via information maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13499-13508, 2020.
[27] Richard Caruana. Multitask learning: A knowledge-basce source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pages 41-48, Morgan Kaufmann, 1993.
[28] Jonathan Baxter. model of inductive bias leamning. Journal of artificial intelligence research, 12:149-11 198. 2000.
[29] Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross-lingual parameter sharing in neural network parser. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pages 845 850, 2015.
[30] Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint 2016.
[31] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi- task lcarning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994-4003, 2016.
[32] Pengsheng Guo, Chen-Yo Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pages 3854-3863. PMLR, 2020.
[33] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. arXiv preprint ar11::1911.12223, 2019.
[34] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere. and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint (04::9994402900 2019.
[35] Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv preprint 172.v:/702.08..00 2017.
[36] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 9132- PMLR, 2020.
[37] Yongxin Yang and Timothy Hospedales. Deep multi-task representation learning: A tensor factorisation approach. arXiv preprint arXiv:1605.0639) 2016.
[38] Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 185 1851-1860, 2019.
[39] Jiaxin Chen, Xiao-Ming Wu, Yanke Li, Qimai Li, Li-Ming Zhan, and Fu-lai Chung, A closer look at the training strategy for modern meta-learning. Advances in Neural Information Processing Systems, 33, 2020.
12

---

[40] Tadas Baltrušaitis. Chaitanya Ahuja, and Louis-Philippe Morency, Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41(2):423 443, 2018.
[41] Shengehao Liu, Yingyu Liang, and Anthony Gitter. Loss-halanced task weighting to reduce negative transfer in multi-tast task learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 9977-9978, 2019.
[42] Anh Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks, Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1395 1405, 2019.
[43] Ethan Perez, Florian Strub, Harm Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reason- ing with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[44] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint 0tt::2003003033. 2020.
[45] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenety2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[46] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. On modulating the gradient for meta-learning. In European Conference Computer Vision, pages 556-572. Springer, 2020.
[47] Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[48] Boris N Oreshkin, Pau Rodriguez, and Alexandre Lacoste. Tadam: task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 719-721777 129-999,22799 2018.
[49] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. 2011.
[50] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- mea-ggained visual classification of aircraft, arXiv preprint arXiv:1306.515// 2013.
13


Reviews:
[{"review_id": "jOc9hSWvPnk", "reviewer": "Reviewer_KUKC", "summary": "This work proposes a method to augment multi-modal meta learning by improving information transfer among tasks. In particular, a modulation matrix is generated for each layer and each task using the outer-product of two learned parameters. The meta learning algorithm then proceeds, but with the inner loop using the per-parameter modulation kernel to generate an output feature map. The proposed method improves positive information transfer among tasks as measured by transference and model performance when evaluated with two meta learning algorithms (MAML and ProtoNet) on several benchmark datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and easy to follow. It builds on prior multi-modal meta learning work (MMAML). In particular, this work identifies a weakness in the design (that the modulated feature map effectively scales the original kernel by a scalar value), and suggests a per-parameter scaling kernel which appears to significantly improve empirical performance and information transfer among tasks. \n\nThe transference analysis lends explanation and intuition into why this method works. Moreover, adapting transference from the multitask domain to the meta learning domain is valuable and could be used in future algorithmic improvments to better understand the effect of proposed meta learning augmentations on positive/neutral/negative information transfer.\n\nGiven the robust meta learning transference analysis and strength of the empirical results, I believe this submission is technically sound. That said, I was unable to find any analysis related to the increased memory overhead or decrease in computational efficiency inherent in using this method. While this is perhaps a less important dimension of empirical analysis, this work would be strengthened if it was included. \n\nFinally, multi-modal meta learning is an important domain with significant potential. Leveraging disparate modalities to improve model performance is/will be critical for many real world machine learning applications and research which develops this area is valuable.\n\nFor these reasons, I recommend this paper as an accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is not mentioned, and limitations, notably increased latency/memory requirements by using -KML, should be analyzed to further strengthen the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "V2SpMkzI0Iv", "reviewer": "Reviewer_uwYd", "summary": "This paper draws a novel connection between multi-task learning (MTL) and few-shot classification as an instantiation of meta-learning. Plus, the paper suggests kernel modulation (KML), a drop-in replacement of FiLM, which can improve the performance of MMAML.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper draws a novel connection between multi-task learning (MTL) and few-shot classification as an instantiation of meta-learning. Borrowing the term \"multimodal\" from their primary reference [^1], the authors treat modes as tasks. Using the concept of \"transference\" of MTL[^2], the authors also showed the usefulness of their proposed KML (Kernel Modulation) scheme, which increases the frequency of positive transfer.\n\nQuestion 1. I see task embedding can improve the transference, as shown in figure 1. However, it is unclear that this change is originated from the use of KML, the other contribution point of the paper, or the use of task embedding. To verify that KML improves transference, I believe there should be a histogram for MprotoNet+FiLM for figure 1.\n\nComment 1. Although [^1] used the term \"modality,\" I think the \"domain\" is the correct term for this paper's setting. Since FC100 and mini-ImageNet have natural image \"modality\" but have different input-label \"domain.\"\n\nKML is proposed to overcome the limitation of FiLM modulation, which MMAML[^1] used. KML directly alters the weight and bias of the convolution layer rather than alters its computed outputs. Since kernel weight requires more modulation parameters, the author used a 1-rank approximation (outer products). The authors report that this 1-rank approximation achieves higher meta-validation accuracy than the full-rank version in Figure 7 in the appendix.\n\nQuestion 2. I am curious about the results from low-rank approximations other than 1-rank, for example, 2-rank or 3-rank.  \nQuestion 3. At figure 7, the authors stopped plotting after training iterations of 200K. Since the slope of full-rank MLP is higher than that of 1-rank, I guess full-rank meta-validation accuracy surpasses that of 1-rank after 200K. I believe a longer span of training iterations than 200K is needed to claim that 1-rank yields better results in terms of accuracy (line 210-211 in the appendix).\n\nComment 2. This KML scheme can be used as a drop-in replacement of FiLM. I believe a comparison with FiLM on the CLEVR dataset, which FiLM is originally evaluated on, can further validate the pros of KML.\n\nTypos\n- line 121, the subscript of theta\n- line 138, met-learner\n- line figure 1 caption, wrong capitalization \"Proposed\"\n\n### Remarks\n(Clarity) Overall, the paper is well written and easy to understand, though I want some parts to be fixed. (Typos)  \n\n[+] (Originality) Novel interpretation of multi-domain few-shot classification through the lens of transference.  \n[+] (Significance) Generalizable improvements on FiLM scheme, although it needs to be verified. (Comment 2)  \n[-] (Quality) I think the MTL analogy of the few-shot classification author made (transference) does not support the advantage of KML well. (Question 1)  \n[-] (Quality) The analysis of KML is insufficient. (Question 2 and 3)  \n[-] (Quality) The paper omits the RL experiment [^1] did. Since [^1] is its primary reference, the authors should conduct the RL experiment for the fairer comparison.  \n\n[^1] Vuorio, Risto, et al. \"Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation.\" _ArXiv:1910.13616 [Cs, Stat]_, Oct. 2019. _arXiv.org_, [http://arxiv.org/abs/1910.13616](http://arxiv.org/abs/1910.13616).  \n[^2] Fifty, Christopher, et al. \"Measuring and Harnessing Transference in Multi-Task Learning.\" _ArXiv:2010.15413 [Cs]_, Feb. 2021. _arXiv.org_, [http://arxiv.org/abs/2010.15413](http://arxiv.org/abs/2010.15413).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors fairly well addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Tt87reMIYPO", "reviewer": "Reviewer_79Fb", "summary": "This paper introduces a novel few-shot multitask learning approach that uses an additional kernel modulation module that computes the residual adjustment matrix for a given pair of tasks. Authors propose an approach for analyzing the knowledge transfer of particular tasks on the accuracy of the other tasks based on transference.  Authors demonstrate efficiency of the proposed method in 5-shot and 1-shot setups on Omniglot, mini-ImageNet, FC100 and other datasets and compare their result with the MAML and ProtoNet baselines. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well-written and easy to understand, except the introduction, which needs to be more to the point. \nAuthors introduced a new way of analyzing how training the model with specific subtasks (called episodes in the paper) affects its performance on other subtasks at any stage of training based on the losses before and after training, which is a reasonable idea that can be useful to the multitask learning community. My only concern is that in Figure 2, it is shown that the amplitude of transference decreases with the number of epochs, which might be caused by the learning rate decay. It is overall not surprising that the variance in predictions - and subsequently the amplitude of transference - decreases with time. More experiments are needed to exclude all these factors from the analysis.\n\n\nThe overall idea of inferring the task-specific model weights is not new [a. “TRACE NORM REGULARIZED DEEP MULTI-TASK LEARNING”, Yang et al., ICLR 2017; b. “DEEP MULTI-TASK REPRESENTATION LEARNING: A TENSOR FACTORISATION APPROACH”m Yang et al.,  ICLR 2017; c. “Attentive Single-Tasking of Multiple Tasks”, Maninis et al., CVPR 2019, and others], and these should be added to the related work section. However, authors are the first to efficiently apply this idea to few-shot multitask learning without a significant overhead in the number of parameters, which is a valuable contribution to the community.  \n\n\nThe experimental evaluation as well as the transference results clearly indicate that incorporation of the proposed kernel modulation increases the overall accuracy on the new tasks in all setups, both for the MAML and ProtoNet baselines. \n\n\nThe tradeoffs and limitations of the proposed approach are not properly discussed. Is it possible that the additional modules g_\\phi result in overfitting to the existing tasks? What is the trade-off in the number of parameters and training time and the accuracy of the proposed method? These should be analyzed as well. \n\n\nMinor: \na) Is there a practical reason for using separate embedding modules g_\\phi_1 and g_\\phi_2 instead of a single module?\nb) L255-257: Probably the authors meant “replacing MAML with ProtoNet”, not the other way around.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not properly discussed (see the last paragraph of the main review).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Sh1efH3DeQ5", "reviewer": "Reviewer_ScBy", "summary": "- The paper analyses the important aspect of multimodal meta-learning in the context of few-shot classification. In this setting, the task distribution contains tasks sampled from multiple datasets with different input domains and labels,  introducing an extra layer of complexity.  It has been shown that sharing a single model allows for knowledge transfer across modes improving performance over unimodal training. However, it requires rethinking the training procedure since gradient updates from very different tasks could interfere with each other.\n- In this regard, the authors propose a metric of \"transference\" to measure the positive or negative transfer across meta-learning tasks, based on a recently proposed approach for multi-task learning.\n- They also propose a meta-learning model based on feature/kernel modulation.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- One of the main issues that I found in this paper is that I'm not able to understand what is (are?) the real contribution(s). Most of the analysis is presented in the supplementary material and it's quite hard to follow the author's reasoning by reading just the main paper. This is something the authors should fix, but in my opinion, it would require substantial rewriting and restructuring the overall paper. My impression is that it lacks a proper structure that helps the reader understand the (huge) amount of work done by the authors. It might be the classic situation where a senior researcher with much more experience in writing could definitely improve the paper and make it ready for publication. Unfortunately, I don't think it is the case at the moment.\n- To give an example, the transference measure and its analysis during training are nice, but the paper presents it as a means to develop a new algorithm for multimodal meta-learning. This is clearly not the case. In the next iteration, I would give more space to the analysis (supp B), while much less to the meta-learning model which is less important/marginal in my opinion. At the moment the two parts are simply stitched together, but there should be a more natural presentation.\n- Another main point is that there's no really a comparison with other methods besides the MMAML paper (baseline). It's ok, but I think the authors may want to test their method with recent approaches. It would be nice to see results on the entire [meta-dataset](https://github.com/google-research/meta-dataset#1-triantafillou-et-al-2020) too. For other methods using conditional models, please see the [leaderboard](https://github.com/google-research/meta-dataset#leaderboard-in-progress). Besides the accuracy results, it would be nice to analyze the LR of other approaches. That would make the paper much more complete.\n\nOther comments:\n- **Lines 46-51.** I would rephrase: it should be clearer what is the main difference between Meta-Learning and MTL. The first optimizes the risk over future tasks sampled from a distribution of tasks. Note that this can be done because we are imposing a constraint on the types of tasks. Instead, MTL optimizes the average risk over a finite number of tasks. \n- **Equations 4-5:** Following these equations I think the modulation is applied to the pre-activations. Is that right? In fact, 4-5 are true only if everything is linear. I was sure that modulations were generally applied after the nonlinearity, but checking [MMAML official code](https://github.com/shaohua0116/MMAML-Classification/blob/bdf1a93e798ab81619563038b95a3c5aa18717e0/maml/models/gated_conv_net.py#L16) it doesn't seem so. How do you use modulation in your model? Please clarify this point.\n    - Anyway, I don't understand the point here. You are claiming that FILM (on the preactivation) is the same as scaling separately the i-th kernels (the convolutional filters) and biases. Again, no real surprise here if everything is linear. It shouldn't be verified empirically unless there is a nonlinearity somewhere and you want to show that the statement holds also in this case. What am I looking at here?\n    - If I understood correctly your line of reasoning is that FILM modulation (scale and shift of the entire i-th feature, which corresponds to scale and shift of the i-th kernel) is not enough for multimodal meta-learning and you propose a method that scale and shift each element of the kernel (MKL), rather than using a single parameter (+ bias).\n- Line 255: \"Our experiments show that the proposed method decreases the number of parameters in g φ by a factor of 150...\" wrt what? What are you comparing here? If you are comparing the naive way of producing MKL modulators with a single FC please be more precise. I don't see any real comparison here in the main paper. Is it all in supp D?. The authors should quantify the number of parameters of the models in the main paper or be more clear about it. Btw, the trick they propose to modulate each parameter/feature independently without exploding in the number of parameters is quite clever.\n- Supp A (lines 35-37): **Hard parameter sharing**: I don't think this sentence is clear. What kind of parameters are you referring to? From my understanding, you consider hard sharing among the initialization of the weights that would be then adapted. This means that the initialization is not modulated before adaptation, hence they are shared. Is it correct? Please clarify because it might be confused with the ANIL approach of having shared parameters across tasks, namely, just there is not adaptation (no inner loop) and the parameters are just updated in the outer loop.\n- Supp B (lines 96-98) **on generalization**: this is simply speculation unless you provide references/results supporting this. \n\n\n\n————————————-\n\nAfter the rebuttal and the discussion with authors and reviewers, I decided to increase my rate from 4 to 7.\nThe paper addresses an important open problem in meta-learning and provides a nice analysis from a multi-task perspective with empirical results. Hence, I recommend acceptance. \nIf accepted, the final version should include all the modifications agreed by the authors, and these should be verified by the reviewers and AC. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are expressed in the supplementary material. No negative societal impacts to declare in my opinion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5h + 4h (after rebuttal) "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission tackles the problem of meta-learning on a multimodal task distribution. It introduces an analytical methodology inspired from recent work on transference in multi-task learning and proposes a new multimodal meta-learning approach called Kernel Modulation (KML) which is claimed to outperform competing approaches.

Reviewers found the problem tackled to be important to the research community and the transference analysis to be an interesting and valuable contribution borrowed from the multi-task learning literature. They expressed reservations about the way in which the contributions were framed (in that the transference analysis feels disconnected from the KML contribution) and how many important details are relegated to the supplementary material. However, given that the authors were very responsive and open to incorporating their feedback, the reviewers and I feel positive about acceptance, if the changes promised are incorporated in the final version (see summary here: https://openreview.net/forum?id=V5prUHOrOP4&noteId=l5jMUHLSraH).
Decision: Accept (Poster)