Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: null
review_content: Dear authors,

Thank you for all your answers and your active participation in the discussion.

In order to help the reviewers and the AC to take the final decision, I suggest you summarize in a single comment a detailed list of all changes and clarifications that you intend to incorporate into the paper. 
This will help us track the differences between the initial and final versions of the paper and verify that these changes are actually happening in case of acceptance. Also, I think this would be particularly helpful for the AC and a good way to evaluate the impact of the review process for this specific paper.

Thanks a lot!
review_title: Request to the authors: summary of the updates to appear in the final paper

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel few-shot multitask learning approach that uses an additional kernel modulation module that computes the residual adjustment matrix for a given pair of tasks. Authors propose an approach for analyzing the knowledge transfer of particular tasks on the accuracy of the other tasks based on transference.  Authors demonstrate efficiency of the proposed method in 5-shot and 1-shot setups on Omniglot, mini-ImageNet, FC100 and other datasets and compare their result with the MAML and ProtoNet baselines. 

main_review: The paper is overall well-written and easy to understand, except the introduction, which needs to be more to the point. 
Authors introduced a new way of analyzing how training the model with specific subtasks (called episodes in the paper) affects its performance on other subtasks at any stage of training based on the losses before and after training, which is a reasonable idea that can be useful to the multitask learning community. My only concern is that in Figure 2, it is shown that the amplitude of transference decreases with the number of epochs, which might be caused by the learning rate decay. It is overall not surprising that the variance in predictions - and subsequently the amplitude of transference - decreases with time. More experiments are needed to exclude all these factors from the analysis.


The overall idea of inferring the task-specific model weights is not new [a. “TRACE NORM REGULARIZED DEEP MULTI-TASK LEARNING”, Yang et al., ICLR 2017; b. “DEEP MULTI-TASK REPRESENTATION LEARNING: A TENSOR FACTORISATION APPROACH”m Yang et al.,  ICLR 2017; c. “Attentive Single-Tasking of Multiple Tasks”, Maninis et al., CVPR 2019, and others], and these should be added to the related work section. However, authors are the first to efficiently apply this idea to few-shot multitask learning without a significant overhead in the number of parameters, which is a valuable contribution to the community.  


The experimental evaluation as well as the transference results clearly indicate that incorporation of the proposed kernel modulation increases the overall accuracy on the new tasks in all setups, both for the MAML and ProtoNet baselines. 


The tradeoffs and limitations of the proposed approach are not properly discussed. Is it possible that the additional modules g_\phi result in overfitting to the existing tasks? What is the trade-off in the number of parameters and training time and the accuracy of the proposed method? These should be analyzed as well. 


Minor: 
a) Is there a practical reason for using separate embedding modules g_\phi_1 and g_\phi_2 instead of a single module?
b) L255-257: Probably the authors meant “replacing MAML with ProtoNet”, not the other way around.

limitations_and_societal_impact: The limitations are not properly discussed (see the last paragraph of the main review).
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposes a method to augment multi-modal meta learning by improving information transfer among tasks. In particular, a modulation matrix is generated for each layer and each task using the outer-product of two learned parameters. The meta learning algorithm then proceeds, but with the inner loop using the per-parameter modulation kernel to generate an output feature map. The proposed method improves positive information transfer among tasks as measured by transference and model performance when evaluated with two meta learning algorithms (MAML and ProtoNet) on several benchmark datasets.
main_review: This paper is well-written and easy to follow. It builds on prior multi-modal meta learning work (MMAML). In particular, this work identifies a weakness in the design (that the modulated feature map effectively scales the original kernel by a scalar value), and suggests a per-parameter scaling kernel which appears to significantly improve empirical performance and information transfer among tasks. 

The transference analysis lends explanation and intuition into why this method works. Moreover, adapting transference from the multitask domain to the meta learning domain is valuable and could be used in future algorithmic improvments to better understand the effect of proposed meta learning augmentations on positive/neutral/negative information transfer.

Given the robust meta learning transference analysis and strength of the empirical results, I believe this submission is technically sound. That said, I was unable to find any analysis related to the increased memory overhead or decrease in computational efficiency inherent in using this method. While this is perhaps a less important dimension of empirical analysis, this work would be strengthened if it was included. 

Finally, multi-modal meta learning is an important domain with significant potential. Leveraging disparate modalities to improve model performance is/will be critical for many real world machine learning applications and research which develops this area is valuable.

For these reasons, I recommend this paper as an accept.
limitations_and_societal_impact: Societal impact is not mentioned, and limitations, notably increased latency/memory requirements by using -KML, should be analyzed to further strengthen the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - The paper analyses the important aspect of multimodal meta-learning in the context of few-shot classification. In this setting, the task distribution contains tasks sampled from multiple datasets with different input domains and labels,  introducing an extra layer of complexity.  It has been shown that sharing a single model allows for knowledge transfer across modes improving performance over unimodal training. However, it requires rethinking the training procedure since gradient updates from very different tasks could interfere with each other.
- In this regard, the authors propose a metric of "transference" to measure the positive or negative transfer across meta-learning tasks, based on a recently proposed approach for multi-task learning.
- They also propose a meta-learning model based on feature/kernel modulation.
main_review: - One of the main issues that I found in this paper is that I'm not able to understand what is (are?) the real contribution(s). Most of the analysis is presented in the supplementary material and it's quite hard to follow the author's reasoning by reading just the main paper. This is something the authors should fix, but in my opinion, it would require substantial rewriting and restructuring the overall paper. My impression is that it lacks a proper structure that helps the reader understand the (huge) amount of work done by the authors. It might be the classic situation where a senior researcher with much more experience in writing could definitely improve the paper and make it ready for publication. Unfortunately, I don't think it is the case at the moment.
- To give an example, the transference measure and its analysis during training are nice, but the paper presents it as a means to develop a new algorithm for multimodal meta-learning. This is clearly not the case. In the next iteration, I would give more space to the analysis (supp B), while much less to the meta-learning model which is less important/marginal in my opinion. At the moment the two parts are simply stitched together, but there should be a more natural presentation.
- Another main point is that there's no really a comparison with other methods besides the MMAML paper (baseline). It's ok, but I think the authors may want to test their method with recent approaches. It would be nice to see results on the entire [meta-dataset](https://github.com/google-research/meta-dataset#1-triantafillou-et-al-2020) too. For other methods using conditional models, please see the [leaderboard](https://github.com/google-research/meta-dataset#leaderboard-in-progress). Besides the accuracy results, it would be nice to analyze the LR of other approaches. That would make the paper much more complete.

Other comments:
- **Lines 46-51.** I would rephrase: it should be clearer what is the main difference between Meta-Learning and MTL. The first optimizes the risk over future tasks sampled from a distribution of tasks. Note that this can be done because we are imposing a constraint on the types of tasks. Instead, MTL optimizes the average risk over a finite number of tasks. 
- **Equations 4-5:** Following these equations I think the modulation is applied to the pre-activations. Is that right? In fact, 4-5 are true only if everything is linear. I was sure that modulations were generally applied after the nonlinearity, but checking [MMAML official code](https://github.com/shaohua0116/MMAML-Classification/blob/bdf1a93e798ab81619563038b95a3c5aa18717e0/maml/models/gated_conv_net.py#L16) it doesn't seem so. How do you use modulation in your model? Please clarify this point.
    - Anyway, I don't understand the point here. You are claiming that FILM (on the preactivation) is the same as scaling separately the i-th kernels (the convolutional filters) and biases. Again, no real surprise here if everything is linear. It shouldn't be verified empirically unless there is a nonlinearity somewhere and you want to show that the statement holds also in this case. What am I looking at here?
    - If I understood correctly your line of reasoning is that FILM modulation (scale and shift of the entire i-th feature, which corresponds to scale and shift of the i-th kernel) is not enough for multimodal meta-learning and you propose a method that scale and shift each element of the kernel (MKL), rather than using a single parameter (+ bias).
- Line 255: "Our experiments show that the proposed method decreases the number of parameters in g φ by a factor of 150..." wrt what? What are you comparing here? If you are comparing the naive way of producing MKL modulators with a single FC please be more precise. I don't see any real comparison here in the main paper. Is it all in supp D?. The authors should quantify the number of parameters of the models in the main paper or be more clear about it. Btw, the trick they propose to modulate each parameter/feature independently without exploding in the number of parameters is quite clever.
- Supp A (lines 35-37): **Hard parameter sharing**: I don't think this sentence is clear. What kind of parameters are you referring to? From my understanding, you consider hard sharing among the initialization of the weights that would be then adapted. This means that the initialization is not modulated before adaptation, hence they are shared. Is it correct? Please clarify because it might be confused with the ANIL approach of having shared parameters across tasks, namely, just there is not adaptation (no inner loop) and the parameters are just updated in the outer loop.
- Supp B (lines 96-98) **on generalization**: this is simply speculation unless you provide references/results supporting this. 



————————————-

After the rebuttal and the discussion with authors and reviewers, I decided to increase my rate from 4 to 7.
The paper addresses an important open problem in meta-learning and provides a nice analysis from a multi-task perspective with empirical results. Hence, I recommend acceptance. 
If accepted, the final version should include all the modifications agreed by the authors, and these should be verified by the reviewers and AC. 
limitations_and_societal_impact: Limitations are expressed in the supplementary material. No negative societal impacts to declare in my opinion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper draws a novel connection between multi-task learning (MTL) and few-shot classification as an instantiation of meta-learning. Plus, the paper suggests kernel modulation (KML), a drop-in replacement of FiLM, which can improve the performance of MMAML.
main_review: This paper draws a novel connection between multi-task learning (MTL) and few-shot classification as an instantiation of meta-learning. Borrowing the term "multimodal" from their primary reference [^1], the authors treat modes as tasks. Using the concept of "transference" of MTL[^2], the authors also showed the usefulness of their proposed KML (Kernel Modulation) scheme, which increases the frequency of positive transfer.

Question 1. I see task embedding can improve the transference, as shown in figure 1. However, it is unclear that this change is originated from the use of KML, the other contribution point of the paper, or the use of task embedding. To verify that KML improves transference, I believe there should be a histogram for MprotoNet+FiLM for figure 1.

Comment 1. Although [^1] used the term "modality," I think the "domain" is the correct term for this paper's setting. Since FC100 and mini-ImageNet have natural image "modality" but have different input-label "domain."

KML is proposed to overcome the limitation of FiLM modulation, which MMAML[^1] used. KML directly alters the weight and bias of the convolution layer rather than alters its computed outputs. Since kernel weight requires more modulation parameters, the author used a 1-rank approximation (outer products). The authors report that this 1-rank approximation achieves higher meta-validation accuracy than the full-rank version in Figure 7 in the appendix.

Question 2. I am curious about the results from low-rank approximations other than 1-rank, for example, 2-rank or 3-rank.  
Question 3. At figure 7, the authors stopped plotting after training iterations of 200K. Since the slope of full-rank MLP is higher than that of 1-rank, I guess full-rank meta-validation accuracy surpasses that of 1-rank after 200K. I believe a longer span of training iterations than 200K is needed to claim that 1-rank yields better results in terms of accuracy (line 210-211 in the appendix).

Comment 2. This KML scheme can be used as a drop-in replacement of FiLM. I believe a comparison with FiLM on the CLEVR dataset, which FiLM is originally evaluated on, can further validate the pros of KML.

Typos
- line 121, the subscript of theta
- line 138, met-learner
- line figure 1 caption, wrong capitalization "Proposed"

### Remarks
(Clarity) Overall, the paper is well written and easy to understand, though I want some parts to be fixed. (Typos)  

[+] (Originality) Novel interpretation of multi-domain few-shot classification through the lens of transference.  
[+] (Significance) Generalizable improvements on FiLM scheme, although it needs to be verified. (Comment 2)  
[-] (Quality) I think the MTL analogy of the few-shot classification author made (transference) does not support the advantage of KML well. (Question 1)  
[-] (Quality) The analysis of KML is insufficient. (Question 2 and 3)  
[-] (Quality) The paper omits the RL experiment [^1] did. Since [^1] is its primary reference, the authors should conduct the RL experiment for the fairer comparison.  

[^1] Vuorio, Risto, et al. "Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation." _ArXiv:1910.13616 [Cs, Stat]_, Oct. 2019. _arXiv.org_, [http://arxiv.org/abs/1910.13616](http://arxiv.org/abs/1910.13616).  
[^2] Fifty, Christopher, et al. "Measuring and Harnessing Transference in Multi-Task Learning." _ArXiv:2010.15413 [Cs]_, Feb. 2021. _arXiv.org_, [http://arxiv.org/abs/2010.15413](http://arxiv.org/abs/2010.15413).
limitations_and_societal_impact: I think the authors fairly well addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The submission tackles the problem of meta-learning on a multimodal task distribution. It introduces an analytical methodology inspired from recent work on transference in multi-task learning and proposes a new multimodal meta-learning approach called Kernel Modulation (KML) which is claimed to outperform competing approaches.

Reviewers found the problem tackled to be important to the research community and the transference analysis to be an interesting and valuable contribution borrowed from the multi-task learning literature. They expressed reservations about the way in which the contributions were framed (in that the transference analysis feels disconnected from the KML contribution) and how many important details are relegated to the supplementary material. However, given that the authors were very responsive and open to incorporating their feedback, the reviewers and I feel positive about acceptance, if the changes promised are incorporated in the final version (see summary here: https://openreview.net/forum?id=V5prUHOrOP4&noteId=l5jMUHLSraH).
Decision: Accept (Poster)